fs-0.5.4/0000755000000000000000000000000012621617365012077 5ustar rootroot00000000000000fs-0.5.4/fs.egg-info/0000755000000000000000000000000012621617365014201 5ustar rootroot00000000000000fs-0.5.4/fs.egg-info/dependency_links.txt0000644000000000000000000000000112621617364020246 0ustar rootroot00000000000000 fs-0.5.4/fs.egg-info/PKG-INFO0000644000000000000000000000767612621617364015315 0ustar rootroot00000000000000Metadata-Version: 1.1 Name: fs Version: 0.5.4 Summary: Filesystem abstraction layer Home-page: http://pypi.python.org/pypi/fs/ Author: Will McGugan Author-email: will@willmcgugan.com License: BSD Description: PyFilesystem ============ PyFilesystem is an abstraction layer for *filesystems*. In the same way that Python's file-like objects provide a common way of accessing files, PyFilesystem provides a common way of accessing entire filesystems. You can write platform-independent code to work with local files, that also works with any of the supported filesystems (zip, ftp, S3 etc.). Pyfilesystem works with Linux, Windows and Mac. Suported Filesystems --------------------- Here are a few of the filesystems that can be accessed with Pyfilesystem: * **DavFS** access files & directories on a WebDAV server * **FTPFS** access files & directories on an FTP server * **MemoryFS** access files & directories stored in memory (non-permanent but very fast) * **MountFS** creates a virtual directory structure built from other filesystems * **MultiFS** a virtual filesystem that combines a list of filesystems into one, and checks them in order when opening files * **OSFS** the native filesystem * **SFTPFS** access files & directores stored on a Secure FTP server * **S3FS** access files & directories stored on Amazon S3 storage * **TahoeLAFS** access files & directories stored on a Tahoe distributed filesystem * **ZipFS** access files and directories contained in a zip file Example ------- The following snippet prints the total number of bytes contained in all your Python files in `C:/projects` (including sub-directories):: from fs.osfs import OSFS projects_fs = OSFS('C:/projects') print sum(projects_fs.getsize(path) for path in projects_fs.walkfiles(wildcard="*.py")) That is, assuming you are on Windows and have a directory called 'projects' in your C drive. If you are on Linux / Mac, you might replace the second line with something like:: projects_fs = OSFS('~/projects') If you later want to display the total size of Python files stored in a zip file, you could make the following change to the first two lines:: from fs.zipfs import ZipFS projects_fs = ZipFS('source.zip') In fact, you could use any of the supported filesystems above, and the code would continue to work as before. An alternative to explicitly importing the filesystem class you want, is to use an FS opener which opens a filesystem from a URL-like syntax:: from fs.opener import fsopendir projects_fs = fsopendir('C:/projects') You could change ``C:/projects`` to ``zip://source.zip`` to open the zip file, or even ``ftp://ftp.example.org/code/projects/`` to sum up the bytes of Python stored on an ftp server. Screencast ---------- This is from an early version of PyFilesystem, but still relevant http://vimeo.com/12680842 Discussion Group ---------------- http://groups.google.com/group/pyfilesystem-discussion Further Information ------------------- http://www.willmcgugan.com/tag/fs/ Platform: any Classifier: Development Status :: 5 - Production/Stable Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: BSD License Classifier: Operating System :: OS Independent Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2.6 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Topic :: System :: Filesystems fs-0.5.4/fs.egg-info/top_level.txt0000644000000000000000000000000312621617364016723 0ustar rootroot00000000000000fs fs-0.5.4/fs.egg-info/requires.txt0000644000000000000000000000001712621617364016576 0ustar rootroot00000000000000setuptools six fs-0.5.4/fs.egg-info/SOURCES.txt0000644000000000000000000000471612621617364016074 0ustar rootroot00000000000000AUTHORS CHANGES.txt LICENSE.txt MANIFEST.in README.txt setup.py fs/__init__.py fs/appdirfs.py fs/appdirs.py fs/base.py fs/browsewin.py fs/compatibility.py fs/errors.py fs/filelike.py fs/ftpfs.py fs/httpfs.py fs/iotools.py fs/local_functools.py fs/memoryfs.py fs/mountfs.py fs/multifs.py fs/opener.py fs/path.py fs/remote.py fs/remotefs.py fs/rpcfs.py fs/s3fs.py fs/sftpfs.py fs/tempfs.py fs/utils.py fs/watch.py fs/xattrs.py fs/zipfs.py fs.egg-info/PKG-INFO fs.egg-info/SOURCES.txt fs.egg-info/dependency_links.txt fs.egg-info/entry_points.txt fs.egg-info/pbr.json fs.egg-info/requires.txt fs.egg-info/top_level.txt fs.egg-info/version_info.json fs/commands/__init__.py fs/commands/fscat.py fs/commands/fscp.py fs/commands/fsinfo.py fs/commands/fsls.py fs/commands/fsmkdir.py fs/commands/fsmount.py fs/commands/fsmv.py fs/commands/fsrm.py fs/commands/fsserve.py fs/commands/fstree.py fs/commands/runner.py fs/contrib/__init__.py fs/contrib/archivefs.py fs/contrib/sqlitefs.py fs/contrib/bigfs/__init__.py fs/contrib/bigfs/subrangefile.py fs/contrib/davfs/__init__.py fs/contrib/davfs/util.py fs/contrib/davfs/xmlobj.py fs/contrib/tahoelafs/__init__.py fs/contrib/tahoelafs/connection.py fs/contrib/tahoelafs/test_tahoelafs.py fs/contrib/tahoelafs/util.py fs/expose/__init__.py fs/expose/django_storage.py fs/expose/ftp.py fs/expose/http.py fs/expose/importhook.py fs/expose/sftp.py fs/expose/xmlrpc.py fs/expose/dokan/__init__.py fs/expose/dokan/libdokan.py fs/expose/fuse/__init__.py fs/expose/fuse/fuse.py fs/expose/fuse/fuse3.py fs/expose/fuse/fuse_ctypes.py fs/expose/wsgi/__init__.py fs/expose/wsgi/dirtemplate.py fs/expose/wsgi/serve_home.py fs/expose/wsgi/wsgi.py fs/osfs/__init__.py fs/osfs/watch.py fs/osfs/watch_inotify.py fs/osfs/watch_win32.py fs/osfs/xattrs.py fs/tests/__init__.py fs/tests/test_archivefs.py fs/tests/test_errors.py fs/tests/test_expose.py fs/tests/test_fs.py fs/tests/test_ftpfs.py fs/tests/test_importhook.py fs/tests/test_iotools.py fs/tests/test_mountfs.py fs/tests/test_multifs.py fs/tests/test_opener.py fs/tests/test_path.py fs/tests/test_remote.py fs/tests/test_rpcfs.py fs/tests/test_s3fs.py fs/tests/test_sqlitefs.py fs/tests/test_utils.py fs/tests/test_watch.py fs/tests/test_wrapfs.py fs/tests/test_xattr.py fs/tests/test_zipfs.py fs/tests/zipfs_binary_test.py fs/tests/data/UTF-8-demo.txt fs/wrapfs/__init__.py fs/wrapfs/debugfs.py fs/wrapfs/hidedotfilesfs.py fs/wrapfs/hidefs.py fs/wrapfs/lazyfs.py fs/wrapfs/limitsizefs.py fs/wrapfs/readonlyfs.py fs/wrapfs/subfs.pyfs-0.5.4/fs.egg-info/pbr.json0000644000000000000000000000005712621617364015660 0ustar rootroot00000000000000{"is_release": false, "git_version": "d685855"}fs-0.5.4/fs.egg-info/entry_points.txt0000644000000000000000000000050712621617364017500 0ustar rootroot00000000000000[console_scripts] fscat = fs.commands.fscat:run fscp = fs.commands.fscp:run fsinfo = fs.commands.fsinfo:run fsls = fs.commands.fsls:run fsmkdir = fs.commands.fsmkdir:run fsmount = fs.commands.fsmount:run fsmv = fs.commands.fsmv:run fsrm = fs.commands.fsrm:run fsserve = fs.commands.fsserve:run fstree = fs.commands.fstree:run fs-0.5.4/fs.egg-info/version_info.json0000644000000000000000000000012712520752765017576 0ustar rootroot00000000000000{ "release_date": null, "version": 0, "maintainer": "", "body": "" }fs-0.5.4/MANIFEST.in0000664000175000017500000000011312512525115013622 0ustar willwill00000000000000 include AUTHORS include README.txt include LICENSE.txt include CHANGES.txtfs-0.5.4/PKG-INFO0000644000000000000000000000767612621617365013214 0ustar rootroot00000000000000Metadata-Version: 1.1 Name: fs Version: 0.5.4 Summary: Filesystem abstraction layer Home-page: http://pypi.python.org/pypi/fs/ Author: Will McGugan Author-email: will@willmcgugan.com License: BSD Description: PyFilesystem ============ PyFilesystem is an abstraction layer for *filesystems*. In the same way that Python's file-like objects provide a common way of accessing files, PyFilesystem provides a common way of accessing entire filesystems. You can write platform-independent code to work with local files, that also works with any of the supported filesystems (zip, ftp, S3 etc.). Pyfilesystem works with Linux, Windows and Mac. Suported Filesystems --------------------- Here are a few of the filesystems that can be accessed with Pyfilesystem: * **DavFS** access files & directories on a WebDAV server * **FTPFS** access files & directories on an FTP server * **MemoryFS** access files & directories stored in memory (non-permanent but very fast) * **MountFS** creates a virtual directory structure built from other filesystems * **MultiFS** a virtual filesystem that combines a list of filesystems into one, and checks them in order when opening files * **OSFS** the native filesystem * **SFTPFS** access files & directores stored on a Secure FTP server * **S3FS** access files & directories stored on Amazon S3 storage * **TahoeLAFS** access files & directories stored on a Tahoe distributed filesystem * **ZipFS** access files and directories contained in a zip file Example ------- The following snippet prints the total number of bytes contained in all your Python files in `C:/projects` (including sub-directories):: from fs.osfs import OSFS projects_fs = OSFS('C:/projects') print sum(projects_fs.getsize(path) for path in projects_fs.walkfiles(wildcard="*.py")) That is, assuming you are on Windows and have a directory called 'projects' in your C drive. If you are on Linux / Mac, you might replace the second line with something like:: projects_fs = OSFS('~/projects') If you later want to display the total size of Python files stored in a zip file, you could make the following change to the first two lines:: from fs.zipfs import ZipFS projects_fs = ZipFS('source.zip') In fact, you could use any of the supported filesystems above, and the code would continue to work as before. An alternative to explicitly importing the filesystem class you want, is to use an FS opener which opens a filesystem from a URL-like syntax:: from fs.opener import fsopendir projects_fs = fsopendir('C:/projects') You could change ``C:/projects`` to ``zip://source.zip`` to open the zip file, or even ``ftp://ftp.example.org/code/projects/`` to sum up the bytes of Python stored on an ftp server. Screencast ---------- This is from an early version of PyFilesystem, but still relevant http://vimeo.com/12680842 Discussion Group ---------------- http://groups.google.com/group/pyfilesystem-discussion Further Information ------------------- http://www.willmcgugan.com/tag/fs/ Platform: any Classifier: Development Status :: 5 - Production/Stable Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: BSD License Classifier: Operating System :: OS Independent Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2.6 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Topic :: System :: Filesystems fs-0.5.4/fs/0000755000000000000000000000000012621617365012507 5ustar rootroot00000000000000fs-0.5.4/fs/errors.py0000664000175000017500000002707512512525115014402 0ustar willwill00000000000000""" Defines the Exception classes thrown by PyFilesystem objects. Exceptions relating to the underlying filesystem are translated in to one of the following Exceptions. Exceptions that relate to a path store that path in `self.path`. All Exception classes are derived from `FSError` which can be used as a catch-all exception. """ __all__ = ['FSError', 'CreateFailedError', 'PathError', 'InvalidPathError', 'InvalidCharsInPathError', 'OperationFailedError', 'UnsupportedError', 'RemoteConnectionError', 'StorageSpaceError', 'PermissionDeniedError', 'FSClosedError', 'OperationTimeoutError', 'RemoveRootError', 'ResourceError', 'NoSysPathError', 'NoMetaError', 'NoPathURLError', 'ResourceNotFoundError', 'ResourceInvalidError', 'DestinationExistsError', 'DirectoryNotEmptyError', 'ParentDirectoryMissingError', 'ResourceLockedError', 'NoMMapError', 'BackReferenceError', 'convert_fs_errors', 'convert_os_errors', ] import sys import errno import six from fs.path import * from fs.local_functools import wraps class FSError(Exception): """Base exception class for the FS module.""" default_message = "Unspecified error" def __init__(self,msg=None,details=None): if msg is None: msg = self.default_message self.msg = msg self.details = details def __str__(self): keys = {} for k,v in self.__dict__.iteritems(): if isinstance(v,unicode): v = v.encode(sys.getfilesystemencoding()) keys[k] = v return str(self.msg % keys) def __unicode__(self): keys = {} for k,v in self.__dict__.iteritems(): if isinstance(v, six.binary_type): v = v.decode(sys.getfilesystemencoding(), 'replace') keys[k] = v return unicode(self.msg, encoding=sys.getfilesystemencoding(), errors='replace') % keys def __reduce__(self): return (self.__class__,(),self.__dict__.copy(),) class CreateFailedError(FSError): """An exception thrown when a FS could not be created""" default_message = "Unable to create filesystem" class PathError(FSError): """Exception for errors to do with a path string. """ default_message = "Path is invalid: %(path)s" def __init__(self,path="",**kwds): self.path = path super(PathError,self).__init__(**kwds) class InvalidPathError(PathError): """Base exception for fs paths that can't be mapped on to the underlaying filesystem.""" default_message = "Path is invalid on this filesystem %(path)s" class InvalidCharsInPathError(InvalidPathError): """The path contains characters that are invalid on this filesystem""" default_message = "Path contains invalid characters: %(path)s" class OperationFailedError(FSError): """Base exception class for errors associated with a specific operation.""" default_message = "Unable to %(opname)s: unspecified error [%(errno)s - %(details)s]" def __init__(self,opname="",path=None,**kwds): self.opname = opname self.path = path self.errno = getattr(kwds.get("details",None),"errno",None) super(OperationFailedError,self).__init__(**kwds) class UnsupportedError(OperationFailedError): """Exception raised for operations that are not supported by the FS.""" default_message = "Unable to %(opname)s: not supported by this filesystem" class RemoteConnectionError(OperationFailedError): """Exception raised when operations encounter remote connection trouble.""" default_message = "%(opname)s: remote connection errror" class StorageSpaceError(OperationFailedError): """Exception raised when operations encounter storage space trouble.""" default_message = "Unable to %(opname)s: insufficient storage space" class PermissionDeniedError(OperationFailedError): default_message = "Unable to %(opname)s: permission denied" class FSClosedError(OperationFailedError): default_message = "Unable to %(opname)s: the FS has been closed" class OperationTimeoutError(OperationFailedError): default_message = "Unable to %(opname)s: operation timed out" class RemoveRootError(OperationFailedError): default_message = "Can't remove root dir" class ResourceError(FSError): """Base exception class for error associated with a specific resource.""" default_message = "Unspecified resource error: %(path)s" def __init__(self,path="",**kwds): self.path = path self.opname = kwds.pop("opname",None) super(ResourceError,self).__init__(**kwds) class NoSysPathError(ResourceError): """Exception raised when there is no syspath for a given path.""" default_message = "No mapping to OS filesystem: %(path)s" class NoMetaError(FSError): """Exception raised when there is no meta value available.""" default_message = "No meta value named '%(meta_name)s' could be retrieved" def __init__(self, meta_name, msg=None): self.meta_name = meta_name super(NoMetaError, self).__init__(msg) def __reduce__(self): return (self.__class__,(self.meta_name,),self.__dict__.copy(),) class NoPathURLError(ResourceError): """Exception raised when there is no URL form for a given path.""" default_message = "No URL form: %(path)s" class ResourceNotFoundError(ResourceError): """Exception raised when a required resource is not found.""" default_message = "Resource not found: %(path)s" class ResourceInvalidError(ResourceError): """Exception raised when a resource is the wrong type.""" default_message = "Resource is invalid: %(path)s" class DestinationExistsError(ResourceError): """Exception raised when a target destination already exists.""" default_message = "Destination exists: %(path)s" class DirectoryNotEmptyError(ResourceError): """Exception raised when a directory to be removed is not empty.""" default_message = "Directory is not empty: %(path)s" class ParentDirectoryMissingError(ResourceError): """Exception raised when a parent directory is missing.""" default_message = "Parent directory is missing: %(path)s" class ResourceLockedError(ResourceError): """Exception raised when a resource can't be used because it is locked.""" default_message = "Resource is locked: %(path)s" class NoMMapError(ResourceError): """Exception raise when getmmap fails to create a mmap""" default_message = "Can't get mmap for %(path)s" class BackReferenceError(ValueError): """Exception raised when too many backrefs exist in a path (ex: '/..', '/docs/../..').""" def convert_fs_errors(func): """Function wrapper to convert FSError instances into OSError.""" @wraps(func) def wrapper(*args,**kwds): try: return func(*args,**kwds) except ResourceNotFoundError, e: raise OSError(errno.ENOENT,str(e)) except ParentDirectoryMissingError, e: if sys.platform == "win32": raise OSError(errno.ESRCH,str(e)) else: raise OSError(errno.ENOENT,str(e)) except ResourceInvalidError, e: raise OSError(errno.EINVAL,str(e)) except PermissionDeniedError, e: raise OSError(errno.EACCES,str(e)) except ResourceLockedError, e: if sys.platform == "win32": raise WindowsError(32,str(e)) else: raise OSError(errno.EACCES,str(e)) except DirectoryNotEmptyError, e: raise OSError(errno.ENOTEMPTY,str(e)) except DestinationExistsError, e: raise OSError(errno.EEXIST,str(e)) except StorageSpaceError, e: raise OSError(errno.ENOSPC,str(e)) except RemoteConnectionError, e: raise OSError(errno.ENETDOWN,str(e)) except UnsupportedError, e: raise OSError(errno.ENOSYS,str(e)) except FSError, e: raise OSError(errno.EFAULT,str(e)) return wrapper def convert_os_errors(func): """Function wrapper to convert OSError/IOError instances into FSError.""" opname = func.__name__ @wraps(func) def wrapper(self,*args,**kwds): try: return func(self,*args,**kwds) except (OSError,IOError), e: (exc_type,exc_inst,tb) = sys.exc_info() path = getattr(e,"filename",None) if path and path[0] == "/" and hasattr(self,"root_path"): path = normpath(path) if isprefix(self.root_path,path): path = path[len(self.root_path):] if not hasattr(e,"errno") or not e.errno: raise OperationFailedError(opname,details=e),None,tb if e.errno == errno.ENOENT: raise ResourceNotFoundError(path,opname=opname,details=e),None,tb if e.errno == errno.EFAULT: # This can happen when listdir a directory that is deleted by another thread # Best to interpret it as a resource not found raise ResourceNotFoundError(path,opname=opname,details=e),None,tb if e.errno == errno.ESRCH: raise ResourceNotFoundError(path,opname=opname,details=e),None,tb if e.errno == errno.ENOTEMPTY: raise DirectoryNotEmptyError(path,opname=opname,details=e),None,tb if e.errno == errno.EEXIST: raise DestinationExistsError(path,opname=opname,details=e),None,tb if e.errno == 183: # some sort of win32 equivalent to EEXIST raise DestinationExistsError(path,opname=opname,details=e),None,tb if e.errno == errno.ENOTDIR: raise ResourceInvalidError(path,opname=opname,details=e),None,tb if e.errno == errno.EISDIR: raise ResourceInvalidError(path,opname=opname,details=e),None,tb if e.errno == errno.EINVAL: raise ResourceInvalidError(path,opname=opname,details=e),None,tb if e.errno == errno.ENOSPC: raise StorageSpaceError(opname,path=path,details=e),None,tb if e.errno == errno.EPERM: raise PermissionDeniedError(opname,path=path,details=e),None,tb if hasattr(errno,"ENONET") and e.errno == errno.ENONET: raise RemoteConnectionError(opname,path=path,details=e),None,tb if e.errno == errno.ENETDOWN: raise RemoteConnectionError(opname,path=path,details=e),None,tb if e.errno == errno.ECONNRESET: raise RemoteConnectionError(opname,path=path,details=e),None,tb if e.errno == errno.EACCES: if sys.platform == "win32": if e.args[0] and e.args[0] == 32: raise ResourceLockedError(path,opname=opname,details=e),None,tb raise PermissionDeniedError(opname,details=e),None,tb # Sometimes windows gives some random errors... if sys.platform == "win32": if e.errno in (13,): raise ResourceInvalidError(path,opname=opname,details=e),None,tb if e.errno == errno.ENAMETOOLONG: raise PathError(path,details=e),None,tb if e.errno == errno.EOPNOTSUPP: raise UnsupportedError(opname,details=e),None,tb if e.errno == errno.ENOSYS: raise UnsupportedError(opname,details=e),None,tb raise OperationFailedError(opname,details=e),None,tb return wrapper fs-0.5.4/fs/zipfs.py0000664000175000017500000002400612512525221014206 0ustar willwill00000000000000""" fs.zipfs ======== A FS object that represents the contents of a Zip file """ import datetime import os.path from fs.base import * from fs.path import * from fs.errors import * from fs.filelike import StringIO from fs import iotools from zipfile import ZipFile, ZIP_DEFLATED, ZIP_STORED, BadZipfile, LargeZipFile from memoryfs import MemoryFS import tempfs from six import PY3 class ZipOpenError(CreateFailedError): """Thrown when the zip file could not be opened""" pass class ZipNotFoundError(CreateFailedError): """Thrown when the requested zip file does not exist""" pass class _TempWriteFile(object): """Proxies a file object and calls a callback when the file is closed.""" def __init__(self, fs, filename, close_callback): self.fs = fs self.filename = filename self._file = self.fs.open(filename, 'wb+') self.close_callback = close_callback def write(self, data): return self._file.write(data) def tell(self): return self._file.tell() def close(self): self._file.close() self.close_callback(self.filename) def flush(self): self._file.flush() def seek(self, offset, whence): return self._file.seek(offset, whence) def __enter__(self): return self def __exit__(self, type, value, traceback): self.close() class _ExceptionProxy(object): """A placeholder for an object that may no longer be used.""" def __getattr__(self, name): raise ValueError("Zip file has been closed") def __setattr__(self, name, value): raise ValueError("Zip file has been closed") def __nonzero__(self): return False class ZipFS(FS): """A FileSystem that represents a zip file.""" _meta = {'thread_safe': True, 'virtual': False, 'read_only': False, 'unicode_paths': True, 'case_insensitive_paths': False, 'network': False, 'atomic.setcontents': False } def __init__(self, zip_file, mode="r", compression="deflated", allow_zip_64=False, encoding="CP437", thread_synchronize=True): """Create a FS that maps on to a zip file. :param zip_file: a (system) path, or a file-like object :param mode: mode to open zip file, 'r' for reading, 'w' for writing or 'a' for appending :param compression: can be 'deflated' (default) to compress data or 'stored' to just store date :param allow_zip_64: set to True to use zip files greater than 2 GB, default is False :param encoding: the encoding to use for unicode filenames :param thread_synchronize: set to True (default) to enable thread-safety :raises `fs.errors.ZipOpenError`: thrown if the zip file could not be opened :raises `fs.errors.ZipNotFoundError`: thrown if the zip file does not exist (derived from ZipOpenError) """ super(ZipFS, self).__init__(thread_synchronize=thread_synchronize) if compression == "deflated": compression_type = ZIP_DEFLATED elif compression == "stored": compression_type = ZIP_STORED else: raise ValueError("Compression should be 'deflated' (default) or 'stored'") if len(mode) > 1 or mode not in "rwa": raise ValueError("mode must be 'r', 'w' or 'a'") self.zip_mode = mode self.encoding = encoding if isinstance(zip_file, basestring): zip_file = os.path.expanduser(os.path.expandvars(zip_file)) zip_file = os.path.normpath(os.path.abspath(zip_file)) self._zip_file_string = True else: self._zip_file_string = False try: self.zf = ZipFile(zip_file, mode, compression_type, allow_zip_64) except BadZipfile, bzf: raise ZipOpenError("Not a zip file or corrupt (%s)" % str(zip_file), details=bzf) except IOError, ioe: if str(ioe).startswith('[Errno 22] Invalid argument'): raise ZipOpenError("Not a zip file or corrupt (%s)" % str(zip_file), details=ioe) raise ZipNotFoundError("Zip file not found (%s)" % str(zip_file), details=ioe) self.zip_path = str(zip_file) self.temp_fs = None if mode in 'wa': self.temp_fs = tempfs.TempFS() self._path_fs = MemoryFS() if mode in 'ra': self._parse_resource_list() self.read_only = mode == 'r' def __str__(self): return "" % self.zip_path def __unicode__(self): return u"" % self.zip_path def _decode_path(self, path): if PY3: return path return path.decode(self.encoding) def _encode_path(self, path): if PY3: return path return path.encode(self.encoding) def _parse_resource_list(self): for path in self.zf.namelist(): #self._add_resource(path.decode(self.encoding)) self._add_resource(self._decode_path(path)) def _add_resource(self, path): if path.endswith('/'): path = path[:-1] if path: self._path_fs.makedir(path, recursive=True, allow_recreate=True) else: dirpath, _filename = pathsplit(path) if dirpath: self._path_fs.makedir(dirpath, recursive=True, allow_recreate=True) f = self._path_fs.open(path, 'w') f.close() def getmeta(self, meta_name, default=NoDefaultMeta): if meta_name == 'read_only': return self.read_only return super(ZipFS, self).getmeta(meta_name, default) def close(self): """Finalizes the zip file so that it can be read. No further operations will work after this method is called.""" if hasattr(self, 'zf') and self.zf: self.zf.close() self.zf = _ExceptionProxy() @synchronize @iotools.filelike_to_stream def open(self, path, mode='r', buffering=-1, encoding=None, errors=None, newline=None, line_buffering=False, **kwargs): path = normpath(relpath(path)) if 'r' in mode: if self.zip_mode not in 'ra': raise OperationFailedError("open file", path=path, msg="1 Zip file must be opened for reading ('r') or appending ('a')") try: if hasattr(self.zf, 'open') and self._zip_file_string: #return self.zf.open(self._encode_path(path), "r") return self.zf.open(self._encode_path(path), 'rU' if 'U' in mode else 'r') else: contents = self.zf.read(self._encode_path(path)) except KeyError: raise ResourceNotFoundError(path) return StringIO(contents) if 'w' in mode: if self.zip_mode not in 'wa': raise OperationFailedError("open file", path=path, msg="2 Zip file must be opened for writing ('w') or appending ('a')") dirname, _filename = pathsplit(path) if dirname: self.temp_fs.makedir(dirname, recursive=True, allow_recreate=True) self._add_resource(path) f = _TempWriteFile(self.temp_fs, path, self._on_write_close) return f raise ValueError("Mode must contain be 'r' or 'w'") @synchronize def getcontents(self, path, mode="rb", encoding=None, errors=None, newline=None): if not self.exists(path): raise ResourceNotFoundError(path) path = normpath(relpath(path)) try: contents = self.zf.read(self._encode_path(path)) except KeyError: raise ResourceNotFoundError(path) except RuntimeError: raise OperationFailedError("read file", path=path, msg="3 Zip file must be opened with 'r' or 'a' to read") if 'b' in mode: return contents return iotools.decode_binary(contents, encoding=encoding, errors=errors, newline=newline) @synchronize def _on_write_close(self, filename): sys_path = self.temp_fs.getsyspath(filename) self.zf.write(sys_path, self._encode_path(filename)) def desc(self, path): return "%s in zip file %s" % (path, self.zip_path) def isdir(self, path): return self._path_fs.isdir(path) def isfile(self, path): return self._path_fs.isfile(path) def exists(self, path): return self._path_fs.exists(path) @synchronize def makedir(self, dirname, recursive=False, allow_recreate=False): dirname = normpath(dirname) if self.zip_mode not in "wa": raise OperationFailedError("create directory", path=dirname, msg="4 Zip file must be opened for writing ('w') or appending ('a')") if not dirname.endswith('/'): dirname += '/' self._add_resource(dirname) def listdir(self, path="/", wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False): return self._path_fs.listdir(path, wildcard, full, absolute, dirs_only, files_only) @synchronize def getinfo(self, path): if not self.exists(path): raise ResourceNotFoundError(path) path = normpath(path).lstrip('/') try: zi = self.zf.getinfo(self._encode_path(path)) zinfo = dict((attrib, getattr(zi, attrib)) for attrib in dir(zi) if not attrib.startswith('_')) for k, v in zinfo.iteritems(): if callable(v): zinfo[k] = v() except KeyError: zinfo = {'file_size': 0} info = {'size': zinfo['file_size']} if 'date_time' in zinfo: info['modified_time'] = info['created_time'] = datetime.datetime(*zinfo['date_time']) info.update(zinfo) if 'FileHeader' in info: del info['FileHeader'] return info fs-0.5.4/fs/wrapfs/0000755000000000000000000000000012621617365014011 5ustar rootroot00000000000000fs-0.5.4/fs/wrapfs/hidedotfilesfs.py0000664000175000017500000000650312512525115017355 0ustar willwill00000000000000""" fs.wrapfs.hidedotfilesfs ======================== An FS wrapper class for hiding dot-files in directory listings. """ from fs.wrapfs import WrapFS from fs.path import * from fnmatch import fnmatch class HideDotFilesFS(WrapFS): """FS wrapper class that hides dot-files in directory listings. The listdir() function takes an extra keyword argument 'hidden' indicating whether hidden dot-files should be included in the output. It is False by default. """ def is_hidden(self, path): """Check whether the given path should be hidden.""" return path and basename(path)[0] == "." def _encode(self, path): return path def _decode(self, path): return path def listdir(self, path="", wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False, hidden=False): kwds = dict(wildcard=wildcard, full=full, absolute=absolute, dirs_only=dirs_only, files_only=files_only) entries = self.wrapped_fs.listdir(path,**kwds) if not hidden: entries = [e for e in entries if not self.is_hidden(e)] return entries def ilistdir(self, path="", wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False, hidden=False): kwds = dict(wildcard=wildcard, full=full, absolute=absolute, dirs_only=dirs_only, files_only=files_only) for e in self.wrapped_fs.ilistdir(path,**kwds): if hidden or not self.is_hidden(e): yield e def walk(self, path="/", wildcard=None, dir_wildcard=None, search="breadth",hidden=False): if search == "breadth": dirs = [path] while dirs: current_path = dirs.pop() paths = [] for filename in self.listdir(current_path,hidden=hidden): path = pathjoin(current_path, filename) if self.isdir(path): if dir_wildcard is not None: if fnmatch(path, dir_wildcard): dirs.append(path) else: dirs.append(path) else: if wildcard is not None: if fnmatch(path, wildcard): paths.append(filename) else: paths.append(filename) yield (current_path, paths) elif search == "depth": def recurse(recurse_path): for path in self.listdir(recurse_path, wildcard=dir_wildcard, full=True, dirs_only=True,hidden=hidden): for p in recurse(path): yield p yield (recurse_path, self.listdir(recurse_path, wildcard=wildcard, files_only=True,hidden=hidden)) for p in recurse(path): yield p else: raise ValueError("Search should be 'breadth' or 'depth'") def isdirempty(self, path): path = normpath(path) iter_dir = iter(self.listdir(path,hidden=True)) try: iter_dir.next() except StopIteration: return True return False fs-0.5.4/fs/wrapfs/subfs.py0000664000175000017500000000621312512525115015501 0ustar willwill00000000000000""" fs.wrapfs.subfs =============== An FS wrapper class for accessing just a subdirectory for an FS. """ from fs.wrapfs import WrapFS from fs.errors import * from fs.path import * class SubFS(WrapFS): """A SubFS represents a sub directory of another filesystem object. SubFS objects are returned by opendir, which effectively creates a 'sandbox' filesystem that can only access files/dirs under a root path within its 'parent' dir. """ def __init__(self, wrapped_fs, sub_dir): self.sub_dir = abspath(normpath(sub_dir)) super(SubFS, self).__init__(wrapped_fs) def _encode(self, path): return pathjoin(self.sub_dir, relpath(normpath(path))) def _decode(self, path): return abspath(normpath(path))[len(self.sub_dir):] def __str__(self): #return self.wrapped_fs.desc(self.sub_dir) return '' % (self.wrapped_fs, self.sub_dir.lstrip('/')) def __unicode__(self): return u'' % (self.wrapped_fs, self.sub_dir.lstrip('/')) def __repr__(self): return "SubFS(%r, %r)" % (self.wrapped_fs, self.sub_dir) def desc(self, path): if path in ('', '/'): return self.wrapped_fs.desc(self.sub_dir) return '%s!%s' % (self.wrapped_fs.desc(self.sub_dir), path) def setcontents(self, path, data, encoding=None, errors=None, chunk_size=64*1024): path = self._encode(path) return self.wrapped_fs.setcontents(path, data, chunk_size=chunk_size) def opendir(self, path): if not self.exists(path): raise ResourceNotFoundError(path) path = self._encode(path) return self.wrapped_fs.opendir(path) def close(self): self.closed = True def removedir(self, path, recursive=False, force=False): # Careful not to recurse outside the subdir path = normpath(path) if path in ('', '/'): raise RemoveRootError(path) super(SubFS, self).removedir(path, force=force) if recursive: try: if dirname(path) not in ('', '/'): self.removedir(dirname(path), recursive=True) except DirectoryNotEmptyError: pass # if path in ("","/"): # if not force: # for path2 in self.listdir(path): # raise DirectoryNotEmptyError(path) # else: # for path2 in self.listdir(path,absolute=True,files_only=True): # try: # self.remove(path2) # except ResourceNotFoundError: # pass # for path2 in self.listdir(path,absolute=True,dirs_only=True): # try: # self.removedir(path2,force=True) # except ResourceNotFoundError: # pass # else: # super(SubFS,self).removedir(path,force=force) # if recursive: # try: # if dirname(path): # self.removedir(dirname(path),recursive=True) # except DirectoryNotEmptyError: # pass fs-0.5.4/fs/wrapfs/limitsizefs.py0000664000175000017500000002075512512525115016730 0ustar willwill00000000000000""" fs.wrapfs.limitsizefs ===================== An FS wrapper class for limiting the size of the underlying FS. This module provides the class LimitSizeFS, an FS wrapper that can limit the total size of files stored in the wrapped FS. """ from __future__ import with_statement from fs.errors import * from fs.path import * from fs.base import FS, threading, synchronize from fs.wrapfs import WrapFS from fs.filelike import FileWrapper class LimitSizeFS(WrapFS): """FS wrapper class to limit total size of files stored.""" def __init__(self, fs, max_size): super(LimitSizeFS,self).__init__(fs) if max_size < 0: try: max_size = fs.getmeta("total_space") + max_size except NoMetaError: msg = "FS doesn't report total_size; "\ "can't use negative max_size" raise ValueError(msg) self.max_size = max_size self._size_lock = threading.Lock() self._file_sizes = PathMap() self.cur_size = self._get_cur_size() def __getstate__(self): state = super(LimitSizeFS,self).__getstate__() del state["cur_size"] del state["_size_lock"] del state["_file_sizes"] return state def __setstate__(self, state): super(LimitSizeFS,self).__setstate__(state) self._size_lock = threading.Lock() self._file_sizes = PathMap() self.cur_size = self._get_cur_size() def _get_cur_size(self,path="/"): return sum(self.getsize(f) for f in self.walkfiles(path)) def getsyspath(self, path, allow_none=False): # If people could grab syspaths, they could route around our # size protection; no dice! if not allow_none: raise NoSysPathError(path) return None def open(self, path, mode='r', buffering=-1, encoding=None, errors=None, newline=None, line_buffering=False, **kwargs): path = relpath(normpath(path)) with self._size_lock: try: size = self.getsize(path) except ResourceNotFoundError: size = 0 f = super(LimitSizeFS,self).open(path, mode=mode, buffering=buffering, errors=errors, newline=newline, line_buffering=line_buffering, **kwargs) if "w" not in mode: self._set_file_size(path,None,1) else: self.cur_size -= size size = 0 self._set_file_size(path,0,1) return LimitSizeFile(f,mode,size,self,path) def _set_file_size(self,path,size,incrcount=None): try: (cursize,count) = self._file_sizes[path] except KeyError: count = 0 try: cursize = self.getsize(path) except ResourceNotFoundError: cursize = 0 if size is None: size = cursize if count is not None: count += 1 if count == 0: del self._file_sizes[path] else: self._file_sizes[path] = (size,count) def setcontents(self, path, data, chunk_size=64*1024): f = None try: f = self.open(path, 'wb') if hasattr(data, 'read'): chunk = data.read(chunk_size) while chunk: f.write(chunk) chunk = data.read(chunk_size) else: f.write(data) finally: if f is not None: f.close() def _file_closed(self, path): self._set_file_size(path,None,-1) def _ensure_file_size(self, path, size, shrink=False): with self._size_lock: try: (cur_size,_) = self._file_sizes[path] except KeyError: try: cur_size = self.getsize(path) except ResourceNotFoundError: cur_size = 0 self._set_file_size(path,cur_size,1) diff = size - cur_size if diff > 0: if self.cur_size + diff > self.max_size: raise StorageSpaceError("write") self.cur_size += diff self._set_file_size(path,size) return size elif diff < 0 and shrink: self.cur_size += diff self._set_file_size(path,size) return size else: return cur_size # We force use of several base FS methods, # since they will fall back to writing out each file # and thus will route through our size checking logic. def copy(self, src, dst, **kwds): FS.copy(self,src,dst,**kwds) def copydir(self, src, dst, **kwds): FS.copydir(self,src,dst,**kwds) def move(self, src, dst, **kwds): if self.getmeta("atomic.rename",False): if kwds.get("overwrite",False) or not self.exists(dst): try: self.rename(src,dst) return except FSError: pass FS.move(self, src, dst, **kwds) def movedir(self, src, dst, **kwds): overwrite = kwds.get("overwrite",False) if self.getmeta("atomic.rename",False): if kwds.get("overwrite",False) or not self.exists(dst): try: self.rename(src,dst) return except FSError: pass FS.movedir(self,src,dst,**kwds) def rename(self, src, dst): if self.getmeta("atomic.rename",False): try: dst_size = self._get_cur_size(dst) except ResourceNotFoundError: dst_size = 0 super(LimitSizeFS,self).rename(src,dst) with self._size_lock: self.cur_size -= dst_size self._file_sizes.pop(src,None) else: if self.isdir(src): self.movedir(src,dst) else: self.move(src,dst) def remove(self, path): with self._size_lock: try: (size,_) = self._file_sizes[path] except KeyError: size = self.getsize(path) super(LimitSizeFS,self).remove(path) self.cur_size -= size self._file_sizes.pop(path,None) def removedir(self, path, recursive=False, force=False): # Walk and remove directories by hand, so they we # keep the size accounting precisely up to date. for nm in self.listdir(path): if not force: raise DirectoryNotEmptyError(path) cpath = pathjoin(path,nm) try: if self.isdir(cpath): self.removedir(cpath,force=True) else: self.remove(cpath) except ResourceNotFoundError: pass super(LimitSizeFS,self).removedir(path,recursive=recursive) def getinfo(self, path): info = super(LimitSizeFS,self).getinfo(path) try: info["size"] = max(self._file_sizes[path][0],info["size"]) except KeyError: pass return info def getsize(self, path): size = super(LimitSizeFS,self).getsize(path) try: size = max(self._file_sizes[path][0],size) except KeyError: pass return size class LimitSizeFile(FileWrapper): """Filelike wrapper class for use by LimitSizeFS.""" def __init__(self, file, mode, size, fs, path): super(LimitSizeFile,self).__init__(file,mode) self.size = size self.fs = fs self.path = path self._lock = fs._lock @synchronize def _write(self, data, flushing=False): pos = self.wrapped_file.tell() new_size = self.fs._ensure_file_size(self.path, pos+len(data)) res = super(LimitSizeFile,self)._write(data, flushing) self.size = new_size return res @synchronize def _truncate(self, size): new_size = self.fs._ensure_file_size(self.path,size,shrink=True) res = super(LimitSizeFile,self)._truncate(size) self.size = new_size return res @synchronize def close(self): super(LimitSizeFile,self).close() self.fs._file_closed(self.path) fs-0.5.4/fs/wrapfs/lazyfs.py0000664000175000017500000000625112512525115015671 0ustar willwill00000000000000""" fs.wrapfs.lazyfs ================ A class for lazy initialization of an FS object. This module provides the class LazyFS, an FS wrapper class that can lazily initialize its underlying FS object. """ import sys try: from threading import Lock except ImportError: from fs.base import DummyLock as Lock from fs.base import FS from fs.wrapfs import WrapFS class LazyFS(WrapFS): """Simple 'lazy initialization' for FS objects. This FS wrapper can be created with an FS instance, an FS class, or a (class,args,kwds) tuple. The actual FS instance will be created on demand the first time it is accessed. """ def __init__(self, fs): super(LazyFS, self).__init__(fs) self._lazy_creation_lock = Lock() def __unicode__(self): try: wrapped_fs = self.__dict__["wrapped_fs"] except KeyError: # It appears that python2.5 has trouble printing out # classes that define a __unicode__ method. try: return u"" % (self._fsclass,) except TypeError: try: return u"" % (self._fsclass.__name__,) except AttributeError: return u">" else: return u"" % (wrapped_fs,) def __getstate__(self): state = super(LazyFS,self).__getstate__() del state["_lazy_creation_lock"] return state def __setstate__(self, state): super(LazyFS,self).__setstate__(state) self._lazy_creation_lock = Lock() def _get_wrapped_fs(self): """Obtain the wrapped FS instance, creating it if necessary.""" try: fs = self.__dict__["wrapped_fs"] except KeyError: self._lazy_creation_lock.acquire() try: try: fs = self.__dict__["wrapped_fs"] except KeyError: fs = self._fsclass(*self._fsargs,**self._fskwds) self.__dict__["wrapped_fs"] = fs finally: self._lazy_creation_lock.release() return fs def _set_wrapped_fs(self, fs): if isinstance(fs,FS): self.__dict__["wrapped_fs"] = fs elif isinstance(fs,type): self._fsclass = fs self._fsargs = [] self._fskwds = {} elif fs is None: del self.__dict__['wrapped_fs'] else: self._fsclass = fs[0] try: self._fsargs = fs[1] except IndexError: self._fsargs = [] try: self._fskwds = fs[2] except IndexError: self._fskwds = {} wrapped_fs = property(_get_wrapped_fs,_set_wrapped_fs) def setcontents(self, path, data, chunk_size=64*1024): return self.wrapped_fs.setcontents(path, data, chunk_size=chunk_size) def close(self): if not self.closed: # If it was never initialized, create a fake one to close. if "wrapped_fs" not in self.__dict__: self.__dict__["wrapped_fs"] = FS() super(LazyFS,self).close() fs-0.5.4/fs/wrapfs/hidefs.py0000664000175000017500000000301112512525115015612 0ustar willwill00000000000000""" fs.wrapfs.hidefs ================ Removes resources from a directory listing if they match a given set of wildcards """ from fs.wrapfs import WrapFS from fs.path import iteratepath from fs.errors import ResourceNotFoundError import re import fnmatch class HideFS(WrapFS): """FS wrapper that hides resources if they match a wildcard(s). For example, to hide all pyc file and subversion directories from a filesystem:: hide_fs = HideFS(my_fs, "*.pyc", ".svn") """ def __init__(self, wrapped_fs, *hide_wildcards): self._hide_wildcards = [re.compile(fnmatch.translate(wildcard)) for wildcard in hide_wildcards] super(HideFS, self).__init__(wrapped_fs) def _should_hide(self, path): return any(any(wildcard.match(part) for wildcard in self._hide_wildcards) for part in iteratepath(path)) def _encode(self, path): if self._should_hide(path): raise ResourceNotFoundError(path) return path def _decode(self, path): return path def exists(self, path): if self._should_hide(path): return False return super(HideFS, self).exists(path) def listdir(self, path="", *args, **kwargs): entries = super(HideFS, self).listdir(path, *args, **kwargs) entries = [entry for entry in entries if not self._should_hide(entry)] return entries if __name__ == "__main__": from fs.osfs import OSFS hfs = HideFS(OSFS('~/projects/pyfilesystem'), "*.pyc", ".svn") hfs.tree() fs-0.5.4/fs/wrapfs/readonlyfs.py0000664000175000017500000000470212512525115016526 0ustar willwill00000000000000""" fs.wrapfs.readonlyfs ==================== An FS wrapper class for blocking operations that would modify the FS. """ from fs.base import NoDefaultMeta from fs.wrapfs import WrapFS from fs.errors import UnsupportedError, NoSysPathError class ReadOnlyFS(WrapFS): """ Makes a FS object read only. Any operation that could potentially modify the underlying file system will throw an UnsupportedError Note that this isn't a secure sandbox, untrusted code could work around the read-only restrictions by getting the base class. Its main purpose is to provide a degree of safety if you want to protect an FS object from accidental modification. """ def getmeta(self, meta_name, default=NoDefaultMeta): if meta_name == 'read_only': return True return self.wrapped_fs.getmeta(meta_name, default) def hasmeta(self, meta_name): if meta_name == 'read_only': return True return self.wrapped_fs.hasmeta(meta_name) def getsyspath(self, path, allow_none=False): """ Doesn't technically modify the filesystem but could be used to work around read-only restrictions. """ if allow_none: return None raise NoSysPathError(path) def open(self, path, mode='r', buffering=-1, encoding=None, errors=None, newline=None, line_buffering=False, **kwargs): """ Only permit read access """ if 'w' in mode or 'a' in mode or '+' in mode: raise UnsupportedError('write') return super(ReadOnlyFS, self).open(path, mode=mode, buffering=buffering, encoding=encoding, errors=errors, newline=newline, line_buffering=line_buffering, **kwargs) def _no_can_do(self, *args, **kwargs): """ Replacement method for methods that can modify the file system """ raise UnsupportedError('write') move = _no_can_do movedir = _no_can_do copy = _no_can_do copydir = _no_can_do makedir = _no_can_do rename = _no_can_do setxattr = _no_can_do delxattr = _no_can_do remove = _no_can_do removedir = _no_can_do settimes = _no_can_do setcontents = _no_can_do createfile = _no_can_do fs-0.5.4/fs/wrapfs/__init__.py0000664000175000017500000004343512512525115016125 0ustar willwill00000000000000""" fs.wrapfs ========= A class for wrapping an existing FS object with additional functionality. This module provides the class WrapFS, a base class for objects that wrap another FS object and provide some transformation of its contents. It could be very useful for implementing e.g. transparent encryption or compression services. For a simple example of how this class could be used, see the 'HideDotFilesFS' class in the module fs.wrapfs.hidedotfilesfs. This wrapper implements the standard unix shell functionality of hiding dot-files in directory listings. """ import re import sys import fnmatch import threading from fs.base import FS, threading, synchronize, NoDefaultMeta from fs.errors import * from fs.path import * from fs.local_functools import wraps def rewrite_errors(func): """Re-write paths in errors raised by wrapped FS objects.""" @wraps(func) def wrapper(self,*args,**kwds): try: return func(self,*args,**kwds) except ResourceError, e: (exc_type,exc_inst,tb) = sys.exc_info() try: e.path = self._decode(e.path) except (AttributeError, ValueError, TypeError): raise e, None, tb raise return wrapper class WrapFS(FS): """FS that wraps another FS, providing translation etc. This class allows simple transforms to be applied to the names and/or contents of files in an FS. It could be used to implement e.g. compression or encryption in a relatively painless manner. The following methods can be overridden to control how files are accessed in the underlying FS object: * _file_wrap(file, mode): called for each file that is opened from the underlying FS; may return a modified file-like object. * _encode(path): encode a path for access in the underlying FS * _decode(path): decode a path from the underlying FS If the required path translation proceeds one component at a time, it may be simpler to override the _encode_name() and _decode_name() methods. """ def __init__(self, fs): super(WrapFS, self).__init__() try: self._lock = fs._lock except (AttributeError,FSError): self._lock = self._lock = threading.RLock() self.wrapped_fs = fs def _file_wrap(self, f, mode): """Apply wrapping to an opened file.""" return f def _encode_name(self, name): """Encode path component for the underlying FS.""" return name def _decode_name(self, name): """Decode path component from the underlying FS.""" return name def _encode(self, path): """Encode path for the underlying FS.""" e_names = [] for name in iteratepath(path): if name == "": e_names.append("") else: e_names.append(self._encode_name(name)) return "/".join(e_names) def _decode(self, path): """Decode path from the underlying FS.""" d_names = [] for name in iteratepath(path): if name == "": d_names.append("") else: d_names.append(self._decode_name(name)) return "/".join(d_names) def _adjust_mode(self, mode): """Adjust the mode used to open a file in the underlying FS. This method takes the mode given when opening a file, and should return a two-tuple giving the mode to be used in this FS as first item, and the mode to be used in the underlying FS as the second. An example of why this is needed is a WrapFS subclass that does transparent file compression - in this case files from the wrapped FS cannot be opened in append mode. """ return (mode, mode) def __unicode__(self): return u"<%s: %s>" % (self.__class__.__name__,self.wrapped_fs,) #def __str__(self): # return unicode(self).encode(sys.getdefaultencoding(),"replace") @rewrite_errors def getmeta(self, meta_name, default=NoDefaultMeta): return self.wrapped_fs.getmeta(meta_name, default) @rewrite_errors def hasmeta(self, meta_name): return self.wrapped_fs.hasmeta(meta_name) @rewrite_errors def validatepath(self, path): return self.wrapped_fs.validatepath(self._encode(path)) @rewrite_errors def getsyspath(self, path, allow_none=False): return self.wrapped_fs.getsyspath(self._encode(path), allow_none) @rewrite_errors def getpathurl(self, path, allow_none=False): return self.wrapped_fs.getpathurl(self._encode(path), allow_none) @rewrite_errors def hassyspath(self, path): return self.wrapped_fs.hassyspath(self._encode(path)) @rewrite_errors def open(self, path, mode='r', **kwargs): (mode, wmode) = self._adjust_mode(mode) f = self.wrapped_fs.open(self._encode(path), wmode, **kwargs) return self._file_wrap(f, mode) @rewrite_errors def setcontents(self, path, data, encoding=None, errors=None, chunk_size=64*1024): # We can't pass setcontents() through to the wrapped FS if the # wrapper has defined a _file_wrap method, as it would bypass # the file contents wrapping. #if self._file_wrap.im_func is WrapFS._file_wrap.im_func: if getattr(self.__class__, '_file_wrap', None) is getattr(WrapFS, '_file_wrap', None): return self.wrapped_fs.setcontents(self._encode(path), data, encoding=encoding, errors=errors, chunk_size=chunk_size) else: return super(WrapFS, self).setcontents(path, data, encoding=encoding, errors=errors, chunk_size=chunk_size) @rewrite_errors def createfile(self, path, wipe=False): return self.wrapped_fs.createfile(self._encode(path), wipe=wipe) @rewrite_errors def exists(self, path): return self.wrapped_fs.exists(self._encode(path)) @rewrite_errors def isdir(self, path): return self.wrapped_fs.isdir(self._encode(path)) @rewrite_errors def isfile(self, path): return self.wrapped_fs.isfile(self._encode(path)) @rewrite_errors def listdir(self, path="", wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False): kwds = dict(wildcard=wildcard, full=full, absolute=absolute, dirs_only=dirs_only, files_only=files_only) full = kwds.pop("full",False) absolute = kwds.pop("absolute",False) wildcard = kwds.pop("wildcard",None) if wildcard is None: wildcard = lambda fn:True elif not callable(wildcard): wildcard_re = re.compile(fnmatch.translate(wildcard)) wildcard = lambda fn:bool (wildcard_re.match(fn)) entries = [] enc_path = self._encode(path) for e in self.wrapped_fs.listdir(enc_path,**kwds): e = basename(self._decode(pathcombine(enc_path,e))) if not wildcard(e): continue if full: e = pathcombine(path,e) elif absolute: e = abspath(pathcombine(path,e)) entries.append(e) return entries @rewrite_errors def ilistdir(self, path="", wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False): kwds = dict(wildcard=wildcard, full=full, absolute=absolute, dirs_only=dirs_only, files_only=files_only) full = kwds.pop("full",False) absolute = kwds.pop("absolute",False) wildcard = kwds.pop("wildcard",None) if wildcard is None: wildcard = lambda fn:True elif not callable(wildcard): wildcard_re = re.compile(fnmatch.translate(wildcard)) wildcard = lambda fn:bool (wildcard_re.match(fn)) enc_path = self._encode(path) for e in self.wrapped_fs.ilistdir(enc_path,**kwds): e = basename(self._decode(pathcombine(enc_path,e))) if not wildcard(e): continue if full: e = pathcombine(path,e) elif absolute: e = abspath(pathcombine(path,e)) yield e @rewrite_errors def listdirinfo(self, path="", wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False): kwds = dict(wildcard=wildcard, full=full, absolute=absolute, dirs_only=dirs_only, files_only=files_only) full = kwds.pop("full",False) absolute = kwds.pop("absolute",False) wildcard = kwds.pop("wildcard",None) if wildcard is None: wildcard = lambda fn:True elif not callable(wildcard): wildcard_re = re.compile(fnmatch.translate(wildcard)) wildcard = lambda fn:bool (wildcard_re.match(fn)) entries = [] enc_path = self._encode(path) for (nm,info) in self.wrapped_fs.listdirinfo(enc_path,**kwds): nm = basename(self._decode(pathcombine(enc_path,nm))) if not wildcard(nm): continue if full: nm = pathcombine(path,nm) elif absolute: nm = abspath(pathcombine(path,nm)) entries.append((nm,info)) return entries @rewrite_errors def ilistdirinfo(self, path="", wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False): kwds = dict(wildcard=wildcard, full=full, absolute=absolute, dirs_only=dirs_only, files_only=files_only) full = kwds.pop("full",False) absolute = kwds.pop("absolute",False) wildcard = kwds.pop("wildcard",None) if wildcard is None: wildcard = lambda fn:True elif not callable(wildcard): wildcard_re = re.compile(fnmatch.translate(wildcard)) wildcard = lambda fn:bool (wildcard_re.match(fn)) enc_path = self._encode(path) for (nm,info) in self.wrapped_fs.ilistdirinfo(enc_path,**kwds): nm = basename(self._decode(pathcombine(enc_path,nm))) if not wildcard(nm): continue if full: nm = pathcombine(path,nm) elif absolute: nm = abspath(pathcombine(path,nm)) yield (nm,info) @rewrite_errors def walk(self,path="/",wildcard=None,dir_wildcard=None,search="breadth",ignore_errors=False): if dir_wildcard is not None: # If there is a dir_wildcard, fall back to the default impl # that uses listdir(). Otherwise we run the risk of enumerating # lots of directories that will just be thrown away. for item in super(WrapFS,self).walk(path,wildcard,dir_wildcard,search,ignore_errors): yield item # Otherwise, the wrapped FS may provide a more efficient impl # which we can use directly. else: if wildcard is not None and not callable(wildcard): wildcard_re = re.compile(fnmatch.translate(wildcard)) wildcard = lambda fn:bool (wildcard_re.match(fn)) for (dirpath,filepaths) in self.wrapped_fs.walk(self._encode(path),search=search,ignore_errors=ignore_errors): filepaths = [basename(self._decode(pathcombine(dirpath,p))) for p in filepaths] dirpath = abspath(self._decode(dirpath)) if wildcard is not None: filepaths = [p for p in filepaths if wildcard(p)] yield (dirpath,filepaths) @rewrite_errors def walkfiles(self,path="/",wildcard=None,dir_wildcard=None,search="breadth",ignore_errors=False): if dir_wildcard is not None: # If there is a dir_wildcard, fall back to the default impl # that uses listdir(). Otherwise we run the risk of enumerating # lots of directories that will just be thrown away. for item in super(WrapFS,self).walkfiles(path,wildcard,dir_wildcard,search,ignore_errors): yield item # Otherwise, the wrapped FS may provide a more efficient impl # which we can use directly. else: if wildcard is not None and not callable(wildcard): wildcard_re = re.compile(fnmatch.translate(wildcard)) wildcard = lambda fn:bool (wildcard_re.match(fn)) for filepath in self.wrapped_fs.walkfiles(self._encode(path),search=search,ignore_errors=ignore_errors): filepath = abspath(self._decode(filepath)) if wildcard is not None: if not wildcard(basename(filepath)): continue yield filepath @rewrite_errors def walkdirs(self,path="/",wildcard=None,search="breadth",ignore_errors=False): if wildcard is not None: # If there is a wildcard, fall back to the default impl # that uses listdir(). Otherwise we run the risk of enumerating # lots of directories that will just be thrown away. for item in super(WrapFS,self).walkdirs(path,wildcard,search,ignore_errors): yield item # Otherwise, the wrapped FS may provide a more efficient impl # which we can use directly. else: for dirpath in self.wrapped_fs.walkdirs(self._encode(path),search=search,ignore_errors=ignore_errors): yield abspath(self._decode(dirpath)) @rewrite_errors def makedir(self, path, *args, **kwds): return self.wrapped_fs.makedir(self._encode(path),*args,**kwds) @rewrite_errors def remove(self, path): return self.wrapped_fs.remove(self._encode(path)) @rewrite_errors def removedir(self, path, *args, **kwds): return self.wrapped_fs.removedir(self._encode(path),*args,**kwds) @rewrite_errors def rename(self, src, dst): return self.wrapped_fs.rename(self._encode(src),self._encode(dst)) @rewrite_errors def getinfo(self, path): return self.wrapped_fs.getinfo(self._encode(path)) @rewrite_errors def settimes(self, path, *args, **kwds): return self.wrapped_fs.settimes(self._encode(path), *args,**kwds) @rewrite_errors def desc(self, path): return self.wrapped_fs.desc(self._encode(path)) @rewrite_errors def copy(self, src, dst, **kwds): return self.wrapped_fs.copy(self._encode(src),self._encode(dst),**kwds) @rewrite_errors def move(self, src, dst, **kwds): return self.wrapped_fs.move(self._encode(src),self._encode(dst),**kwds) @rewrite_errors def movedir(self, src, dst, **kwds): return self.wrapped_fs.movedir(self._encode(src),self._encode(dst),**kwds) @rewrite_errors def copydir(self, src, dst, **kwds): return self.wrapped_fs.copydir(self._encode(src),self._encode(dst),**kwds) @rewrite_errors def getxattr(self, path, name, default=None): try: return self.wrapped_fs.getxattr(self._encode(path),name,default) except AttributeError: raise UnsupportedError("getxattr") @rewrite_errors def setxattr(self, path, name, value): try: return self.wrapped_fs.setxattr(self._encode(path),name,value) except AttributeError: raise UnsupportedError("setxattr") @rewrite_errors def delxattr(self, path, name): try: return self.wrapped_fs.delxattr(self._encode(path),name) except AttributeError: raise UnsupportedError("delxattr") @rewrite_errors def listxattrs(self, path): try: return self.wrapped_fs.listxattrs(self._encode(path)) except AttributeError: raise UnsupportedError("listxattrs") def __getattr__(self, attr): # These attributes can be used by the destructor, but may not be # defined if there are errors in the constructor. if attr == "closed": return False if attr == "wrapped_fs": return None if attr.startswith("_"): raise AttributeError(attr) return getattr(self.wrapped_fs,attr) @rewrite_errors def close(self): if not self.closed: self.wrapped_fs.close() super(WrapFS,self).close() self.wrapped_fs = None def wrap_fs_methods(decorator, cls=None, exclude=[]): """Apply the given decorator to all FS methods on the given class. This function can be used in two ways. When called with two arguments it applies the given function 'decorator' to each FS method of the given class. When called with just a single argument, it creates and returns a class decorator which will do the same thing when applied. So you can use it like this:: wrap_fs_methods(mydecorator,MyFSClass) Or on more recent Python versions, like this:: @wrap_fs_methods(mydecorator) class MyFSClass(FS): ... """ def apply_decorator(cls): for method_name in wrap_fs_methods.method_names: if method_name in exclude: continue method = getattr(cls,method_name,None) if method is not None: setattr(cls,method_name,decorator(method)) return cls if cls is not None: return apply_decorator(cls) else: return apply_decorator wrap_fs_methods.method_names = ["open","exists","isdir","isfile","listdir", "makedir","remove","setcontents","removedir","rename","getinfo","copy", "move","copydir","movedir","close","getxattr","setxattr","delxattr", "listxattrs","validatepath","getsyspath","createfile", "hasmeta", "getmeta","listdirinfo", "ilistdir","ilistdirinfo"] fs-0.5.4/fs/wrapfs/debugfs.py0000664000175000017500000001256012512525115016000 0ustar willwill00000000000000''' @author: Marek Palatinus @license: Public domain DebugFS is a wrapper around filesystems to help developers debug their work. I wrote this class mainly for debugging TahoeLAFS and for fine tuning TahoeLAFS over Dokan with higher-level aplications like Total Comander, Winamp etc. Did you know that Total Commander need to open file before it delete them? :-) I hope DebugFS can be helpful also for other filesystem developers, especially for those who are trying to implement their first one (like me). DebugFS prints to stdout (by default) all attempts to filesystem interface, prints parameters and results. Basic usage: fs = DebugFS(OSFS('~'), identifier='OSFS@home', \ skip=('_lock', 'listdir', 'listdirinfo')) print fs.listdir('.') print fs.unsupportedfunction() Error levels: DEBUG: Print everything (asking for methods, calls, response, exception) INFO: Print calls, responses, exception ERROR: Print only exceptions CRITICAL: Print only exceptions not derived from fs.errors.FSError How to change error level: import logging logger = logging.getLogger('fs.debugfs') logger.setLevel(logging.CRITICAL) fs = DebugFS(OSFS('~') print fs.listdir('.') ''' import logging from logging import DEBUG, INFO, ERROR, CRITICAL import sys import fs from fs.errors import FSError logger = fs.getLogger('fs.debugfs') logger.setLevel(logging.DEBUG) logger.addHandler(logging.StreamHandler()) class DebugFS(object): def __init__(self, fs, identifier=None, skip=(), verbose=True): ''' fs - Reference to object to debug identifier - Custom string-like object will be added to each log line as identifier. skip - list of method names which DebugFS should not log ''' self.__wrapped_fs = fs self.__identifier = identifier self.__skip = skip self.__verbose = verbose super(DebugFS, self).__init__() def __log(self, level, message): if self.__identifier: logger.log(level, '(%s) %s' % (self.__identifier, message)) else: logger.log(level, message) def __parse_param(self, value): if isinstance(value, basestring): if len(value) > 60: value = "%s ... (length %d)" % (repr(value[:60]), len(value)) else: value = repr(value) elif isinstance(value, list): value = "%s (%d items)" % (repr(value[:3]), len(value)) elif isinstance(value, dict): items = {} for k, v in value.items()[:3]: items[k] = v value = "%s (%d items)" % (repr(items), len(value)) else: value = repr(value) return value def __parse_args(self, *arguments, **kwargs): args = [self.__parse_param(a) for a in arguments] for k, v in kwargs.items(): args.append("%s=%s" % (k, self.__parse_param(v))) args = ','.join(args) if args: args = "(%s)" % args return args def __report(self, msg, key, value, *arguments, **kwargs): if key in self.__skip: return args = self.__parse_args(*arguments, **kwargs) value = self.__parse_param(value) self.__log(INFO, "%s %s%s -> %s" % (msg, str(key), args, value)) def __getattr__(self, key): if key.startswith('__'): # Internal calls, nothing interesting return object.__getattribute__(self, key) try: attr = getattr(self.__wrapped_fs, key) except AttributeError, e: self.__log(DEBUG, "Asking for not implemented method %s" % key) raise e except Exception, e: self.__log(CRITICAL, "Exception %s: %s" % \ (e.__class__.__name__, str(e))) raise e if not callable(attr): if key not in self.__skip: self.__report("Get attribute", key, attr) return attr def _method(*args, **kwargs): try: value = attr(*args, **kwargs) self.__report("Call method", key, value, *args, **kwargs) except FSError, e: self.__log(ERROR, "Call method %s%s -> Exception %s: %s" % \ (key, self.__parse_args(*args, **kwargs), \ e.__class__.__name__, str(e))) (exc_type,exc_inst,tb) = sys.exc_info() raise e, None, tb except Exception, e: self.__log(CRITICAL, "Call method %s%s -> Non-FS exception %s: %s" %\ (key, self.__parse_args(*args, **kwargs), \ e.__class__.__name__, str(e))) (exc_type,exc_inst,tb) = sys.exc_info() raise e, None, tb return value if self.__verbose: if key not in self.__skip: self.__log(DEBUG, "Asking for method %s" % key) return _method fs-0.5.4/fs/browsewin.py0000664000175000017500000001407212512525115015076 0ustar willwill00000000000000#!/usr/bin/env python """ fs.browsewin ============ Creates a window which can be used to browse the contents of a filesystem. To use, call the 'browse' method with a filesystem object. Double click a file or directory to display its properties. Requires wxPython. """ import wx import wx.gizmos from fs.path import isdotfile, pathsplit from fs.errors import FSError class InfoFrame(wx.Frame): def __init__(self, parent, path, desc, info): wx.Frame.__init__(self, parent, -1, style=wx.DEFAULT_FRAME_STYLE, size=(500, 500)) self.SetTitle("FS Object info - %s (%s)" % (path, desc)) keys = info.keys() keys.sort() self.list_ctrl = wx.ListCtrl(self, -1, style=wx.LC_REPORT|wx.SUNKEN_BORDER) self.list_ctrl.InsertColumn(0, "Key") self.list_ctrl.InsertColumn(1, "Value") self.list_ctrl.SetColumnWidth(0, 190) self.list_ctrl.SetColumnWidth(1, 300) for key in sorted(keys, key=lambda k:k.lower()): self.list_ctrl.Append((key, unicode(info.get(key)))) self.Center() class BrowseFrame(wx.Frame): def __init__(self, fs, hide_dotfiles=False): wx.Frame.__init__(self, None, size=(1000, 600)) self.fs = fs self.hide_dotfiles = hide_dotfiles self.SetTitle("FS Browser - " + unicode(fs)) self.tree = wx.gizmos.TreeListCtrl(self, -1, style=wx.TR_DEFAULT_STYLE | wx.TR_HIDE_ROOT) self.tree.AddColumn("File System") self.tree.AddColumn("Description") self.tree.AddColumn("Size") self.tree.AddColumn("Created") self.tree.SetColumnWidth(0, 300) self.tree.SetColumnWidth(1, 250) self.tree.SetColumnWidth(2, 150) self.tree.SetColumnWidth(3, 250) self.root_id = self.tree.AddRoot('root', data=wx.TreeItemData( {'path':"/", 'expanded':False} )) rid = self.tree.GetItemData(self.root_id) isz = (16, 16) il = wx.ImageList(isz[0], isz[1]) self.fldridx = il.Add(wx.ArtProvider_GetBitmap(wx.ART_FOLDER, wx.ART_OTHER, isz)) self.fldropenidx = il.Add(wx.ArtProvider_GetBitmap(wx.ART_FILE_OPEN, wx.ART_OTHER, isz)) self.fileidx = il.Add(wx.ArtProvider_GetBitmap(wx.ART_NORMAL_FILE, wx.ART_OTHER, isz)) self.tree.SetImageList(il) self.il = il self.tree.SetItemImage(self.root_id, self.fldridx, wx.TreeItemIcon_Normal) self.tree.SetItemImage(self.root_id, self.fldropenidx, wx.TreeItemIcon_Expanded) self.Bind(wx.EVT_TREE_ITEM_EXPANDING, self.OnItemExpanding) self.Bind(wx.EVT_TREE_ITEM_ACTIVATED, self.OnItemActivated) wx.CallAfter(self.OnInit) def OnInit(self): self.expand(self.root_id) def expand(self, item_id): item_data = self.tree.GetItemData(item_id).GetData() path = item_data["path"] if not self.fs.isdir(path): return if item_data['expanded']: return try: paths = ( [(True, p) for p in self.fs.listdir(path, absolute=True, dirs_only=True)] + [(False, p) for p in self.fs.listdir(path, absolute=True, files_only=True)] ) except FSError, e: msg = "Failed to get directory listing for %s\n\nThe following error was reported:\n\n%s" % (path, e) wx.MessageDialog(self, msg, "Error listing directory", wx.OK).ShowModal() paths = [] #paths = [(self.fs.isdir(p), p) for p in self.fs.listdir(path, absolute=True)] if self.hide_dotfiles: paths = [p for p in paths if not isdotfile(p[1])] if not paths: #self.tree.SetItemHasChildren(item_id, False) #self.tree.Collapse(item_id) return paths.sort(key=lambda p:(not p[0], p[1].lower())) for is_dir, new_path in paths: name = pathsplit(new_path)[-1] new_item = self.tree.AppendItem(item_id, name, data=wx.TreeItemData({'path':new_path, 'expanded':False})) info = self.fs.getinfo(new_path) if is_dir: self.tree.SetItemHasChildren(new_item) self.tree.SetItemImage(new_item, self.fldridx, 0, wx.TreeItemIcon_Normal) self.tree.SetItemImage(new_item, self.fldropenidx, 0, wx.TreeItemIcon_Expanded) self.tree.SetItemText(new_item, "", 2) ct = info.get('created_time', None) if ct is not None: self.tree.SetItemText(new_item, ct.ctime(), 3) else: self.tree.SetItemText(new_item, 'unknown', 3) else: self.tree.SetItemImage(new_item, self.fileidx, 0, wx.TreeItemIcon_Normal) self.tree.SetItemText(new_item, str(info.get('size', '?'))+ " bytes", 2) ct = info.get('created_time', None) if ct is not None: self.tree.SetItemText(new_item, ct.ctime(), 3) else: self.tree.SetItemText(new_item, 'unknown', 3) self.tree.SetItemText(new_item, self.fs.desc(new_path), 1) item_data['expanded'] = True self.tree.Expand(item_id) def OnItemExpanding(self, e): self.expand(e.GetItem()) e.Skip() def OnItemActivated(self, e): item_data = self.tree.GetItemData(e.GetItem()).GetData() path = item_data["path"] info = self.fs.getinfo(path) info_frame = InfoFrame(self, path, self.fs.desc(path), info) info_frame.Show() info_frame.CenterOnParent() def browse(fs, hide_dotfiles=False): """Displays a window containing a tree control that displays an FS object. Double-click a file/folder to display extra info. :param fs: A filesystem object :param hide_dotfiles: If True, files and folders that begin with a dot will be hidden """ app = wx.PySimpleApp() frame = BrowseFrame(fs, hide_dotfiles=hide_dotfiles) frame.Show() app.MainLoop() if __name__ == "__main__": from osfs import OSFS home_fs = OSFS("~/") browse(home_fs, True) fs-0.5.4/fs/utils.py0000664000175000017500000005370412621467141014231 0ustar willwill00000000000000# -*- coding: utf-8 -*- """ The `utils` module provides a number of utility functions that don't belong in the Filesystem interface. Generally the functions in this module work with multiple Filesystems, for instance moving and copying between non-similar Filesystems. """ __all__ = ['copyfile', 'movefile', 'movedir', 'copydir', 'countbytes', 'isfile', 'isdir', 'find_duplicates', 'print_fs', 'open_atomic_write'] import os import sys import stat import six from six import PY3 from fs.mountfs import MountFS from fs.path import pathjoin from fs.errors import DestinationExistsError, RemoveRootError from fs.base import FS def copyfile(src_fs, src_path, dst_fs, dst_path, overwrite=True, chunk_size=64*1024): """Copy a file from one filesystem to another. Will use system copyfile, if both files have a syspath. Otherwise file will be copied a chunk at a time. :param src_fs: Source filesystem object :param src_path: -- Source path :param dst_fs: Destination filesystem object :param dst_path: Destination filesystem object :param chunk_size: Size of chunks to move if system copyfile is not available (default 64K) """ # If the src and dst fs objects are the same, then use a direct copy if src_fs is dst_fs: src_fs.copy(src_path, dst_path, overwrite=overwrite) return src_syspath = src_fs.getsyspath(src_path, allow_none=True) dst_syspath = dst_fs.getsyspath(dst_path, allow_none=True) if not overwrite and dst_fs.exists(dst_path): raise DestinationExistsError(dst_path) # System copy if there are two sys paths if src_syspath is not None and dst_syspath is not None: FS._shutil_copyfile(src_syspath, dst_syspath) return src_lock = getattr(src_fs, '_lock', None) if src_lock is not None: src_lock.acquire() try: src = None try: src = src_fs.open(src_path, 'rb') dst_fs.setcontents(dst_path, src, chunk_size=chunk_size) finally: if src is not None: src.close() finally: if src_lock is not None: src_lock.release() def copyfile_non_atomic(src_fs, src_path, dst_fs, dst_path, overwrite=True, chunk_size=64*1024): """A non atomic version of copyfile (will not block other threads using src_fs or dst_fst) :param src_fs: Source filesystem object :param src_path: -- Source path :param dst_fs: Destination filesystem object :param dst_path: Destination filesystem object :param chunk_size: Size of chunks to move if system copyfile is not available (default 64K) """ if not overwrite and dst_fs.exists(dst_path): raise DestinationExistsError(dst_path) src = None dst = None try: src = src_fs.open(src_path, 'rb') dst = dst_fs.open(dst_path, 'wb') write = dst.write read = src.read chunk = read(chunk_size) while chunk: write(chunk) chunk = read(chunk_size) finally: if src is not None: src.close() if dst is not None: dst.close() def movefile(src_fs, src_path, dst_fs, dst_path, overwrite=True, chunk_size=64*1024): """Move a file from one filesystem to another. Will use system copyfile, if both files have a syspath. Otherwise file will be copied a chunk at a time. :param src_fs: Source filesystem object :param src_path: Source path :param dst_fs: Destination filesystem object :param dst_path: Destination filesystem object :param chunk_size: Size of chunks to move if system copyfile is not available (default 64K) """ src_syspath = src_fs.getsyspath(src_path, allow_none=True) dst_syspath = dst_fs.getsyspath(dst_path, allow_none=True) if not overwrite and dst_fs.exists(dst_path): raise DestinationExistsError(dst_path) if src_fs is dst_fs: src_fs.move(src_path, dst_path, overwrite=overwrite) return # System copy if there are two sys paths if src_syspath is not None and dst_syspath is not None: FS._shutil_movefile(src_syspath, dst_syspath) return src_lock = getattr(src_fs, '_lock', None) if src_lock is not None: src_lock.acquire() try: src = None try: # Chunk copy src = src_fs.open(src_path, 'rb') dst_fs.setcontents(dst_path, src, chunk_size=chunk_size) except: raise else: src_fs.remove(src_path) finally: if src is not None: src.close() finally: if src_lock is not None: src_lock.release() def movefile_non_atomic(src_fs, src_path, dst_fs, dst_path, overwrite=True, chunk_size=64*1024): """A non atomic version of movefile (wont block other threads using src_fs or dst_fs) :param src_fs: Source filesystem object :param src_path: Source path :param dst_fs: Destination filesystem object :param dst_path: Destination filesystem object :param chunk_size: Size of chunks to move if system copyfile is not available (default 64K) """ if not overwrite and dst_fs.exists(dst_path): raise DestinationExistsError(dst_path) src = None dst = None try: # Chunk copy src = src_fs.open(src_path, 'rb') dst = dst_fs.open(dst_path, 'wb') write = dst.write read = src.read chunk = read(chunk_size) while chunk: write(chunk) chunk = read(chunk_size) except: raise else: src_fs.remove(src_path) finally: if src is not None: src.close() if dst is not None: dst.close() def movedir(fs1, fs2, create_destination=True, ignore_errors=False, chunk_size=64*1024): """Moves contents of a directory from one filesystem to another. :param fs1: A tuple of (, ) :param fs2: Destination filesystem, or a tuple of (, ) :param create_destination: If True, the destination will be created if it doesn't exist :param ignore_errors: If True, exceptions from file moves are ignored :param chunk_size: Size of chunks to move if a simple copy is used """ if not isinstance(fs1, tuple): raise ValueError("first argument must be a tuple of (, )") fs1, dir1 = fs1 parent_fs1 = fs1 parent_dir1 = dir1 fs1 = fs1.opendir(dir1) if parent_dir1 in ('', '/'): raise RemoveRootError(dir1) if isinstance(fs2, tuple): fs2, dir2 = fs2 if create_destination: fs2.makedir(dir2, allow_recreate=True, recursive=True) fs2 = fs2.opendir(dir2) mount_fs = MountFS(auto_close=False) mount_fs.mount('src', fs1) mount_fs.mount('dst', fs2) mount_fs.copydir('src', 'dst', overwrite=True, ignore_errors=ignore_errors, chunk_size=chunk_size) parent_fs1.removedir(parent_dir1, force=True) def copydir(fs1, fs2, create_destination=True, ignore_errors=False, chunk_size=64*1024): """Copies contents of a directory from one filesystem to another. :param fs1: Source filesystem, or a tuple of (, ) :param fs2: Destination filesystem, or a tuple of (, ) :param create_destination: If True, the destination will be created if it doesn't exist :param ignore_errors: If True, exceptions from file moves are ignored :param chunk_size: Size of chunks to move if a simple copy is used """ if isinstance(fs1, tuple): fs1, dir1 = fs1 fs1 = fs1.opendir(dir1) if isinstance(fs2, tuple): fs2, dir2 = fs2 if create_destination: fs2.makedir(dir2, allow_recreate=True, recursive=True) fs2 = fs2.opendir(dir2) mount_fs = MountFS(auto_close=False) mount_fs.mount('src', fs1) mount_fs.mount('dst', fs2) mount_fs.copydir('src', 'dst', overwrite=True, ignore_errors=ignore_errors, chunk_size=chunk_size) def copydir_progress(progress_callback, fs1, fs2, create_destination=True, ignore_errors=False, chunk_size=64*1024): """ Copies the contents of a directory from one fs to another, with a callback function to display progress. `progress_callback` should be a function with two parameters; `step` and `num_steps`. `num_steps` is the number of steps in the copy process, and `step` is the current step. `num_steps` may be None if the number of steps is still being calculated. """ if isinstance(fs1, tuple): fs1, dir1 = fs1 fs1 = fs1.opendir(dir1) if isinstance(fs2, tuple): fs2, dir2 = fs2 if create_destination: fs2.makedir(dir2, allow_recreate=True, recursive=True) fs2 = fs2.opendir(dir2) def do_callback(step, num_steps): try: progress_callback(step, num_steps) except: pass do_callback(0, None) file_count = 0 copy_paths = [] for dir_path, file_paths in fs1.walk(): copy_paths.append((dir_path, file_paths)) file_count += len(file_paths) do_callback(0, file_count) step = 0 for i, (dir_path, file_paths) in enumerate(copy_paths): try: fs2.makedir(dir_path, allow_recreate=True) for path in file_paths: copy_path = pathjoin(dir_path, path) with fs1.open(copy_path, 'rb') as src_file: fs2.setcontents(copy_path, src_file, chunk_size=chunk_size) step += 1 except: if ignore_errors: continue raise do_callback(step, file_count) def remove_all(fs, path): """Remove everything in a directory. Returns True if successful. :param fs: A filesystem :param path: Path to a directory """ sub_fs = fs.opendir(path) for sub_path in sub_fs.listdir(): if sub_fs.isdir(sub_path): sub_fs.removedir(sub_path, force=True) else: sub_fs.remove(sub_path) return fs.isdirempty(path) def copystructure(src_fs, dst_fs): """Copies the directory structure from one filesystem to another, so that all directories in `src_fs` will have a corresponding directory in `dst_fs` :param src_fs: Filesystem to copy structure from :param dst_fs: Filesystem to copy structure to """ for path in src_fs.walkdirs(): dst_fs.makedir(path, allow_recreate=True) def countbytes(fs): """Returns the total number of bytes contained within files in a filesystem. :param fs: A filesystem object """ total = sum(fs.getsize(f) for f in fs.walkfiles()) return total def isdir(fs,path,info=None): """Check whether a path within a filesystem is a directory. If you're able to provide the info dict for the path, this may be possible without querying the filesystem (e.g. by checking st_mode). """ if info is not None: st_mode = info.get("st_mode") if st_mode: if stat.S_ISDIR(st_mode): return True if stat.S_ISREG(st_mode): return False return fs.isdir(path) def isfile(fs,path,info=None): """Check whether a path within a filesystem is a file. If you're able to provide the info dict for the path, this may be possible without querying the filesystem (e.g. by checking st_mode). """ if info is not None: st_mode = info.get("st_mode") if st_mode: if stat.S_ISREG(st_mode): return True if stat.S_ISDIR(st_mode): return False return fs.isfile(path) def contains_files(fs, path='/'): """Check if there are any files in the filesystem""" try: iter(fs.walkfiles(path)).next() except StopIteration: return False return True def find_duplicates(fs, compare_paths=None, quick=False, signature_chunk_size=16*1024, signature_size=10*16*1024): """A generator that yields the paths of duplicate files in an FS object. Files are considered identical if the contents are the same (dates or other attributes not take in to account). :param fs: A filesystem object :param compare_paths: An iterable of paths within the FS object, or all files if omitted :param quick: If set to True, the quick method of finding duplicates will be used, which can potentially return false positives if the files have the same size and start with the same data. Do not use when deleting files! :param signature_chunk_size: The number of bytes to read before generating a signature checksum value :param signature_size: The total number of bytes read to generate a signature For example, the following will list all the duplicate .jpg files in "~/Pictures":: >>> from fs.utils import find_duplicates >>> from fs.osfs import OSFS >>> fs = OSFS('~/Pictures') >>> for dups in find_duplicates(fs, fs.walkfiles('*.jpg')): ... print list(dups) """ from collections import defaultdict from zlib import crc32 if compare_paths is None: compare_paths = fs.walkfiles() # Create a dictionary that maps file sizes on to the paths of files with # that filesize. So we can find files of the same size with a quick lookup file_sizes = defaultdict(list) for path in compare_paths: file_sizes[fs.getsize(path)].append(path) size_duplicates = [paths for paths in file_sizes.itervalues() if len(paths) > 1] signatures = defaultdict(list) # A signature is a tuple of CRC32s for each 4x16K of the file # This allows us to find potential duplicates with a dictionary lookup for paths in size_duplicates: for path in paths: signature = [] fread = None bytes_read = 0 try: fread = fs.open(path, 'rb') while signature_size is None or bytes_read < signature_size: data = fread.read(signature_chunk_size) if not data: break bytes_read += len(data) signature.append(crc32(data)) finally: if fread is not None: fread.close() signatures[tuple(signature)].append(path) # If 'quick' is True then the signature comparison is adequate (although # it may result in false positives) if quick: for paths in signatures.itervalues(): if len(paths) > 1: yield paths return def identical(p1, p2): """ Returns True if the contents of two files are identical. """ f1, f2 = None, None try: f1 = fs.open(p1, 'rb') f2 = fs.open(p2, 'rb') while True: chunk1 = f1.read(16384) if not chunk1: break chunk2 = f2.read(16384) if chunk1 != chunk2: return False return True finally: if f1 is not None: f1.close() if f2 is not None: f2.close() # If we want to be accurate then we need to compare suspected duplicates # byte by byte. # All path groups in this loop have the same size and same signature, so are # highly likely to be identical. for paths in signatures.itervalues(): while len(paths) > 1: test_p = paths.pop() dups = [test_p] for path in paths: if identical(test_p, path): dups.append(path) if len(dups) > 1: yield dups paths = list(set(paths).difference(dups)) def print_fs(fs, path='/', max_levels=5, file_out=None, terminal_colors=None, hide_dotfiles=False, dirs_first=False, files_wildcard=None, dirs_only=False): """Prints a filesystem listing to stdout (including sub directories). This mostly useful as a debugging aid. Be careful about printing a OSFS, or any other large filesystem. Without max_levels set, this function will traverse the entire directory tree. For example, the following will print a tree of the files under the current working directory:: >>> from fs.osfs import * >>> from fs.utils import * >>> fs = OSFS('.') >>> print_fs(fs) :param fs: A filesystem object :param path: Path of a directory to list (default "/") :param max_levels: Maximum levels of dirs to list (default 5), set to None for no maximum :param file_out: File object to write output to (defaults to sys.stdout) :param terminal_colors: If True, terminal color codes will be written, set to False for non-console output. The default (None) will select an appropriate setting for the platform. :param hide_dotfiles: if True, files or directories beginning with '.' will be removed """ if file_out is None: file_out = sys.stdout file_encoding = getattr(file_out, 'encoding', u'utf-8') or u'utf-8' file_encoding = file_encoding.upper() if terminal_colors is None: if sys.platform.startswith('win'): terminal_colors = False else: terminal_colors = hasattr(file_out, 'isatty') and file_out.isatty() def write(line): if PY3: file_out.write((line + u'\n')) else: file_out.write((line + u'\n').encode(file_encoding, 'replace')) def wrap_prefix(prefix): if not terminal_colors: return prefix return u'\x1b[32m%s\x1b[0m' % prefix def wrap_dirname(dirname): if not terminal_colors: return dirname return u'\x1b[1;34m%s\x1b[0m' % dirname def wrap_error(msg): if not terminal_colors: return msg return u'\x1b[31m%s\x1b[0m' % msg def wrap_filename(fname): if not terminal_colors: return fname if fname.startswith(u'.'): fname = u'\x1b[33m%s\x1b[0m' % fname return fname dircount = [0] filecount = [0] def print_dir(fs, path, levels=[]): if file_encoding == 'UTF-8' and terminal_colors: char_vertline = u'│' char_newnode = u'├' char_line = u'──' char_corner = u'╰' else: char_vertline = u'|' char_newnode = u'|' char_line = u'--' char_corner = u'`' try: dirs = fs.listdir(path, dirs_only=True) if dirs_only: files = [] else: files = fs.listdir(path, files_only=True, wildcard=files_wildcard) dir_listing = ( [(True, p) for p in dirs] + [(False, p) for p in files] ) except Exception, e: prefix = ''.join([(char_vertline + ' ', ' ')[last] for last in levels]) + ' ' write(wrap_prefix(prefix[:-1] + ' ') + wrap_error(u"unable to retrieve directory list (%s) ..." % str(e))) return 0 if hide_dotfiles: dir_listing = [(isdir, p) for isdir, p in dir_listing if not p.startswith('.')] if dirs_first: dir_listing.sort(key = lambda (isdir, p):(not isdir, p.lower())) else: dir_listing.sort(key = lambda (isdir, p):p.lower()) for i, (is_dir, item) in enumerate(dir_listing): if is_dir: dircount[0] += 1 else: filecount[0] += 1 is_last_item = (i == len(dir_listing) - 1) prefix = ''.join([(char_vertline + ' ', ' ')[last] for last in levels]) if is_last_item: prefix += char_corner else: prefix += char_newnode if is_dir: write('%s %s' % (wrap_prefix(prefix + char_line), wrap_dirname(item))) if max_levels is not None and len(levels) + 1 >= max_levels: pass #write(wrap_prefix(prefix[:-1] + ' ') + wrap_error('max recursion levels reached')) else: print_dir(fs, pathjoin(path, item), levels[:] + [is_last_item]) else: write('%s %s' % (wrap_prefix(prefix + char_line), wrap_filename(item))) return len(dir_listing) print_dir(fs, path) return dircount[0], filecount[0] class AtomicWriter(object): """Context manager to perform atomic writes""" def __init__(self, fs, path, mode='w'): self.fs = fs self.path = path self.mode = mode self.tmp_path = path + '~' self._f = None def __enter__(self): self._f = self.fs.open(self.tmp_path, self.mode) return self._f def __exit__(self, exc_type, exc_value, traceback): if exc_type is None: if self._f is not None: if hasattr('_f', 'flush'): self._f.flush() if hasattr(self._f, 'fileno'): os.fsync(self._f.fileno()) self._f.close() self._f = None self.fs.rename(self.tmp_path, self.path) else: if self._f is not None: self._f.close() def open_atomic_write(fs, path, mode='w'): """Open a file for 'atomic' writing This returns a context manager which ensures that a file is written in its entirety or not at all. """ return AtomicWriter(fs, path, mode=mode) if __name__ == "__main__": from fs.tempfs import TempFS from six import b t1 = TempFS() t1.setcontents("foo", b("test")) t1.makedir("bar") t1.setcontents("bar/baz", b("another test")) t1.tree() t2 = TempFS() print t2.listdir() movedir(t1, t2) print t2.listdir() t1.tree() t2.tree() fs-0.5.4/fs/contrib/0000755000000000000000000000000012621617365014147 5ustar rootroot00000000000000fs-0.5.4/fs/contrib/archivefs.py0000664000175000017500000005201712512525115016472 0ustar willwill00000000000000""" fs.contrib.archivefs ======== A FS object that represents the contents of an archive. """ import time import stat import datetime import os.path from fs.base import * from fs.path import * from fs.errors import * from fs.filelike import StringIO from fs import mountfs import libarchive ENCODING = libarchive.ENCODING class SizeUpdater(object): '''A file-like object to allow writing to a file within the archive. When closed this object will update the archive entry's size within the archive.''' def __init__(self, entry, stream): self.entry = entry self.stream = stream self.size = 0 def __del__(self): self.close() def write(self, data): self.size += len(data) self.stream.write(data) def close(self): self.stream.close() self.entry.size = self.size class ArchiveFS(FS): """A FileSystem that represents an archive supported by libarchive.""" _meta = { 'thread_safe' : True, 'virtual' : False, 'read_only' : False, 'unicode_paths' : True, 'case_insensitive_paths' : False, 'network' : False, 'atomic.setcontents' : False } def __init__(self, f, mode='r', format=None, thread_synchronize=True): """Create a FS that maps on to an archive file. :param f: a (system) path, or a file-like object :param format: required for 'w' mode. The archive format ('zip, 'tar', etc) :param thread_synchronize: set to True (default) to enable thread-safety """ super(ArchiveFS, self).__init__(thread_synchronize=thread_synchronize) if isinstance(f, basestring): self.fileobj = None self.root_path = f else: self.fileobj = f self.root_path = getattr(f, 'name', None) self.contents = PathMap() self.archive = libarchive.SeekableArchive(f, format=format, mode=mode) if 'r' in mode: for item in self.archive: for part in recursepath(item.pathname)[1:]: part = relpath(part) if part == item.pathname: self.contents[part] = item else: self.contents[part] = libarchive.Entry(pathname=part, mode=stat.S_IFDIR, size=0, mtime=item.mtime) def __str__(self): return "" % self.root_path def __unicode__(self): return u"" % self.root_path def getmeta(self, meta_name, default=NoDefaultMeta): if meta_name == 'read_only': return self.read_only return super(ArchiveFS, self).getmeta(meta_name, default) @synchronize def close(self): if getattr(self, 'archive', None) is None: return self.archive.close() @synchronize def open(self, path, mode="r", **kwargs): path = normpath(relpath(path)) if path == '': # We need to open the archive itself, not one of it's entries. return file(self.root_path, mode) if 'a' in mode: raise Exception('Unsupported mode ' + mode) if 'r' in mode: return self.archive.readstream(path) else: entry = self.archive.entry_class(pathname=path, mode=stat.S_IFREG, size=0, mtime=time.time()) self.contents[path] = entry return SizeUpdater(entry, self.archive.writestream(path)) @synchronize def getcontents(self, path, mode="rb", encoding=None, errors=None, newline=None): if not self.exists(path): raise ResourceNotFoundError(path) with self.open(path, mode, encoding=encoding, errors=errors, newline=newline) as f: return f.read() def desc(self, path): return "%s in zip file" % path def getsyspath(self, path, allow_none=False): path = normpath(path).lstrip('/') return join(self.root_path, path) def isdir(self, path): info = self.getinfo(path) # Don't use stat.S_ISDIR, it won't work when mode == S_IFREG | S_IFDIR. return info.get('st_mode', 0) & stat.S_IFDIR == stat.S_IFDIR def isfile(self, path): info = self.getinfo(path) # Don't use stat.S_ISREG, it won't work when mode == S_IFREG | S_IFDIR. return info.get('st_mode', 0) & stat.S_IFREG == stat.S_IFREG def exists(self, path): path = normpath(path).lstrip('/') if path == '': # We are being asked about root (the archive itself) return True return path in self.contents def listdir(self, path="/", wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False): return self._listdir_helper(path, self.contents.names(path), wildcard, full, absolute, dirs_only, files_only) def makedir(self, dirname, recursive=False, allow_recreate=False): entry = self.archive.entry_class(pathname=dirname, mode=stat.S_IFDIR, size=0, mtime=time.time()) self.contents[dirname] = entry self.archive.write(entry) @synchronize def getinfo(self, path): if not self.exists(path): raise ResourceNotFoundError(path) path = normpath(path).lstrip('/') info = { 'size': 0 } entry = self.contents.get(path) for attr in dir(entry): if attr.startswith('_'): continue elif attr == 'mtime': info['modified_time'] = datetime.datetime.fromtimestamp(entry.mtime) elif attr == 'mode': info['st_mode'] = entry.mode else: value = getattr(entry, attr) if callable(value): continue info[attr] = value return info def getsize(self, path): return self.getinfo(path)['st_size'] class ArchiveMountFS(mountfs.MountFS): '''A subclass of MountFS that automatically identifies archives. Once identified archives are mounted in place of the archive file.''' def __init__(self, rootfs, auto_close=True, auto_mount=True, max_size=None): self.auto_mount = auto_mount self.max_size = max_size super(ArchiveMountFS, self).__init__(auto_close=auto_close) self.rootfs = rootfs self.mountdir('/', rootfs) @synchronize def close(self): # Close and delete references to any other fs instances. if self.rootfs is not None: self.rootfs.close() self.rootfs = None super(ArchiveMountFS, self).close() def ismount(self, path): "Checks if the given path has a file system mounted on it." try: object = self.mount_tree[path] except KeyError: return False return isinstance(object, mountfs.MountFS.DirMount) def unmount(self, path): """Unmounts a path. :param path: Path to unmount """ # This might raise a KeyError, but that is what MountFS will do, so # shall we. fs = self.mount_tree.pop(path) # TODO: it may be necessary to remember what paths were auto-mounted, # so we can close those here. It may not be safe to close a file system # that the user provided. However, it is definitely NOT safe to leave # one open. if callable(getattr(fs, 'close', None)): fs.close() def _delegate(self, path, auto_mount=True): """A _delegate() override that will automatically mount archives that are encountered in the path. For example, the path /foo/bar.zip/baz.txt contains the archive path /foo/bar.zip. If this archive can be mounted by ArchiveFS, it will be. Then the file system call will be delegated to that mounted file system, which will act upon /baz.txt within the archive. This is lazy initialization which means users of this class need not crawl the file system for archives and mount them all up-front. This behavior can be overridden by self.auto_mount=False or by passing the auto_mount=False keyword argument. """ if self.auto_mount and auto_mount: for ppath in recursepath(path)[1:]: if self.ismount(ppath): # If something is already mounted here, no need to continue. break if libarchive.is_archive_name(ppath): # It looks like an archive, we might mount it. # First check that the size is acceptable. if self.max_size: if self.rootfs.exists(ppath) and \ self.rootfs.getsize(ppath) > self.max_size: break # Looks good, the proof is in the pudding, so let's try to # mount this *supposed* archive... full_path = self.rootfs.getsyspath(ppath) try: # TODO: it would be really nice if we could open the path using # self.rootfs.open(), that way we could support archives on a file # system other than osfs (even nested archives). However, the libarchive # wrapper is not sophisticated enough to handle a Python file-like object, # it uses an actual fd. self.mountdir(ppath, ArchiveFS(full_path, 'r')) # That worked!! Stop recursing path, we support just one archive per path! break except: # Must NOT have been an archive after all, but maybe # there is one deeper in the directory... continue return super(ArchiveMountFS, self)._delegate(path) def getsyspath(self, path, allow_none=False): """A getsyspath() override that returns paths relative to the root fs.""" root = self.rootfs.getsyspath('/', allow_none=allow_none) if root: return join(root, path.lstrip('/')) def open(self, path, *args, **kwargs): """An open() override that opens an archive. It is not fooled by mounted archives. If the path is a mounted archive, it is unmounted and the archive file is opened and returned.""" if libarchive.is_archive_name(path) and self.ismount(path): self.unmount(path) fs, _mount_path, delegate_path = self._delegate(path, auto_mount=False) return fs.open(delegate_path, *args, **kwargs) def getinfo(self, path): """A getinfo() override that allows archives to masqueraded as directories. If the path is not an archive, the call is delegated. In the event that the path is an archive, that archive is mounted to ensure it can actually be treaded like a directory.""" fs, _mount_path, delegate_path = self._delegate(path) if isinstance(fs, ArchiveFS) and path == _mount_path: info = self.rootfs.getinfo(path) info['st_mode'] = info.get('st_mode', 0) | stat.S_IFDIR return info return super(ArchiveMountFS, self).getinfo(path) def isdir(self, path): """An isdir() override that allows archives to masquerade as directories. If the path is not an archive, the call is delegated. In the event that the path is an archive, that archive is mounted to ensure it can actually be treated like a directory.""" fs, _mount_path, delegate_path = self._delegate(path) if isinstance(fs, ArchiveFS) and path == _mount_path: # If the path is an archive mount point, it is a directory. return True return super(ArchiveMountFS, self).isdir(path) def isfile(self, path): """An isfile() override that checks if the given path is a file or not. It is not fooled by a mounted archive. If the path is not an archive, True is returned. If the path is not an archive, the call is delegated.""" fs, _mount_path, delegate_path = self._delegate(path) if isinstance(fs, ArchiveFS) and path == _mount_path: # If the path is an archive mount point, it is a file. return True else: return fs.isfile(delegate_path) def getsize(self, path): """A getsize() override that returns the size of an archive. It is not fooled by a mounted archive. If the path is not an archive, the call is delegated.""" fs, _mount_path, delegate_path = self._delegate(path, auto_mount=False) if isinstance(fs, ArchiveFS) and path == _mount_path: return self.rootfs.getsize(path) else: return fs.getsize(delegate_path) def remove(self, path): """A remove() override that deletes an archive directly. It is not fooled by a mounted archive. If the path is not an archive, the call is delegated.""" if libarchive.is_archive_name(path) and self.ismount(path): self.unmount(path) fs, _mount_path, delegate_path = self._delegate(path, auto_mount=False) return fs.remove(delegate_path) def makedir(self, path, *args, **kwargs): """A makedir() override that handles creation of a directory at an archive location properly. If the path is not an archive, the call is delegated.""" # If the caller is trying to create a directory where an archive lives # we should raise an error. In the case when allow_recreate=True, this # call would succeed without the check below. fs, _mount_path, delegate_path = self._delegate(path, auto_mount=False) if isinstance(fs, ArchiveFS) and path == _mount_path: raise ResourceInvalidError(path, msg="Cannot create directory, there's " "already a file of that name: %(path)s") return fs.makedir(delegate_path, *args, **kwargs) def copy(self, src, dst, overwrite=False, chunk_size=1024*64): """An optimized copy() that will skip mounting an archive if one is involved as either the src or dst. This allows the file containing the archive to be copied.""" src_is_archive = libarchive.is_archive_name(src) # If src path is a mounted archive, unmount it. if src_is_archive and self.ismount(src): self.unmount(src) # Now delegate the path, if the path is an archive, don't remount it. srcfs, _ignored, src = self._delegate(src, auto_mount=(not src_is_archive)) # Follow the same steps for dst. dst_is_archive = libarchive.is_archive_name(dst) if dst_is_archive and self.ismount(dst): self.unmount(dst) dstfs, _ignored, dst = self._delegate(dst, auto_mount=(not dst_is_archive)) # srcfs, src and dstfs, dst are now the file system and path for our src and dst. if srcfs is dstfs and srcfs is not self: # Both src and dst are on the same fs, let it do the copy. srcfs.copy(src, dst, overwrite=overwrite, chunk_size=chunk_size) else: # Src and dst are on different file systems. Just do the copy... srcfd = None try: srcfd = srcfs.open(src, 'rb') dstfs.setcontents(dst, srcfd, chunk_size=chunk_size) except ResourceNotFoundError: if srcfs.exists(src) and not dstfs.exists(dirname(dst)): raise ParentDirectoryMissingError(dst) finally: if srcfd: srcfd.close() def move(self, src, dst, overwrite=False, chunk_size=1024*64): """An optimized move() that delegates the work to the overridden copy() and remove() methods.""" self.copy(src, dst, overwrite=overwrite, chunk_size=chunk_size) self.remove(src) def rename(self, src, dst): """An rename() implementation that ensures the rename does not span file systems. It also ensures that an archive can be renamed (without trying to mount either the src or destination paths).""" src_is_archive = libarchive.is_archive_name(src) # If src path is a mounted archive, unmount it. if src_is_archive and self.ismount(src): self.unmount(src) # Now delegate the path, if the path is an archive, don't remount it. srcfs, _ignored, src = self._delegate(src, auto_mount=(not src_is_archive)) # Follow the same steps for dst. dst_is_archive = libarchive.is_archive_name(dst) if dst_is_archive and self.ismount(dst): self.unmount(dst) dstfs, _ignored, dst = self._delegate(dst, auto_mount=(not dst_is_archive)) # srcfs, src and dstfs, dst are now the file system and path for our src and dst. if srcfs is dstfs and srcfs is not self: # Both src and dst are on the same fs, let it do the copy. return srcfs.rename(src, dst) raise OperationFailedError("rename resource", path=src) def walk(self, path="/", wildcard=None, dir_wildcard=None, search="breadth", ignore_errors=False, archives_as_files=True): """Walks a directory tree and yields the root path and contents. Yields a tuple of the path of each directory and a list of its file contents. :param path: root path to start walking :type path: string :param wildcard: if given, only return files that match this wildcard :type wildcard: a string containing a wildcard (e.g. `*.txt`) or a callable that takes the file path and returns a boolean :param dir_wildcard: if given, only walk directories that match the wildcard :type dir_wildcard: a string containing a wildcard (e.g. `*.txt`) or a callable that takes the directory name and returns a boolean :param search: a string identifying the method used to walk the directories. There are two such methods: * ``"breadth"`` yields paths in the top directories first * ``"depth"`` yields the deepest paths first :param ignore_errors: ignore any errors reading the directory :type ignore_errors: bool :param archives_as_files: treats archives as files rather than directories. :type ignore_errors: bool :rtype: iterator of (current_path, paths) """ path = normpath(path) def isdir(path): if not self.isfile(path): return True if not archives_as_files and self.ismount(path): return True return False def listdir(path, *args, **kwargs): dirs_only = kwargs.pop('dirs_only', False) if ignore_errors: try: listing = self.listdir(path, *args, **kwargs) except: return [] else: listing = self.listdir(path, *args, **kwargs) if dirs_only: listing = filter(isdir, listing) return listing if wildcard is None: wildcard = lambda f:True elif not callable(wildcard): wildcard_re = re.compile(fnmatch.translate(wildcard)) wildcard = lambda fn:bool (wildcard_re.match(fn)) if dir_wildcard is None: dir_wildcard = lambda f:True elif not callable(dir_wildcard): dir_wildcard_re = re.compile(fnmatch.translate(dir_wildcard)) dir_wildcard = lambda fn:bool (dir_wildcard_re.match(fn)) if search == "breadth": dirs = [path] dirs_append = dirs.append dirs_pop = dirs.pop while dirs: current_path = dirs_pop() paths = [] paths_append = paths.append try: for filename in listdir(current_path): path = pathcombine(current_path, filename) if isdir(path): if dir_wildcard(path): dirs_append(path) else: if wildcard(filename): paths_append(filename) except ResourceNotFoundError: # Could happen if another thread / process deletes something whilst we are walking pass yield (current_path, paths) elif search == "depth": def recurse(recurse_path): try: for path in listdir(recurse_path, wildcard=dir_wildcard, full=True, dirs_only=True): for p in recurse(path): yield p except ResourceNotFoundError: # Could happen if another thread / process deletes something whilst we are walking pass yield (recurse_path, listdir(recurse_path, wildcard=wildcard, files_only=True)) for p in recurse(path): yield p else: raise ValueError("Search should be 'breadth' or 'depth'") def main(): ArchiveFS() if __name__ == '__main__': main() fs-0.5.4/fs/contrib/bigfs/0000755000000000000000000000000012621617365015241 5ustar rootroot00000000000000fs-0.5.4/fs/contrib/bigfs/subrangefile.py0000664000175000017500000000475212512525115020263 0ustar willwill00000000000000""" fs.contrib.bigfs.subrangefile ============================= A file-like object that allows wrapping of part of a binary file for reading. This avoids needless copies of data for large binary files if StringIO would be used. Written by Koen van de Sande http://www.tibed.net Contributed under the terms of the BSD License: http://www.opensource.org/licenses/bsd-license.php """ class SubrangeFile: """File-like class with read-only, binary mode restricting access to a subrange of the whole file""" def __init__(self, f, startOffset, fileSize): if not hasattr(f, 'read'): self.f = open(f, "rb") self.name = f else: self.f = f self.name = str(f) self.startOffset = startOffset self.fileSize = fileSize self.seek(0) def __str__(self): return "" % (self.name, self.startOffset, self.fileSize) def __unicode__(self): return unicode(self.__str__()) def size(self): return self.fileSize def seek(self, offset, whence=0): if whence == 0: offset = self.startOffset + offset elif whence == 1: offset = self.startOffset + self.tell() + offset elif whence == 2: if offset > 0: offset = 0 offset = self.startOffset + self.fileSize + offset self.f.seek(offset) def tell(self): return self.f.tell() - self.startOffset def __maxSize(self,size=None): iSize = self.fileSize if not size is None: if size < iSize: iSize = size if self.tell() + iSize > self.fileSize: iSize = self.fileSize - self.tell() return iSize def readline(self,size=None): toRead = self.__maxSize(size) return self.f.readline(toRead) def read(self,size=None): toRead = self.__maxSize(size) return self.f.read(toRead) def readlines(self,size=None): toRead = self.__maxSize(size) temp = self.f.readlines(toRead) # now cut off more than we should read... result = [] counter = 0 for line in temp: if counter + len(line) > toRead: if toRead == counter: break result.append(line[0:(toRead-counter)]) break else: result.append(line) counter += len(line) return result fs-0.5.4/fs/contrib/bigfs/__init__.py0000664000175000017500000003035512512525115017352 0ustar willwill00000000000000""" fs.contrib.bigfs ================ A FS object that represents the contents of a BIG file (C&C Generals, BfME C&C3, C&C Red Alert 3, C&C4 file format) Written by Koen van de Sande http://www.tibed.net Contributed under the terms of the BSD License: http://www.opensource.org/licenses/bsd-license.php """ from struct import pack, unpack from fs.base import * from fs.memoryfs import MemoryFS from fs.filelike import StringIO from fs.contrib.bigfs.subrangefile import SubrangeFile class BIGEntry: def __init__(self, filename, offset, storedSize, isCompressed, realSize): self.filename = filename self.offset = offset self.storedSize = storedSize self.realSize = realSize self.isCompressed = isCompressed def getfile(self, baseFile): f = SubrangeFile(baseFile, self.offset, self.storedSize) if not self.isCompressed: return f else: return self.decompress(f, wrapAsFile=True) def getcontents(self, baseFile): f = SubrangeFile(baseFile, self.offset, self.storedSize) if not self.isCompressed: return f.read() else: return self.decompress(f, wrapAsFile=False) def decompress(self, g, wrapAsFile=True): buf = g.read(2) magic = unpack(">H", buf)[0] if (magic & 0x3EFF) == 0x10FB: # it is compressed if magic & 0x8000: outputSize = unpack(">I", g.read(4))[0] if magic & 0x100: unknown1 = unpack(">I", g.read(4))[0] else: outputSize = unpack(">I", "\0" + g.read(3))[0] if magic & 0x100: unknown1 = unpack(">I", "\0" + g.read(3))[0] output = [] while True: opcode = unpack("B", g.read(1))[0] if not (opcode & 0x80): # opcode: bit7==0 to get here # read second opcode opcode2 = unpack("B", g.read(1))[0] #print "0x80", toBits(opcode), toBits(opcode2), opcode & 0x03, (((opcode & 0x60) << 3) | opcode2) + Q, ((opcode & 0x1C) >> 2) + 2 + R # copy at most 3 bytes to output stream (lowest 2 bits of opcode) count = opcode & 0x03 for i in range(count): output.append(g.read(1)) # you always have to look at least one byte, hence the +1 # use bit6 and bit5 (bit7=0 to trigger the if-statement) of opcode, and 8 bits of opcode2 (10-bits) lookback = (((opcode & 0x60) << 3) | opcode2) + 1 # use bit4..2 of opcode count = ((opcode & 0x1C) >> 2) + 3 for i in range(count): output.append(output[-lookback]) elif not (opcode & 0x40): # opcode: bit7..6==10 to get here opcode2 = unpack("B", g.read(1))[0] opcode3 = unpack("B", g.read(1))[0] #print "0x40", toBits(opcode), toBits(opcode2), toBits(opcode3) # copy count bytes (upper 2 bits of opcode2) count = opcode2 >> 6 for i in range(count): output.append(g.read(1)) # look back again (lower 6 bits of opcode2, all 8 bits of opcode3, total 14-bits) lookback = (((opcode2 & 0x3F) << 8) | opcode3) + 1 # lower 6 bits of opcode are the count to copy count = (opcode & 0x3F) + 4 for i in range(count): output.append(output[-lookback]) elif not (opcode & 0x20): # opcode: bit7..5=110 to get here opcode2 = unpack("B", g.read(1))[0] opcode3 = unpack("B", g.read(1))[0] opcode4 = unpack("B", g.read(1))[0] # copy at most 3 bytes to output stream (lowest 2 bits of opcode) count = opcode & 0x03 for i in range(count): output.append(g.read(1)) # look back: bit4 of opcode, all bits of opcode2 and opcode3, total 17-bits lookback = (((opcode & 0x10) >> 4) << 16) | (opcode2 << 8) | (opcode3) + 1 # bit3..2 of opcode and the whole of opcode4 count = (((((opcode & 0x0C) >> 2) << 8)) | opcode4) + 5 #print "0x20", toBits(opcode), toBits(opcode2), toBits(opcode3), toBits(opcode4), lookback, count for i in range(count): output.append(output[-lookback]) else: # opcode: bit7..5==1 to get here # use lowest 5 bits for count count = ((opcode & 0x1F) << 2) + 4 if count > 0x70: # this is end of input # turn into a small-copy count = opcode & 0x03 #print "0xEXITCOPY", count for i in range(count): output.append(g.read(1)) break # "big copy" operation: up to 112 bytes (minumum of 4, multiple of 4) for i in range(count): output.append(g.read(1)) #print "0xLO", toBits(opcode), count if wrapAsFile: return StringIO("".join(output)) else: return "".join(output) def __str__(self): return " 1 or mode not in "r": raise ValueError("mode must be 'r'") self.file_mode = mode self.big_path = str(filename) self.entries = {} try: self.bf = open(filename, "rb") except IOError: raise ResourceNotFoundError(str(filename), msg="BIG file does not exist: %(path)s") self._path_fs = MemoryFS() if mode in 'ra': self._parse_resource_list(self.bf) def __str__(self): return "" % self.big_path def __unicode__(self): return unicode(self.__str__()) def _parse_resource_list(self, g): magicWord = g.read(4) if magicWord != "BIGF" and magicWord != "BIG4": raise ValueError("Magic word of BIG file invalid: " + filename + " " + repr(magicWord)) header = g.read(12) header = unpack(">III", header) BIGSize = header[0] fileCount = header[1] bodyOffset = header[2] for i in range(fileCount): fileHeader = g.read(8) fileHeader = unpack(">II", fileHeader) pos = g.tell() buf = g.read(4096) marker = buf.find("\0") if marker == -1: raise ValueError("Could not parse filename in BIG file: Too long or invalid file") name = buf[:marker] # TODO: decode the encoding of name (or normalize the path?) isCompressed, uncompressedSize = self.__isCompressed(g, fileHeader[0], fileHeader[1]) be = BIGEntry(name, fileHeader[0], fileHeader[1], isCompressed, uncompressedSize) name = normpath(name) self.entries[name] = be self._add_resource(name) g.seek(pos + marker + 1) def __isCompressed(self, g, offset, size): g.seek(offset) buf = g.read(2) magic = unpack(">H", buf)[0] if (magic & 0x3EFF) == 0x10FB: # it is compressed if magic & 0x8000: # decompressed size is uint32 return True, unpack(">I", g.read(4))[0] else: # use only 3 bytes return True, unpack(">I", "\0" + g.read(3))[0] return False, size def _add_resource(self, path): if path.endswith('/'): path = path[:-1] if path: self._path_fs.makedir(path, recursive=True, allow_recreate=True) else: dirpath, filename = pathsplit(path) if dirpath: self._path_fs.makedir(dirpath, recursive=True, allow_recreate=True) f = self._path_fs.open(path, 'w') f.close() def close(self): """Finalizes the zip file so that it can be read. No further operations will work after this method is called.""" if hasattr(self, 'bf') and self.bf: self.bf.close() self.bf = _ExceptionProxy() @synchronize def open(self, path, mode="r", **kwargs): path = normpath(relpath(path)) if 'r' in mode: if self.file_mode not in 'ra': raise OperationFailedError("open file", path=path, msg="Big file must be opened for reading ('r') or appending ('a')") try: return self.entries[path].getfile(self.bf) except KeyError: raise ResourceNotFoundError(path) if 'w' in mode: raise OperationFailedError("open file", path=path, msg="Big file cannot be edited ATM") raise ValueError("Mode must contain be 'r' or 'w'") @synchronize def getcontents(self, path): if not self.exists(path): raise ResourceNotFoundError(path) path = normpath(path) try: contents = self.entries[path].getcontents(self.bf) except KeyError: raise ResourceNotFoundError(path) except RuntimeError: raise OperationFailedError("read file", path=path, msg="Big file must be oppened with 'r' or 'a' to read") return contents def desc(self, path): if self.isdir(path): return "Dir in big file: %s" % self.big_path else: return "File in big file: %s" % self.big_path def isdir(self, path): return self._path_fs.isdir(path) def isfile(self, path): return self._path_fs.isfile(path) def exists(self, path): return self._path_fs.exists(path) @synchronize def makedir(self, dirname, recursive=False, allow_recreate=False): dirname = normpath(dirname) if self.file_mode not in "wa": raise OperationFailedError("create directory", path=dirname, msg="Big file must be opened for writing ('w') or appending ('a')") if not dirname.endswith('/'): dirname += '/' self._add_resource(dirname) def listdir(self, path="/", wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False): return self._path_fs.listdir(path, wildcard, full, absolute, dirs_only, files_only) @synchronize def getinfo(self, path): if not self.exists(path): raise ResourceNotFoundError(path) path = normpath(path).lstrip('/') info = {'size': 0} if path in self.entries: be = self.entries[path] info['size'] = be.realSize info['file_size'] = be.realSize info['stored_size'] = be.storedSize info['is_compressed'] = be.isCompressed info['offset'] = be.offset info['internal_filename'] = be.filename info['filename'] = path return info fs-0.5.4/fs/contrib/davfs/0000755000000000000000000000000012621617365015252 5ustar rootroot00000000000000fs-0.5.4/fs/contrib/davfs/xmlobj.py0000664000175000017500000001467612621464031017127 0ustar willwill00000000000000# Copyright (c) 2009-2010, Cloud Matrix Pty. Ltd. # All rights reserved; available under the terms of the MIT License. """ fs.contrib.davfs.xmlobj: dexml model definitions for WebDAV This module defines the various XML element structures for WebDAV as a set of dexml.Model subclasses. """ from urlparse import urlparse, urlunparse from httplib import responses as STATUS_CODE_TEXT STATUS_CODE_TEXT[207] = "Multi-Status" import dexml from dexml import fields Error = dexml.Error class _davbase(dexml.Model): """Base class for all davfs XML models.""" class meta: namespace = "DAV:" namespace_prefix = "D" order_sensitive = False class HrefField(fields.String): """Field representing a tag.""" def __init__(self,*args,**kwds): kwds["tagname"] = "href" super(HrefField,self).__init__(*args,**kwds) def parse_value(self,value): url = urlparse(value.encode("UTF-8")) return urlunparse((url.scheme,url.netloc,url.path,url.params,url.query,url.fragment)) def render_value(self,value): url = urlparse(value.encode("UTF-8")) return urlunparse((url.scheme,url.netloc,url.path,url.params,url.query,url.fragment)) class TimeoutField(fields.Field): """Field representing a WebDAV timeout value.""" def __init__(self,*args,**kwds): if "tagname" not in kwds: kwds["tagname"] = "timeout" super(TimeoutField,self).__init__(*args,**kwds) @classmethod def parse_value(cls,value): if value == "Infinite": return None if value.startswith("Second-"): return int(value[len("Second-"):]) raise ValueError("invalid timeout specifier: %s" % (value,)) def render_value(self,value): if value is None: return "Infinite" else: return "Second-" + str(value) class StatusField(fields.Value): """Field representing a WebDAV status-line value. The value may be set as either a string or an integer, and is converted into a StatusString instance. """ def __init__(self,*args,**kwds): kwds["tagname"] = "status" super(StatusField,self).__init__(*args,**kwds) def __get__(self,instance,owner): val = super(StatusField,self).__get__(instance,owner) if val is not None: val = StatusString(val,instance,self) return val def __set__(self,instance,value): if isinstance(value,basestring): # sanity check it bits = value.split(" ") if len(bits) < 3 or bits[0] != "HTTP/1.1": raise ValueError("Not a valid status: '%s'" % (value,)) int(bits[1]) elif isinstance(value,int): # convert it to a message value = StatusString._value_for_code(value) super(StatusField,self).__set__(instance,value) class StatusString(str): """Special string representing a HTTP status line. It's a string, but it exposes the integer attribute "code" giving just the actual response code. """ def __new__(cls,val,inst,owner): return str.__new__(cls,val) def __init__(self,val,inst,owner): self._owner = owner self._inst = inst @staticmethod def _value_for_code(code): msg = STATUS_CODE_TEXT.get(code,"UNKNOWN STATUS CODE") return "HTTP/1.1 %d %s" % (code,msg) def _get_code(self): return int(self.split(" ")[1]) def _set_code(self,code): newval = self._value_for_code(code) self._owner.__set__(self._inst,newval) code = property(_get_code,_set_code) class multistatus(_davbase): """XML model for a multi-status response message.""" responses = fields.List("response",minlength=1) description = fields.String(tagname="responsedescription",required=False) class response(_davbase): """XML model for an individual response in a multi-status message.""" href = HrefField() # TODO: ensure only one of hrefs/propstats hrefs = fields.List(HrefField(),required=False) status = StatusField(required=False) propstats = fields.List("propstat",required=False) description = fields.String(tagname="responsedescription",required=False) class propstat(_davbase): """XML model for a propstat response message.""" props = fields.XmlNode(tagname="prop",encoding="UTF-8") status = StatusField() description = fields.String(tagname="responsedescription",required=False) class propfind(_davbase): """XML model for a propfind request message.""" allprop = fields.Boolean(tagname="allprop",required=False) propname = fields.Boolean(tagname="propname",required=False) prop = fields.XmlNode(tagname="prop",required=False,encoding="UTF-8") class propertyupdate(_davbase): """XML model for a propertyupdate request message.""" commands = fields.List(fields.Choice("remove","set")) class remove(_davbase): """XML model for a propertyupdate remove command.""" props = fields.XmlNode(tagname="prop",encoding="UTF-8") class set(_davbase): """XML model for a propertyupdate set command.""" props = fields.XmlNode(tagname="prop",encoding="UTF-8") class lockdiscovery(_davbase): """XML model for a lockdiscovery request message.""" locks = fields.List("activelock") class activelock(_davbase): """XML model for an activelock response message.""" lockscope = fields.Model("lockscope") locktype = fields.Model("locktype") depth = fields.String(tagname="depth") owner = fields.XmlNode(tagname="owner",encoding="UTF-8",required=False) timeout = TimeoutField(required=False) locktoken = fields.Model("locktoken",required=False) class lockscope(_davbase): """XML model for a lockscope response message.""" shared = fields.Boolean(tagname="shared",empty_only=True) exclusive = fields.Boolean(tagname="exclusive",empty_only=True) class locktoken(_davbase): """XML model for a locktoken response message.""" tokens = fields.List(HrefField()) class lockentry(_davbase): """XML model for a lockentry response message.""" lockscope = fields.Model("lockscope") locktype = fields.Model("locktype") class lockinfo(_davbase): """XML model for a lockinfo response message.""" lockscope = fields.Model("lockscope") locktype = fields.Model("locktype") owner = fields.XmlNode(tagname="owner",encoding="UTF-8") class locktype(_davbase): """XML model for a locktype response message.""" type = fields.XmlNode(encoding="UTF-8") fs-0.5.4/fs/contrib/davfs/util.py0000664000175000017500000001330712512525115016577 0ustar willwill00000000000000# Copyright (c) 2009-2010, Cloud Matrix Pty. Ltd. # All rights reserved; available under the terms of the MIT License. """ fs.contrib.davfs.util: utils for FS WebDAV implementation. """ import os import re import cookielib def get_fileno(file): """Get the os-level fileno of a file-like object. This function decodes several common file wrapper structures in an attempt to determine the underlying OS-level fileno for an object. """ while not hasattr(file,"fileno"): if hasattr(file,"file"): file = file.file elif hasattr(file,"_file"): file = file._file elif hasattr(file,"_fileobj"): file = file._fileobj else: raise AttributeError return file.fileno() def get_filesize(file): """Get the "size" attribute of a file-like object.""" while not hasattr(file,"size"): if hasattr(file,"file"): file = file.file elif hasattr(file,"_file"): file = file._file elif hasattr(file,"_fileobj"): file = file._fileobj else: raise AttributeError return file.size def file_chunks(f,chunk_size=1024*64): """Generator yielding chunks of a file. This provides a simple way to iterate through binary data chunks from a file. Recall that using a file directly as an iterator generates the *lines* from that file, which is useless and very inefficient for binary data. """ chunk = f.read(chunk_size) while chunk: yield chunk chunk = f.read(chunk_size) def normalize_req_body(body,chunk_size=1024*64): """Convert given request body into (size,data_iter) pair. This function is used to accept a variety of different inputs in HTTP requests, converting them to a standard format. """ if hasattr(body,"getvalue"): value = body.getvalue() return (len(value),[value]) elif hasattr(body,"read"): try: size = int(get_filesize(body)) except (AttributeError,TypeError): try: size = os.fstat(get_fileno(body)).st_size except (AttributeError,OSError): size = None return (size,file_chunks(body,chunk_size)) else: body = str(body) return (len(body),[body]) class FakeReq: """Compatability interface to use cookielib with raw httplib objects.""" def __init__(self,connection,scheme,path): self.connection = connection self.scheme = scheme self.path = path def get_full_url(self): return self.scheme + "://" + self.connection.host + self.path def get_type(self): return self.scheme def get_host(self): return self.connection.host def is_unverifiable(self): return True def get_origin_req_host(self): return self.connection.host def has_header(self,header): return False def add_unredirected_header(self,header,value): self.connection.putheader(header,value) class FakeResp: """Compatability interface to use cookielib with raw httplib objects.""" def __init__(self,response): self.response = response def info(self): return self def getheaders(self,header): header = header.lower() headers = self.response.getheaders() return [v for (h,v) in headers if h.lower() == header] # The standard cooklielib cookie parser doesn't seem to handle multiple # cookies correctory, so we replace it with a better version. This code # is a tweaked version of the cookielib function of the same name. # _test_cookie = "sessionid=e9c9b002befa93bd865ce155270307ef; Domain=.cloud.me; expires=Wed, 10-Feb-2010 03:27:20 GMT; httponly; Max-Age=1209600; Path=/, sessionid_https=None; Domain=.cloud.me; expires=Wed, 10-Feb-2010 03:27:20 GMT; httponly; Max-Age=1209600; Path=/; secure" if len(cookielib.parse_ns_headers([_test_cookie])) != 2: def parse_ns_headers(ns_headers): """Improved parser for netscape-style cookies. This version can handle multiple cookies in a single header. """ known_attrs = ("expires", "domain", "path", "secure","port", "max-age") result = [] for ns_header in ns_headers: pairs = [] version_set = False for ii, param in enumerate(re.split(r"(;\s)|(,\s(?=[a-zA-Z0-9_\-]+=))", ns_header)): if param is None: continue param = param.rstrip() if param == "" or param[0] == ";": continue if param[0] == ",": if pairs: if not version_set: pairs.append(("version", "0")) result.append(pairs) pairs = [] continue if "=" not in param: k, v = param, None else: k, v = re.split(r"\s*=\s*", param, 1) k = k.lstrip() if ii != 0: lc = k.lower() if lc in known_attrs: k = lc if k == "version": # This is an RFC 2109 cookie. version_set = True if k == "expires": # convert expires date to seconds since epoch if v.startswith('"'): v = v[1:] if v.endswith('"'): v = v[:-1] v = cookielib.http2time(v) # None if invalid pairs.append((k, v)) if pairs: if not version_set: pairs.append(("version", "0")) result.append(pairs) return result cookielib.parse_ns_headers = parse_ns_headers assert len(cookielib.parse_ns_headers([_test_cookie])) == 2 fs-0.5.4/fs/contrib/davfs/__init__.py0000664000175000017500000010254612512525115017365 0ustar willwill00000000000000""" fs.contrib.davfs ================ FS implementation accessing a WebDAV server. This module provides a relatively-complete WebDAV Level 1 client that exposes a WebDAV server as an FS object. Locks are not currently supported. Requires the dexml module: http://pypi.python.org/pypi/dexml/ """ # Copyright (c) 2009-2010, Cloud Matrix Pty. Ltd. # All rights reserved; available under the terms of the MIT License. from __future__ import with_statement import os import sys import httplib import socket from urlparse import urlparse import stat as statinfo from urllib import quote as urlquote from urllib import unquote as urlunquote import base64 import re import time import datetime import cookielib import fnmatch import xml.dom.pulldom import threading from collections import deque import fs from fs.base import * from fs.path import * from fs.errors import * from fs.remote import RemoteFileBuffer from fs import iotools from fs.contrib.davfs.util import * from fs.contrib.davfs import xmlobj from fs.contrib.davfs.xmlobj import * import six from six import b import errno _RETRYABLE_ERRORS = [errno.EADDRINUSE] try: _RETRYABLE_ERRORS.append(errno.ECONNRESET) _RETRYABLE_ERRORS.append(errno.ECONNABORTED) except AttributeError: _RETRYABLE_ERRORS.append(104) class DAVFS(FS): """Access a remote filesystem via WebDAV. This FS implementation provides access to a remote filesystem via the WebDAV protocol. Basic Level 1 WebDAV is supported; locking is not currently supported, but planned for the future. HTTP Basic authentication is supported; provide a dict giving username and password in the "credentials" argument, or a callback for obtaining one in the "get_credentials" argument. To use custom HTTP connector classes (e.g. to implement proper certificate checking for SSL connections) you can replace the factory functions in the DAVFS.connection_classes dictionary, or provide the "connection_classes" argument. """ connection_classes = { "http": httplib.HTTPConnection, "https": httplib.HTTPSConnection, } _DEFAULT_PORT_NUMBERS = { "http": 80, "https": 443, } _meta = { 'virtual' : False, 'read_only' : False, 'unicode_paths' : True, 'case_insensitive_paths' : False, 'network' : True } def __init__(self,url,credentials=None,get_credentials=None,thread_synchronize=True,connection_classes=None,timeout=None): """DAVFS constructor. The only required argument is the root url of the remote server. If authentication is required, provide the 'credentials' keyword argument and/or the 'get_credentials' keyword argument. The former is a dict of credentials info, while the latter is a callback function returning such a dict. Only HTTP Basic Auth is supported at this stage, so the only useful keys in a credentials dict are 'username' and 'password'. """ if not url.endswith("/"): url = url + "/" self.url = url self.timeout = timeout self.credentials = credentials self.get_credentials = get_credentials if connection_classes is not None: self.connection_classes = self.connection_classes.copy() self.connection_classes.update(connection_classes) self._connections = [] self._free_connections = {} self._connection_lock = threading.Lock() self._cookiejar = cookielib.CookieJar() super(DAVFS,self).__init__(thread_synchronize=thread_synchronize) # Check that the server speaks WebDAV, and normalize the URL # after any redirects have been followed. self.url = url pf = propfind(prop="") resp = self._request("/","PROPFIND",pf.render(),{"Depth":"0"}) try: if resp.status == 404: raise ResourceNotFoundError("/",msg="root url gives 404") if resp.status in (401,403): raise PermissionDeniedError("listdir (http %s)" % resp.status) if resp.status != 207: msg = "server at %s doesn't speak WebDAV" % (self.url,) raise RemoteConnectionError("",msg=msg,details=resp.read()) finally: resp.close() self.url = resp.request_url self._url_p = urlparse(self.url) def close(self): for con in self._connections: con.close() super(DAVFS,self).close() def _take_connection(self,url): """Get a connection to the given url's host, re-using if possible.""" scheme = url.scheme.lower() hostname = url.hostname port = url.port if not port: try: port = self._DEFAULT_PORT_NUMBERS[scheme] except KeyError: msg = "unsupported protocol: '%s'" % (url.scheme,) raise RemoteConnectionError(msg=msg) # Can we re-use an existing connection? with self._connection_lock: now = time.time() try: free_connections = self._free_connections[(hostname,port)] except KeyError: self._free_connections[(hostname,port)] = deque() free_connections = self._free_connections[(hostname,port)] else: while free_connections: (when,con) = free_connections.popleft() if when + 30 > now: return (False,con) self._discard_connection(con) # Nope, we need to make a fresh one. try: ConClass = self.connection_classes[scheme] except KeyError: msg = "unsupported protocol: '%s'" % (url.scheme,) raise RemoteConnectionError(msg=msg) con = ConClass(url.hostname,url.port,timeout=self.timeout) self._connections.append(con) return (True,con) def _give_connection(self,url,con): """Return a connection to the pool, or destroy it if dead.""" scheme = url.scheme.lower() hostname = url.hostname port = url.port if not port: try: port = self._DEFAULT_PORT_NUMBERS[scheme] except KeyError: msg = "unsupported protocol: '%s'" % (url.scheme,) raise RemoteConnectionError(msg=msg) with self._connection_lock: now = time.time() try: free_connections = self._free_connections[(hostname,port)] except KeyError: self._free_connections[(hostname,port)] = deque() free_connections = self._free_connections[(hostname,port)] free_connections.append((now,con)) def _discard_connection(self,con): con.close() self._connections.remove(con) def __str__(self): return '' % (self.url,) __repr__ = __str__ def __getstate__(self): state = super(DAVFS,self).__getstate__() del state["_connection_lock"] del state["_connections"] del state["_free_connections"] # Python2.5 cannot load pickled urlparse.ParseResult objects. del state["_url_p"] # CookieJar objects contain a lock, so they can't be pickled. del state["_cookiejar"] return state def __setstate__(self,state): super(DAVFS,self).__setstate__(state) self._connections = [] self._free_connections = {} self._connection_lock = threading.Lock() self._url_p = urlparse(self.url) self._cookiejar = cookielib.CookieJar() def getpathurl(self, path, allow_none=False): """Convert a client-side path into a server-side URL.""" path = relpath(normpath(path)) if path.endswith("/"): path = path[:-1] if isinstance(path,unicode): path = path.encode("utf8") return self.url + urlquote(path) def _url2path(self,url): """Convert a server-side URL into a client-side path.""" path = urlunquote(urlparse(url).path) root = urlunquote(self._url_p.path) path = path[len(root)-1:].decode("utf8") while path.endswith("/"): path = path[:-1] return path def _isurl(self,path,url): """Check whether the given URL corresponds to the given local path.""" path = normpath(relpath(path)) upath = relpath(normpath(self._url2path(url))) return path == upath def _request(self,path,method,body="",headers={}): """Issue a HTTP request to the remote server. This is a simple wrapper around httplib that does basic error and sanity checking e.g. following redirects and providing authentication. """ url = self.getpathurl(path) visited = [] resp = None try: resp = self._raw_request(url,method,body,headers) # Loop to retry for redirects and authentication responses. while resp.status in (301,302,401,403): resp.close() if resp.status in (301,302,): visited.append(url) url = resp.getheader("Location",None) if not url: raise OperationFailedError(msg="no location header in 301 response") if url in visited: raise OperationFailedError(msg="redirection seems to be looping") if len(visited) > 10: raise OperationFailedError("too much redirection") elif resp.status in (401,403): if self.get_credentials is None: break else: creds = self.get_credentials(self.credentials) if creds is None: break else: self.credentials = creds resp = self._raw_request(url,method,body,headers) except Exception: if resp is not None: resp.close() raise resp.request_url = url return resp def _raw_request(self,url,method,body,headers,num_tries=0): """Perform a single HTTP request, without any error handling.""" if self.closed: raise RemoteConnectionError("",msg="FS is closed") if isinstance(url,basestring): url = urlparse(url) if self.credentials is not None: username = self.credentials.get("username","") password = self.credentials.get("password","") if username is not None and password is not None: creds = "%s:%s" % (username,password,) creds = "Basic %s" % (base64.b64encode(creds).strip(),) headers["Authorization"] = creds (size,chunks) = normalize_req_body(body) try: (fresh,con) = self._take_connection(url) try: con.putrequest(method,url.path) if size is not None: con.putheader("Content-Length",str(size)) if hasattr(body,"md5"): md5 = body.md5.decode("hex").encode("base64") con.putheader("Content-MD5",md5) for hdr,val in headers.iteritems(): con.putheader(hdr,val) self._cookiejar.add_cookie_header(FakeReq(con,url.scheme,url.path)) con.endheaders() for chunk in chunks: con.send(chunk) if self.closed: raise RemoteConnectionError("",msg="FS is closed") resp = con.getresponse() self._cookiejar.extract_cookies(FakeResp(resp),FakeReq(con,url.scheme,url.path)) except Exception: self._discard_connection(con) raise else: old_close = resp.close def new_close(): del resp.close old_close() con.close() self._give_connection(url,con) resp.close = new_close return resp except socket.error, e: if not fresh: return self._raw_request(url,method,body,headers,num_tries) if e.args[0] in _RETRYABLE_ERRORS: if num_tries < 3: num_tries += 1 return self._raw_request(url,method,body,headers,num_tries) try: msg = e.args[1] except IndexError: msg = str(e) raise RemoteConnectionError("",msg=msg,details=e) def setcontents(self,path, data=b'', encoding=None, errors=None, chunk_size=1024 * 64): if isinstance(data, six.text_type): data = data.encode(encoding=encoding, errors=errors) resp = self._request(path, "PUT", data) resp.close() if resp.status == 405: raise ResourceInvalidError(path) if resp.status == 409: raise ParentDirectoryMissingError(path) if resp.status not in (200,201,204): raise_generic_error(resp,"setcontents",path) @iotools.filelike_to_stream def open(self,path,mode="r", **kwargs): mode = mode.replace("b","").replace("t","") # Truncate the file if requested contents = b("") if "w" in mode: self.setcontents(path,contents) else: contents = self._request(path,"GET") if contents.status == 404: # Create the file if it's missing in append mode. if "a" not in mode: contents.close() raise ResourceNotFoundError(path) contents = b("") self.setcontents(path,contents) elif contents.status in (401,403): contents.close() raise PermissionDeniedError("open") elif contents.status != 200: contents.close() raise_generic_error(contents,"open",path) elif self.isdir(path): contents.close() raise ResourceInvalidError(path) # For streaming reads, return the socket contents directly. if mode == "r-": contents.size = contents.getheader("Content-Length",None) if contents.size is not None: try: contents.size = int(contents.size) except ValueError: contents.size = None if not hasattr(contents,"__exit__"): contents.__enter__ = lambda *a: contents contents.__exit__ = lambda *a: contents.close() return contents # For everything else, use a RemoteFileBuffer. # This will take care of closing the socket when it's done. return RemoteFileBuffer(self,path,mode,contents) def exists(self,path): pf = propfind(prop="") response = self._request(path,"PROPFIND",pf.render(),{"Depth":"0"}) response.close() if response.status == 207: return True if response.status == 404: return False raise_generic_error(response,"exists",path) def isdir(self,path): pf = propfind(prop="") response = self._request(path,"PROPFIND",pf.render(),{"Depth":"0"}) try: if response.status == 404: return False if response.status != 207: raise_generic_error(response,"isdir",path) body = response.read() msres = multistatus.parse(body) for res in msres.responses: if self._isurl(path,res.href): for ps in res.propstats: if ps.props.getElementsByTagNameNS("DAV:","collection"): return True return False finally: response.close() def isfile(self,path): pf = propfind(prop="") response = self._request(path,"PROPFIND",pf.render(),{"Depth":"0"}) try: if response.status == 404: return False if response.status != 207: raise_generic_error(response,"isfile",path) msres = multistatus.parse(response.read()) for res in msres.responses: if self._isurl(path,res.href): for ps in res.propstats: rt = ps.props.getElementsByTagNameNS("DAV:","resourcetype") cl = ps.props.getElementsByTagNameNS("DAV:","collection") if rt and not cl: return True return False finally: response.close() def listdir(self,path="./",wildcard=None,full=False,absolute=False,dirs_only=False,files_only=False): return list(self.ilistdir(path=path,wildcard=wildcard,full=full,absolute=absolute,dirs_only=dirs_only,files_only=files_only)) def ilistdir(self,path="./",wildcard=None,full=False,absolute=False,dirs_only=False,files_only=False): props = "" dir_ok = False for res in self._do_propfind(path,props): if self._isurl(path,res.href): # The directory itself, check it's actually a directory for ps in res.propstats: if ps.props.getElementsByTagNameNS("DAV:","collection"): dir_ok = True break else: nm = basename(self._url2path(res.href)) entry_ok = False if dirs_only: for ps in res.propstats: if ps.props.getElementsByTagNameNS("DAV:","collection"): entry_ok = True break elif files_only: for ps in res.propstats: if ps.props.getElementsByTagNameNS("DAV:","collection"): break else: entry_ok = True else: entry_ok = True if not entry_ok: continue if wildcard is not None: if isinstance(wildcard,basestring): if not fnmatch.fnmatch(nm,wildcard): continue else: if not wildcard(nm): continue if full: yield relpath(pathjoin(path,nm)) elif absolute: yield abspath(pathjoin(path,nm)) else: yield nm if not dir_ok: raise ResourceInvalidError(path) def listdirinfo(self,path="./",wildcard=None,full=False,absolute=False,dirs_only=False,files_only=False): return list(self.ilistdirinfo(path=path,wildcard=wildcard,full=full,absolute=absolute,dirs_only=dirs_only,files_only=files_only)) def ilistdirinfo(self,path="./",wildcard=None,full=False,absolute=False,dirs_only=False,files_only=False): props = "" \ "" dir_ok = False for res in self._do_propfind(path,props): if self._isurl(path,res.href): # The directory itself, check it's actually a directory for ps in res.propstats: if ps.props.getElementsByTagNameNS("DAV:","collection"): dir_ok = True break else: # An entry in the directory, check if it's of the # appropriate type and add to entries list as required. info = self._info_from_propfind(res) nm = basename(self._url2path(res.href)) entry_ok = False if dirs_only: for ps in res.propstats: if ps.props.getElementsByTagNameNS("DAV:","collection"): entry_ok = True break elif files_only: for ps in res.propstats: if ps.props.getElementsByTagNameNS("DAV:","collection"): break else: entry_ok = True else: entry_ok = True if not entry_ok: continue if wildcard is not None: if isinstance(wildcard,basestring): if not fnmatch.fnmatch(nm,wildcard): continue else: if not wildcard(nm): continue if full: yield (relpath(pathjoin(path,nm)),info) elif absolute: yield (abspath(pathjoin(path,nm)),info) else: yield (nm,info) if not dir_ok: raise ResourceInvalidError(path) def makedir(self,path,recursive=False,allow_recreate=False): response = self._request(path,"MKCOL") response.close() if response.status == 201: return True if response.status == 409: if not recursive: raise ParentDirectoryMissingError(path) self.makedir(dirname(path),recursive=True,allow_recreate=True) self.makedir(path,recursive=False,allow_recreate=allow_recreate) return True if response.status == 405: if not self.isdir(path): raise ResourceInvalidError(path) if not allow_recreate: raise DestinationExistsError(path) return True if response.status < 200 or response.status >= 300: raise_generic_error(response,"makedir",path) def remove(self,path): if self.isdir(path): raise ResourceInvalidError(path) response = self._request(path,"DELETE") response.close() if response.status == 405: raise ResourceInvalidError(path) if response.status < 200 or response.status >= 300: raise_generic_error(response,"remove",path) return True def removedir(self,path,recursive=False,force=False): if self.isfile(path): raise ResourceInvalidError(path) if not force and self.listdir(path): raise DirectoryNotEmptyError(path) response = self._request(path,"DELETE") response.close() if response.status == 405: raise ResourceInvalidError(path) if response.status < 200 or response.status >= 300: raise_generic_error(response,"removedir",path) if recursive and path not in ("","/"): try: self.removedir(dirname(path),recursive=True) except DirectoryNotEmptyError: pass return True def rename(self,src,dst): self._move(src,dst) def getinfo(self,path): info = {} info["name"] = basename(path) pf = propfind(prop="") response = self._request(path,"PROPFIND",pf.render(),{"Depth":"0"}) try: if response.status != 207: raise_generic_error(response,"getinfo",path) msres = multistatus.parse(response.read()) for res in msres.responses: if self._isurl(path,res.href): info.update(self._info_from_propfind(res)) if "st_mode" not in info: info["st_mode"] = 0700 | statinfo.S_IFREG return info finally: response.close() def _do_propfind(self,path,props): """Incremental PROPFIND parsing, for use with ilistdir/ilistdirinfo. This generator method incrementally parses the results returned by a PROPFIND, yielding each object as it becomes available. If the server is able to send responses in chunked encoding, then this can substantially speed up iterating over the results. """ pf = propfind(prop=""+props+"") response = self._request(path,"PROPFIND",pf.render(),{"Depth":"1"}) try: if response.status == 404: raise ResourceNotFoundError(path) if response.status != 207: raise_generic_error(response,"listdir",path) xmlevents = xml.dom.pulldom.parse(response,bufsize=1024) for (evt,node) in xmlevents: if evt == xml.dom.pulldom.START_ELEMENT: if node.namespaceURI == "DAV:": if node.localName == "response": xmlevents.expandNode(node) yield xmlobj.response.parse(node) finally: response.close() def _info_from_propfind(self,res): info = {} for ps in res.propstats: findElements = ps.props.getElementsByTagNameNS # TODO: should check for status of the propfind first... # check for directory indicator if findElements("DAV:","collection"): info["st_mode"] = 0700 | statinfo.S_IFDIR # check for content length cl = findElements("DAV:","getcontentlength") if cl: cl = "".join(c.nodeValue for c in cl[0].childNodes) try: info["size"] = int(cl) except ValueError: pass # check for last modified time lm = findElements("DAV:","getlastmodified") if lm: lm = "".join(c.nodeValue for c in lm[0].childNodes) try: # TODO: more robust datetime parsing fmt = "%a, %d %b %Y %H:%M:%S GMT" mtime = datetime.datetime.strptime(lm,fmt) info["modified_time"] = mtime except ValueError: pass # check for etag etag = findElements("DAV:","getetag") if etag: etag = "".join(c.nodeValue for c in etag[0].childNodes) if etag: info["etag"] = etag if "st_mode" not in info: info["st_mode"] = 0700 | statinfo.S_IFREG return info def copy(self,src,dst,overwrite=False,chunk_size=None): if self.isdir(src): msg = "Source is not a file: %(path)s" raise ResourceInvalidError(src, msg=msg) self._copy(src,dst,overwrite=overwrite) def copydir(self,src,dst,overwrite=False,ignore_errors=False,chunk_size=0): if self.isfile(src): msg = "Source is not a directory: %(path)s" raise ResourceInvalidError(src, msg=msg) self._copy(src,dst,overwrite=overwrite) def _copy(self,src,dst,overwrite=False): headers = {"Destination":self.getpathurl(dst)} if overwrite: headers["Overwrite"] = "T" else: headers["Overwrite"] = "F" response = self._request(src,"COPY",headers=headers) response.close() if response.status == 412: raise DestinationExistsError(dst) if response.status == 409: raise ParentDirectoryMissingError(dst) if response.status < 200 or response.status >= 300: raise_generic_error(response,"copy",src) def move(self,src,dst,overwrite=False,chunk_size=None): if self.isdir(src): msg = "Source is not a file: %(path)s" raise ResourceInvalidError(src, msg=msg) self._move(src,dst,overwrite=overwrite) def movedir(self,src,dst,overwrite=False,ignore_errors=False,chunk_size=0): if self.isfile(src): msg = "Source is not a directory: %(path)s" raise ResourceInvalidError(src, msg=msg) self._move(src,dst,overwrite=overwrite) def _move(self,src,dst,overwrite=False): headers = {"Destination":self.getpathurl(dst)} if overwrite: headers["Overwrite"] = "T" else: headers["Overwrite"] = "F" response = self._request(src,"MOVE",headers=headers) response.close() if response.status == 412: raise DestinationExistsError(dst) if response.status == 409: raise ParentDirectoryMissingError(dst) if response.status < 200 or response.status >= 300: raise_generic_error(response,"move",src) @staticmethod def _split_xattr(name): """Split extended attribute name into (namespace,localName) pair.""" idx = len(name)-1 while idx >= 0 and name[idx].isalnum(): idx -= 1 return (name[:idx+1],name[idx+1:]) def getxattr(self,path,name,default=None): (namespaceURI,localName) = self._split_xattr(name) # TODO: encode xml character entities in the namespace if namespaceURI: pf = propfind(prop="<"+localName+" />") else: pf = propfind(prop="<"+localName+" />") response = self._request(path,"PROPFIND",pf.render(),{"Depth":"0"}) try: if response.status != 207: raise_generic_error(response,"getxattr",path) msres = multistatus.parse(response.read()) finally: response.close() for res in msres.responses: if self._isurl(path,res.href): for ps in res.propstats: if namespaceURI: findElements = ps.props.getElementsByTagNameNS propNode = findElements(namespaceURI,localName) else: findElements = ps.props.getElementsByTagName propNode = findElements(localName) if propNode: propNode = propNode[0] if ps.status.code == 200: return "".join(c.toxml() for c in propNode.childNodes) if ps.status.code == 404: return default raise OperationFailedError("getxattr",msres.render()) return default def setxattr(self,path,name,value): (namespaceURI,localName) = self._split_xattr(name) # TODO: encode xml character entities in the namespace if namespaceURI: p = "<%s xmlns='%s'>%s" % (localName,namespaceURI,value,localName) else: p = "<%s>%s" % (localName,value,localName) pu = propertyupdate() pu.commands.append(set(props=""+p+"")) response = self._request(path,"PROPPATCH",pu.render(),{"Depth":"0"}) response.close() if response.status < 200 or response.status >= 300: raise_generic_error(response,"setxattr",path) def delxattr(self,path,name): (namespaceURI,localName) = self._split_xattr(name) # TODO: encode xml character entities in the namespace if namespaceURI: p = "<%s xmlns='%s' />" % (localName,namespaceURI,) else: p = "<%s />" % (localName,) pu = propertyupdate() pu.commands.append(remove(props=""+p+"")) response = self._request(path,"PROPPATCH",pu.render(),{"Depth":"0"}) response.close() if response.status < 200 or response.status >= 300: raise_generic_error(response,"delxattr",path) def listxattrs(self,path): pf = propfind(propname=True) response = self._request(path,"PROPFIND",pf.render(),{"Depth":"0"}) try: if response.status != 207: raise_generic_error(response,"listxattrs",path) msres = multistatus.parse(response.read()) finally: response.close() props = [] for res in msres.responses: if self._isurl(path,res.href): for ps in res.propstats: for node in ps.props.childNodes: if node.nodeType != node.ELEMENT_NODE: continue if node.namespaceURI: if node.namespaceURI in ("DAV:","PYFS:",): continue propname = node.namespaceURI + node.localName else: propname = node.nodeName props.append(propname) return props # TODO: bulk getxattrs() and setxattrs() methods def raise_generic_error(response,opname,path): if response.status == 404: raise ResourceNotFoundError(path,details=response.read()) if response.status in (401,403): raise PermissionDeniedError(opname,details=response.read()) if response.status == 423: raise ResourceLockedError(path,opname=opname,details=response.read()) if response.status == 501: raise UnsupportedError(opname,details=response.read()) if response.status == 405: raise ResourceInvalidError(path,opname=opname,details=response.read()) raise OperationFailedError(opname,msg="Server Error: %s" % (response.status,),details=response.read()) fs-0.5.4/fs/contrib/tahoelafs/0000755000000000000000000000000012621617365016115 5ustar rootroot00000000000000fs-0.5.4/fs/contrib/tahoelafs/util.py0000664000175000017500000001117312512525115017441 0ustar willwill00000000000000''' Created on 25.9.2010 @author: marekp ''' import sys import platform import stat as statinfo import fs.errors as errors from fs.path import pathsplit try: # For non-CPython or older CPython versions. # Simplejson also comes with C speedup module which # is not in standard CPython >=2.6 library. import simplejson as json except ImportError: try: import json except ImportError: print "simplejson (http://pypi.python.org/pypi/simplejson/) required" raise from .connection import Connection python3 = int(platform.python_version_tuple()[0]) > 2 if python3: from urllib.error import HTTPError else: from urllib2 import HTTPError class TahoeUtil: def __init__(self, webapi): self.connection = Connection(webapi) def createdircap(self): return self.connection.post(u'/uri', params={u't': u'mkdir'}).read() def unlink(self, dircap, path=None): path = self.fixwinpath(path, False) self.connection.delete(u'/uri/%s%s' % (dircap, path)) def info(self, dircap, path): path = self.fixwinpath(path, False) meta = json.load(self.connection.get(u'/uri/%s%s' % (dircap, path), {u't': u'json'})) return self._info(path, meta) def fixwinpath(self, path, direction=True): ''' No, Tahoe really does not support file streams... This is ugly hack, because it is not Tahoe-specific. Should be move to middleware if will be any. ''' if platform.system() != 'Windows': return path if direction and ':' in path: path = path.replace(':', '__colon__') elif not direction and '__colon__' in path: path = path.replace('__colon__', ':') return path def _info(self, path, data): if isinstance(data, list): type = data[0] data = data[1] elif isinstance(data, dict): type = data['type'] else: raise errors.ResourceInvalidError('Metadata in unknown format!') if type == 'unknown': raise errors.ResourceNotFoundError(path) info = {'name': unicode(self.fixwinpath(path, True)), 'type': type, 'size': data.get('size', 0), 'ctime': None, 'uri': data.get('rw_uri', data.get('ro_uri'))} if 'metadata' in data: info['ctime'] = data['metadata'].get('ctime') if info['type'] == 'dirnode': info['st_mode'] = 0777 | statinfo.S_IFDIR else: info['st_mode'] = 0644 return info def list(self, dircap, path=None): path = self.fixwinpath(path, False) data = json.load(self.connection.get(u'/uri/%s%s' % (dircap, path), {u't': u'json'})) if len(data) < 2 or data[0] != 'dirnode': raise errors.ResourceInvalidError('Metadata in unknown format!') data = data[1]['children'] for i in data.keys(): x = self._info(i, data[i]) yield x def mkdir(self, dircap, path): path = self.fixwinpath(path, False) path = pathsplit(path) self.connection.post(u"/uri/%s%s" % (dircap, path[0]), data={u't': u'mkdir', u'name': path[1]}) def move(self, dircap, src, dst): if src == '/' or dst == '/': raise errors.UnsupportedError("Too dangerous operation, aborting") src = self.fixwinpath(src, False) dst = self.fixwinpath(dst, False) src_tuple = pathsplit(src) dst_tuple = pathsplit(dst) if src_tuple[0] == dst_tuple[0]: # Move inside one directory self.connection.post(u"/uri/%s%s" % (dircap, src_tuple[0]), data={u't': u'rename', u'from_name': src_tuple[1], u'to_name': dst_tuple[1]}) return # Move to different directory. Firstly create link on dst, then remove from src try: self.info(dircap, dst) except errors.ResourceNotFoundError: pass else: self.unlink(dircap, dst) uri = self.info(dircap, src)['uri'] self.connection.put(u"/uri/%s%s" % (dircap, dst), data=uri, params={u't': u'uri'}) if uri != self.info(dircap, dst)['uri']: raise errors.OperationFailedError('Move failed') self.unlink(dircap, src) fs-0.5.4/fs/contrib/tahoelafs/test_tahoelafs.py0000664000175000017500000000265712512525115021500 0ustar willwill00000000000000#!/usr/bin/python """ Test the TahoeLAFS @author: Marek Palatinus """ import sys import logging import unittest from fs.base import FS import fs.errors as errors from fs.tests import FSTestCases, ThreadingTestCases from fs.contrib.tahoelafs import TahoeLAFS, Connection logging.getLogger().setLevel(logging.DEBUG) logging.getLogger('fs.tahoelafs').addHandler(logging.StreamHandler(sys.stdout)) WEBAPI = 'http://insecure.tahoe-lafs.org' # The public grid is too slow for threading testcases, disabling for now... class TestTahoeLAFS(unittest.TestCase,FSTestCases):#,ThreadingTestCases): # Disabled by default because it takes a *really* long time. __test__ = False def setUp(self): self.dircap = TahoeLAFS.createdircap(WEBAPI) self.fs = TahoeLAFS(self.dircap, cache_timeout=0, webapi=WEBAPI) def tearDown(self): self.fs.close() def test_dircap(self): # Is dircap in correct format? self.assert_(self.dircap.startswith('URI:DIR2:') and len(self.dircap) > 50) def test_concurrent_copydir(self): # makedir() on TahoeLAFS is currently not atomic pass def test_makedir_winner(self): # makedir() on TahoeLAFS is currently not atomic pass def test_big_file(self): pass if __name__ == '__main__': unittest.main() fs-0.5.4/fs/contrib/tahoelafs/connection.py0000664000175000017500000000735512512525115020632 0ustar willwill00000000000000import platform import logging import fs.errors as errors from fs import SEEK_END python3 = int(platform.python_version_tuple()[0]) > 2 if python3: from urllib.parse import urlencode, pathname2url, quote from urllib.request import Request, urlopen else: from urllib import urlencode, pathname2url from urllib2 import Request, urlopen, quote class PutRequest(Request): def __init__(self, *args, **kwargs): self.get_method = lambda: u'PUT' Request.__init__(self, *args, **kwargs) class DeleteRequest(Request): def __init__(self, *args, **kwargs): self.get_method = lambda: u'DELETE' Request.__init__(self, *args, **kwargs) class Connection: def __init__(self, webapi): self.webapi = webapi self.headers = {'Accept': 'text/plain'} def _get_headers(self, f, size=None): ''' Retrieve length of string or file object and prepare HTTP headers. ''' if isinstance(f, basestring): # Just set up content length size = len(f) elif getattr(f, 'read', None): if size == None: # When size is already known, skip this f.seek(0, SEEK_END) size = f.tell() f.seek(0) else: raise errors.UnsupportedError("Cannot handle type %s" % type(f)) headers = {'Content-Length': size} headers.update(self.headers) return headers def _urlencode(self, data): _data = {} for k, v in data.items(): _data[k.encode('utf-8')] = v.encode('utf-8') return urlencode(_data) def _quotepath(self, path, params={}): q = quote(path.encode('utf-8'), safe='/') if params: return u"%s?%s" % (q, self._urlencode(params)) return q def _urlopen(self, req): try: return urlopen(req) except Exception, e: if not getattr(e, 'getcode', None): raise errors.RemoteConnectionError(str(e)) code = e.getcode() if code == 500: # Probably out of space or unhappiness error raise errors.StorageSpaceError(e.fp.read()) elif code in (400, 404, 410): # Standard not found raise errors.ResourceNotFoundError(e.fp.read()) raise errors.ResourceInvalidError(e.fp.read()) def post(self, path, data={}, params={}): data = self._urlencode(data) path = self._quotepath(path, params) req = Request(''.join([self.webapi, path]), data, headers=self.headers) return self._urlopen(req) def get(self, path, data={}, offset=None, length=None): data = self._urlencode(data) path = self._quotepath(path) if data: path = u'?'.join([path, data]) headers = {} headers.update(self.headers) if offset: if length: headers['Range'] = 'bytes=%d-%d' % \ (int(offset), int(offset+length)) else: headers['Range'] = 'bytes=%d-' % int(offset) req = Request(''.join([self.webapi, path]), headers=headers) return self._urlopen(req) def put(self, path, data, size=None, params={}): path = self._quotepath(path, params) headers = self._get_headers(data, size=size) req = PutRequest(''.join([self.webapi, path]), data, headers=headers) return self._urlopen(req) def delete(self, path, data={}): path = self._quotepath(path) req = DeleteRequest(''.join([self.webapi, path]), data, headers=self.headers) return self._urlopen(req) fs-0.5.4/fs/contrib/tahoelafs/__init__.py0000664000175000017500000003616012512525115020226 0ustar willwill00000000000000''' fs.contrib.tahoelafs ==================== This modules provides a PyFilesystem interface to the Tahoe Least Authority File System. Tahoe-LAFS is a distributed, encrypted, fault-tolerant storage system: http://tahoe-lafs.org/ You will need access to a Tahoe-LAFS "web api" service. Example (it will use publicly available (but slow) Tahoe-LAFS cloud):: from fs.contrib.tahoelafs import TahoeLAFS, Connection dircap = TahoeLAFS.createdircap(webapi='http://insecure.tahoe-lafs.org') print "Your dircap (unique key to your storage directory) is", dircap print "Keep it safe!" fs = TahoeLAFS(dircap, autorun=False, webapi='http://insecure.tahoe-lafs.org') f = fs.open("foo.txt", "a") f.write('bar!') f.close() print "Now visit %s and enjoy :-)" % fs.getpathurl('foo.txt') When any problem occurred, you can turn on internal debugging messages:: import logging l = logging.getLogger() l.setLevel(logging.DEBUG) l.addHandler(logging.StreamHandler(sys.stdout)) ... your Python code using TahoeLAFS ... TODO: * unicode support * try network errors / bad happiness * exceptions * tests * sanitize all path types (., /) * support for extra large file uploads (poster module) * Possibility to block write until upload done (Tahoe mailing list) * Report something sane when Tahoe crashed/unavailable * solve failed unit tests (makedir_winner, ...) * file times * docs & author * python3 support * remove creating blank files (depends on FileUploadManager) TODO (Not TahoeLAFS specific tasks): * RemoteFileBuffer on the fly buffering support * RemoteFileBuffer unit tests * RemoteFileBuffer submit to trunk * Implement FileUploadManager + faking isfile/exists of just processing file * pyfilesystem docs is outdated (rename, movedir, ...) ''' import stat as statinfo import logging from logging import DEBUG, INFO, ERROR, CRITICAL import fs import fs.errors as errors from fs.path import abspath, relpath, normpath, dirname, pathjoin from fs.base import FS, NullFile from fs import _thread_synchronize_default, SEEK_END from fs.remote import CacheFSMixin, RemoteFileBuffer from fs.base import fnmatch, NoDefaultMeta from util import TahoeUtil from connection import Connection from six import b logger = fs.getLogger('fs.tahoelafs') def _fix_path(func): """Method decorator for automatically normalising paths.""" def wrapper(self, *args, **kwds): if len(args): args = list(args) args[0] = _fixpath(args[0]) return func(self, *args, **kwds) return wrapper def _fixpath(path): """Normalize the given path.""" return abspath(normpath(path)) class _TahoeLAFS(FS): """FS providing raw access to a Tahoe-LAFS Filesystem. This class implements all the details of interacting with a Tahoe-backed filesystem, but you probably don't want to use it in practice. Use the TahoeLAFS class instead, which has some internal caching to improve performance. """ _meta = { 'virtual' : False, 'read_only' : False, 'unicode_paths' : True, 'case_insensitive_paths' : False, 'network' : True } def __init__(self, dircap, largefilesize=10*1024*1024, webapi='http://127.0.0.1:3456'): '''Creates instance of TahoeLAFS. :param dircap: special hash allowing user to work with TahoeLAFS directory. :param largefilesize: - Create placeholder file for files larger than this treshold. Uploading and processing of large files can last extremely long (many hours), so placing this placeholder can help you to remember that upload is processing. Setting this to None will skip creating placeholder files for any uploads. ''' self.dircap = dircap if not dircap.endswith('/') else dircap[:-1] self.largefilesize = largefilesize self.connection = Connection(webapi) self.tahoeutil = TahoeUtil(webapi) super(_TahoeLAFS, self).__init__(thread_synchronize=_thread_synchronize_default) def __str__(self): return "" % self.dircap @classmethod def createdircap(cls, webapi='http://127.0.0.1:3456'): return TahoeUtil(webapi).createdircap() def getmeta(self,meta_name,default=NoDefaultMeta): if meta_name == "read_only": return self.dircap.startswith('URI:DIR2-RO') return super(_TahoeLAFS,self).getmeta(meta_name,default) @_fix_path def open(self, path, mode='r', **kwargs): self._log(INFO, 'Opening file %s in mode %s' % (path, mode)) newfile = False if not self.exists(path): if 'w' in mode or 'a' in mode: newfile = True else: self._log(DEBUG, "File %s not found while opening for reads" % path) raise errors.ResourceNotFoundError(path) elif self.isdir(path): self._log(DEBUG, "Path %s is directory, not a file" % path) raise errors.ResourceInvalidError(path) elif 'w' in mode: newfile = True if newfile: self._log(DEBUG, 'Creating empty file %s' % path) if self.getmeta("read_only"): raise errors.UnsupportedError('read only filesystem') self.setcontents(path, b('')) handler = NullFile() else: self._log(DEBUG, 'Opening existing file %s for reading' % path) handler = self.getrange(path,0) return RemoteFileBuffer(self, path, mode, handler, write_on_flush=False) @_fix_path def desc(self, path): try: return self.getinfo(path) except: return '' @_fix_path def exists(self, path): try: self.getinfo(path) self._log(DEBUG, "Path %s exists" % path) return True except errors.ResourceNotFoundError: self._log(DEBUG, "Path %s does not exists" % path) return False except errors.ResourceInvalidError: self._log(DEBUG, "Path %s does not exists, probably misspelled URI" % path) return False @_fix_path def getsize(self, path): try: size = self.getinfo(path)['size'] self._log(DEBUG, "Size of %s is %d" % (path, size)) return size except errors.ResourceNotFoundError: return 0 @_fix_path def isfile(self, path): try: isfile = (self.getinfo(path)['type'] == 'filenode') except errors.ResourceNotFoundError: #isfile = not path.endswith('/') isfile = False self._log(DEBUG, "Path %s is file: %d" % (path, isfile)) return isfile @_fix_path def isdir(self, path): try: isdir = (self.getinfo(path)['type'] == 'dirnode') except errors.ResourceNotFoundError: isdir = False self._log(DEBUG, "Path %s is directory: %d" % (path, isdir)) return isdir def listdir(self, *args, **kwargs): return [ item[0] for item in self.listdirinfo(*args, **kwargs) ] def listdirinfo(self, *args, **kwds): return list(self.ilistdirinfo(*args,**kwds)) def ilistdir(self, *args, **kwds): for item in self.ilistdirinfo(*args,**kwds): yield item[0] @_fix_path def ilistdirinfo(self, path="/", wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False): self._log(DEBUG, "Listing directory (listdirinfo) %s" % path) if dirs_only and files_only: raise ValueError("dirs_only and files_only can not both be True") for item in self.tahoeutil.list(self.dircap, path): if dirs_only and item['type'] == 'filenode': continue elif files_only and item['type'] == 'dirnode': continue if wildcard is not None: if isinstance(wildcard,basestring): if not fnmatch.fnmatch(item['name'], wildcard): continue else: if not wildcard(item['name']): continue if full: item_path = relpath(pathjoin(path, item['name'])) elif absolute: item_path = abspath(pathjoin(path, item['name'])) else: item_path = item['name'] yield (item_path, item) @_fix_path def remove(self, path): self._log(INFO, 'Removing file %s' % path) if self.getmeta("read_only"): raise errors.UnsupportedError('read only filesystem') if not self.isfile(path): if not self.isdir(path): raise errors.ResourceNotFoundError(path) raise errors.ResourceInvalidError(path) try: self.tahoeutil.unlink(self.dircap, path) except Exception, e: raise errors.ResourceInvalidError(path) @_fix_path def removedir(self, path, recursive=False, force=False): self._log(INFO, "Removing directory %s" % path) if self.getmeta("read_only"): raise errors.UnsupportedError('read only filesystem') if not self.isdir(path): if not self.isfile(path): raise errors.ResourceNotFoundError(path) raise errors.ResourceInvalidError(path) if not force and self.listdir(path): raise errors.DirectoryNotEmptyError(path) self.tahoeutil.unlink(self.dircap, path) if recursive and path != '/': try: self.removedir(dirname(path), recursive=True) except errors.DirectoryNotEmptyError: pass @_fix_path def makedir(self, path, recursive=False, allow_recreate=False): self._log(INFO, "Creating directory %s" % path) if self.getmeta("read_only"): raise errors.UnsupportedError('read only filesystem') if self.exists(path): if not self.isdir(path): raise errors.ResourceInvalidError(path) if not allow_recreate: raise errors.DestinationExistsError(path) if not recursive and not self.exists(dirname(path)): raise errors.ParentDirectoryMissingError(path) self.tahoeutil.mkdir(self.dircap, path) def movedir(self, src, dst, overwrite=False): self.move(src, dst, overwrite=overwrite) def move(self, src, dst, overwrite=False): self._log(INFO, "Moving file from %s to %s" % (src, dst)) if self.getmeta("read_only"): raise errors.UnsupportedError('read only filesystem') src = _fixpath(src) dst = _fixpath(dst) if not self.exists(dirname(dst)): raise errors.ParentDirectoryMissingError(dst) if not overwrite and self.exists(dst): raise errors.DestinationExistsError(dst) self.tahoeutil.move(self.dircap, src, dst) def rename(self, src, dst): self.move(src, dst) def copy(self, src, dst, overwrite=False, chunk_size=16384): if self.getmeta("read_only"): raise errors.UnsupportedError('read only filesystem') # FIXME: this is out of date; how to do native tahoe copy? # FIXME: Workaround because isfile() not exists on _TahoeLAFS FS.copy(self, src, dst, overwrite, chunk_size) def copydir(self, src, dst, overwrite=False, ignore_errors=False, chunk_size=16384): if self.getmeta("read_only"): raise errors.UnsupportedError('read only filesystem') # FIXME: this is out of date; how to do native tahoe copy? # FIXME: Workaround because isfile() not exists on _TahoeLAFS FS.copydir(self, src, dst, overwrite, ignore_errors, chunk_size) def _log(self, level, message): if not logger.isEnabledFor(level): return logger.log(level, u'(%d) %s' % (id(self), unicode(message).encode('ASCII', 'replace'))) @_fix_path def getpathurl(self, path, allow_none=False, webapi=None): ''' Retrieve URL where the file/directory is stored ''' if webapi == None: webapi = self.connection.webapi self._log(DEBUG, "Retrieving URL for %s over %s" % (path, webapi)) path = self.tahoeutil.fixwinpath(path, False) return u"%s/uri/%s%s" % (webapi, self.dircap, path) @_fix_path def getrange(self, path, offset, length=None): return self.connection.get(u'/uri/%s%s' % (self.dircap, path), offset=offset, length=length) @_fix_path def setcontents(self, path, file, chunk_size=64*1024): self._log(INFO, 'Uploading file %s' % path) size=None if self.getmeta("read_only"): raise errors.UnsupportedError('read only filesystem') # Workaround for large files: # First create zero file placeholder, then # upload final content. if self.largefilesize != None and getattr(file, 'read', None): # As 'file' can be also a string, need to check, # if 'file' looks like duck. Sorry, file. file.seek(0, SEEK_END) size = file.tell() file.seek(0) if size > self.largefilesize: self.connection.put(u'/uri/%s%s' % (self.dircap, path), "PyFilesystem.TahoeLAFS: Upload started, final size %d" % size) self.connection.put(u'/uri/%s%s' % (self.dircap, path), file, size=size) @_fix_path def getinfo(self, path): self._log(INFO, 'Reading meta for %s' % path) info = self.tahoeutil.info(self.dircap, path) #import datetime #info['created_time'] = datetime.datetime.now() #info['modified_time'] = datetime.datetime.now() #info['accessed_time'] = datetime.datetime.now() if info['type'] == 'filenode': info["st_mode"] = 0x700 | statinfo.S_IFREG elif info['type'] == 'dirnode': info["st_mode"] = 0x700 | statinfo.S_IFDIR return info class TahoeLAFS(CacheFSMixin,_TahoeLAFS): """FS providing cached access to a Tahoe Filesystem. This class is the preferred means to access a Tahoe filesystem. It maintains an internal cache of recently-accessed metadata to speed up operations. """ def __init__(self, *args, **kwds): kwds.setdefault("cache_timeout",60) super(TahoeLAFS,self).__init__(*args,**kwds) fs-0.5.4/fs/contrib/__init__.py0000664000175000017500000000010712512525115016250 0ustar willwill00000000000000""" fs.contrib: third-party contributed FS implementations. """ fs-0.5.4/fs/contrib/sqlitefs.py0000664000175000017500000006112412512525115016351 0ustar willwill00000000000000''' Sqlite3 based file system using PyFileSystem Developed and Contributed by Sensible Softwares Pvt. Ltd. (http://bootstraptoday.com) Contributed under the terms of the BSD License: http://www.opensource.org/licenses/bsd-license.php ''' import tempfile import datetime from fs.path import iteratepath, normpath,dirname,forcedir from fs.path import frombase, basename,pathjoin from fs.base import * from fs.errors import * from fs import _thread_synchronize_default import apsw def fetchone(cursor): ''' return a single row from the cursor (equivalent to pysqlite fetchone function) ''' row = None try: row = cursor.next() except: pass return(row) def remove_end_slash(dirname): if dirname.endswith('/'): return dirname[:-1] return dirname class SqliteFsFileBase(object): ''' base class for representing the files in the sqlite file system ''' def __init__(self, fs, path, id, real_file=None): assert(fs != None) assert(path != None) assert(id != None) self.fs = fs self.path = path self.id = id self.closed = False #real file like object. Most of the methods are passed to this object self.real_stream= real_file def close(self): if not self.closed and self.real_stream is not None: self._do_close() self.fs._on_close(self) self.real_stream.close() self.closed = True def __str__(self): return "" % (self.fs, self.path) __repr__ = __str__ def __unicode__(self): return u"" % (self.fs, self.path) def __del__(self): if not self.closed: self.close() def flush(self): self.real_stream.flush() def __iter__(self): raise OperationFailedError('__iter__', self.path) def next(self): raise OperationFailedError('next', self.path) def readline(self, *args, **kwargs): raise OperationFailedError('readline', self.path) def read(self, size=None): raise OperationFailedError('read', self.path) def seek(self, *args, **kwargs): return self.real_stream.seek(*args, **kwargs) def tell(self): return self.real_stream.tell() def truncate(self, *args, **kwargs): raise OperationFailedError('truncate', self.path) def write(self, data): raise OperationFailedError('write', self.path) def writelines(self, *args, **kwargs): raise OperationFailedError('writelines', self.path) def __enter__(self): return self def __exit__(self,exc_type,exc_value,traceback): self.close() return False class SqliteWritableFile(SqliteFsFileBase): ''' represents an sqlite file. Usually used for 'writing'. OnClose will actually 'copy the contents from temp disk file to sqlite blob ''' def __init__(self,fs, path, id): super(SqliteWritableFile, self).__init__(fs, path, id) #open a temp file and return that. self.real_stream = tempfile.SpooledTemporaryFile(max_size='128*1000') def _do_close(self): #push the contents of the file to blob self.fs._writeblob(self.id, self.real_stream) def truncate(self, *args, **kwargs): return self.real_stream.truncate(*args, **kwargs) def write(self, data): return self.real_stream.write(data) def writelines(self, *args, **kwargs): return self.real_stream.writelines(*args, **kwargs) class SqliteReadableFile(SqliteFsFileBase): def __init__(self,fs, path, id, real_file): super(SqliteReadableFile, self).__init__(fs, path, id, real_file) assert(self.real_stream != None) def _do_close(self): pass def __iter__(self): return iter(self.real_stream) def next(self): return self.real_stream.next() def readline(self, *args, **kwargs): return self.real_stream.readline(*args, **kwargs) def read(self, size=None): if( size==None): size=-1 return self.real_stream.read(size) class SqliteFS(FS): ''' sqlite based file system to store the files in sqlite database as 'blobs' We need two tables one to store file or directory meta data another for storing the file contain. Two are seperate so that same file can be refered from multiple directories. FsFileMetaData table : id : file id name : name of file parent : id of parent directory for the file. FsDirMetaData table: name : name of the directory (wihtout parent directory names) fullpath : full path of the directory including the parent directory name parent_id : id of the parent directory FsFileTable: size : file size in bytes (this is actual file size). Blob size may be different if compressed type : file type (extension or mime type) compression : For future use, Initially None. Later it will define type of compression used to compress the file last_modified : timestamp of last modification author : who changed it last content : blob where actual file contents are stored. TODO : Need an open files table or a flag in sqlite database. To avoid opening the file twice. (even from the different process or thread) ''' def __init__(self, sqlite_filename): super(SqliteFS, self).__init__() self.dbpath =sqlite_filename self.dbcon =None self.__actual_query_cur = None self.__actual_update_cur =None self.open_files = [] def close(self): ''' unlock all files. and close all open connections. ''' self.close_all() self._closedb() super(SqliteFS,self).close() def _initdb(self): if( self.dbcon is None): self.dbcon = apsw.Connection(self.dbpath) self._create_tables() @property def _querycur(self): assert(self.dbcon != None) if( self.__actual_query_cur == None): self.__actual_query_cur = self.dbcon.cursor() return(self.__actual_query_cur) @property def _updatecur(self): assert(self.dbcon != None) if( self.__actual_update_cur == None): self.__actual_update_cur = self.dbcon.cursor() return(self.__actual_update_cur) def _closedb(self): self.dbcon.close() def close_all(self): ''' close all open files ''' openfiles = list(self.open_files) for fileobj in openfiles: fileobj.close() def _create_tables(self): cur = self._updatecur cur.execute("CREATE TABLE IF NOT EXISTS FsFileMetaData(name text, fileid INTEGER, parent INTEGER)") cur.execute("CREATE TABLE IF NOT EXISTS FsDirMetaData(name text, fullpath TEXT, parentid INTEGER,\ created timestamp)") cur.execute("CREATE TABLE IF NOT EXISTS FsFileTable(type text, compression text, author TEXT, \ created timestamp, last_modified timestamp, last_accessed timestamp, \ locked BOOL, size INTEGER, contents BLOB)") #if the root directory name is created rootid = self._get_dir_id('/') if( rootid is None): cur.execute("INSERT INTO FsDirMetaData (name, fullpath) VALUES ('/','/')") def _get_dir_id(self, dirpath): ''' get the id for given directory path. ''' dirpath = remove_end_slash(dirpath) if( dirpath== None or len(dirpath)==0): dirpath = '/' self._querycur.execute("SELECT rowid from FsDirMetaData where fullpath=?",(dirpath,)) dirid = None dirrow = fetchone(self._querycur) if( dirrow): dirid = dirrow[0] return(dirid) def _get_file_id(self, dir_id, filename): ''' get the file id from the path ''' assert(dir_id != None) assert(filename != None) file_id = None self._querycur.execute("select rowid from FsFileMetaData where name=? and parent=?",(filename,dir_id)) row = fetchone(self._querycur) if( row ): file_id = row[0] return(file_id) def _get_file_contentid(self, file_id): ''' return the file content id from the 'content' table (i.e. FsFileTable) ''' assert(file_id != None) content_id = None self._querycur.execute("select fileid from FsFileMetaData where ROWID=?",(file_id,)) row = fetchone(self._querycur) assert(row != None) content_id = row[0] return(content_id) def _create_file_entry(self, dirid, filename, **kwargs): ''' create file entry in the file table ''' assert(dirid != None) assert(filename != None) #insert entry in file metadata table author = kwargs.pop('author', None) created = datetime.datetime.now().isoformat() last_modified = created compression = 'raw' size = 0 self._updatecur.execute("INSERT INTO FsFileTable(author, compression, size, created, last_modified) \ values(?, ?, ?, ?, ?)",(author, compression, size, created, last_modified)) content_id = self.dbcon.last_insert_rowid() #insert entry in file table self._updatecur.execute("INSERT INTO FsFileMetaData(name, parent, fileid) VALUES(?,?,?)",(filename, dirid, content_id)) #self.dbcon.commit() fileid = self.dbcon.last_insert_rowid() return(fileid) def _writeblob(self, fileid, stream): ''' extract the data from stream and write it as blob. ''' size = stream.tell() last_modified = datetime.datetime.now().isoformat() self._updatecur.execute('UPDATE FsFileTable SET size=?, last_modified=?, contents=? where rowid=?', (size, last_modified, apsw.zeroblob(size), fileid)) blob_stream=self.dbcon.blobopen("main", "FsFileTable", "contents", fileid, True) # 1 is for read/write stream.seek(0) blob_stream.write(stream.read()) blob_stream.close() def _on_close(self, fileobj): #Unlock file on close. assert(fileobj != None and fileobj.id != None) self._lockfileentry(fileobj.id, lock=False) #Now remove it from openfile list. self.open_files.remove(fileobj) def _islocked(self, fileid): ''' check if the file is locked. ''' locked=False if( fileid): content_id = self._get_file_contentid(fileid) assert(content_id != None) self._querycur.execute("select locked from FsFileTable where rowid=?",(content_id,)) row = fetchone(self._querycur) assert(row != None) locked = row[0] return(locked) def _lockfileentry(self, contentid, lock=True): ''' lock the file entry in the database. ''' assert(contentid != None) last_accessed=datetime.datetime.now().isoformat() self._updatecur.execute('UPDATE FsFileTable SET locked=?, last_accessed=? where rowid=?', (lock, last_accessed, contentid)) def _makedir(self, parent_id, dname): self._querycur.execute("SELECT fullpath from FsDirMetaData where rowid=?",(parent_id,)) row = fetchone(self._querycur) assert(row != None) parentpath = row[0] fullpath= pathjoin(parentpath, dname) fullpath= remove_end_slash(fullpath) created = datetime.datetime.now().isoformat() self._updatecur.execute('INSERT INTO FsDirMetaData(name, fullpath, parentid,created) \ VALUES(?,?,?,?)', (dname, fullpath, parent_id,created)) def _rename_file(self, src, dst): ''' rename source file 'src' to destination file 'dst' ''' srcdir = dirname(src) srcfname = basename(src) dstdir = dirname(dst) dstfname = basename(dst) #Make sure that the destination directory exists and destination file #doesnot exist. dstdirid = self._get_dir_id(dstdir) if( dstdirid == None): raise ParentDirectoryMissingError(dst) dstfile_id = self._get_file_id(dstdirid, dstfname) if( dstfile_id != None): raise DestinationExistsError(dst) #All checks are done. Delete the entry for the source file. #Create an entry for the destination file. srcdir_id = self._get_dir_id(srcdir) assert(srcdir_id != None) srcfile_id = self._get_file_id(srcdir_id, srcfname) assert(srcfile_id != None) srccontent_id = self._get_file_contentid(srcfile_id) self._updatecur.execute('DELETE FROM FsFileMetaData where ROWID=?',(srcfile_id,)) self._updatecur.execute("INSERT INTO FsFileMetaData(name, parent, fileid) \ VALUES(?,?,?)",(dstfname, dstdirid, srccontent_id)) def _rename_dir(self, src, dst): src = remove_end_slash(src) dst = remove_end_slash(dst) dstdirid = self._get_dir_id(dst) if( dstdirid != None): raise DestinationExistsError(dst) dstparent = dirname(dst) dstparentid = self._get_dir_id(dstparent) if(dstparentid == None): raise ParentDirectoryMissingError(dst) srcdirid = self._get_dir_id(src) assert(srcdirid != None) dstdname = basename(dst) self._updatecur.execute('UPDATE FsDirMetaData SET name=?, fullpath=?, \ parentid=? where ROWID=?',(dstdname, dst, dstparentid, srcdirid,)) def _get_dir_list(self, dirid, path, full): assert(dirid != None) assert(path != None) if( full==True): dirsearchpath = path + r'%' self._querycur.execute('SELECT fullpath FROM FsDirMetaData where fullpath LIKE ?', (dirsearchpath,)) else: #search inside this directory only self._querycur.execute('SELECT fullpath FROM FsDirMetaData where parentid=?', (dirid,)) dirlist = [row[0] for row in self._querycur] return dirlist def _get_file_list(self, dirpath, full): assert(dirpath != None) if( full==True): searchpath = dirpath + r"%" self._querycur.execute('SELECT FsFileMetaData.name, FsDirMetaData.fullpath \ FROM FsFileMetaData, FsDirMetaData where FsFileMetaData.parent=FsDirMetaData.ROWID \ and FsFileMetaData.parent in (SELECT rowid FROM FsDirMetaData \ where fullpath LIKE ?)',(searchpath,)) else: parentid = self._get_dir_id(dirpath) self._querycur.execute('SELECT FsFileMetaData.name, FsDirMetaData.fullpath \ FROM FsFileMetaData, FsDirMetaData where FsFileMetaData.parent=FsDirMetaData.ROWID \ and FsFileMetaData.parent =?',(parentid,)) filelist = [pathjoin(row[1],row[0]) for row in self._querycur] return(filelist) def _get_dir_info(self, path): ''' get the directory information dictionary. ''' info = dict() info['st_mode'] = 0755 return info def _get_file_info(self, path): filedir = dirname(path) filename = basename(path) dirid = self._get_dir_id(filedir) assert(dirid is not None) fileid = self._get_file_id(dirid, filename) assert(fileid is not None) contentid = self._get_file_contentid(fileid) assert(contentid is not None) self._querycur.execute('SELECT author, size, created, last_modified, last_accessed \ FROM FsFileTable where rowid=?',(contentid,)) row = fetchone(self._querycur) assert(row != None) info = dict() info['author'] = row[0] info['size'] = row[1] info['created'] = row[2] info['last_modified'] = row[3] info['last_accessed'] = row[4] info['st_mode'] = 0666 return(info) def _isfile(self,path): path = normpath(path) filedir = dirname(path) filename = basename(path) dirid = self._get_dir_id(filedir) return(dirid is not None and self._get_file_id(dirid, filename) is not None) def _isdir(self,path): path = normpath(path) return(self._get_dir_id(path) is not None) def _isexist(self,path): return self._isfile(path) or self._isdir(path) @synchronize def open(self, path, mode='r', **kwargs): self._initdb() path = normpath(path) filedir = dirname(path) filename = basename(path) dir_id = self._get_dir_id(filedir) if( dir_id == None): raise ResourceNotFoundError(filedir) file_id = self._get_file_id(dir_id, filename) if( self._islocked(file_id)): raise ResourceLockedError(path) sqfsfile=None if 'r' in mode: if file_id is None: raise ResourceNotFoundError(path) content_id = self._get_file_contentid(file_id) #make sure lock status is updated before the blob is opened self._lockfileentry(content_id, lock=True) blob_stream=self.dbcon.blobopen("main", "FsFileTable", "contents", file_id, False) # 1 is for read/write sqfsfile = SqliteReadableFile(self, path, content_id, blob_stream) elif 'w' in mode or 'a' in mode: if( file_id is None): file_id= self._create_file_entry(dir_id, filename) assert(file_id != None) content_id = self._get_file_contentid(file_id) #file_dir_entry.accessed_time = datetime.datetime.now() self._lockfileentry(content_id, lock=True) sqfsfile = SqliteWritableFile(self, path, content_id) if( sqfsfile): self.open_files.append(sqfsfile) return sqfsfile raise ResourceNotFoundError(path) @synchronize def isfile(self, path): self._initdb() return self._isfile(path) @synchronize def isdir(self, path): self._initdb() return self._isdir(path) @synchronize def listdir(self, path='/', wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False): path = normpath(path) dirid = self._get_dir_id(path) if( dirid == None): raise ResourceInvalidError(path) dirlist = self._get_dir_list(dirid, path,full) if( dirs_only): pathlist = dirlist else: filelist = self._get_file_list(path, full) if( files_only == True): pathlist = filelist else: pathlist = filelist + dirlist if( wildcard and dirs_only == False): pass if( absolute == False): pathlist = map(lambda dpath:frombase(path,dpath), pathlist) return(pathlist) @synchronize def makedir(self, path, recursive=False, allow_recreate=False): self._initdb() path = remove_end_slash(normpath(path)) if(self._isexist(path)==False): parentdir = dirname(path) dname = basename(path) parent_id = self._get_dir_id(parentdir) if( parent_id ==None): if( recursive == False): raise ParentDirectoryMissingError(path) else: self.makedir(parentdir, recursive,allow_recreate) parent_id = self._get_dir_id(parentdir) self._makedir(parent_id,dname) else: raise DestinationExistsError(path) @synchronize def remove(self, path): self._initdb() path = normpath(path) if( self.isdir(path)==True): #path is actually a directory raise ResourceInvalidError(path) filedir = dirname(path) filename = basename(path) dirid = self._get_dir_id(filedir) fileid = self._get_file_id(dirid, filename) if( fileid == None): raise ResourceNotFoundError(path) content_id = self._get_file_contentid(fileid) self._updatecur.execute("DELETE FROM FsFileMetaData where ROWID=?",(fileid,)) #check there is any other file pointing to same location. If not #delete the content as well. self._querycur.execute('SELECT count(*) FROM FsFileMetaData where fileid=?', (content_id,)) row = fetchone(self._querycur) if( row == None or row[0] == 0): self._updatecur.execute("DELETE FROM FsFileTable where ROWID=?",(content_id,)) @synchronize def removedir(self,path, recursive=False, force=False): self._initdb() path = normpath(path) if( self.isfile(path)==True): #path is actually a file raise ResourceInvalidError(path) dirid = self._get_dir_id(path) if( dirid == None): raise ResourceNotFoundError(path) #check if there are any files in this directory self._querycur.execute("SELECT COUNT(*) FROM FsFileMetaData where parent=?",(dirid,)) row = fetchone(self._qurycur) if( row[0] > 0): raise DirectoryNotEmptyError(path) self._updatecur.execute("DELETE FROM FsDirMetaData where ROWID=?",(dirid,)) @synchronize def rename(self,src, dst): self._initdb() src = normpath(src) dst = normpath(dst) if self._isexist(dst)== False: #first check if this is a directory rename or a file rename if( self.isfile(src)): self._rename_file(src, dst) elif self.isdir(src): self._rename_dir(src, dst) else: raise ResourceNotFoundError(path) else: raise DestinationExistsError(dst) @synchronize def getinfo(self, path): self._initdb() path = normpath(path) isfile = False isdir = self.isdir(path) if( isdir == False): isfile=self.isfile(path) if( not isfile and not isdir): raise ResourceNotFoundError(path) if isdir: info= self._get_dir_info(path) else: info= self._get_file_info(path) return(info) #import msvcrt # built-in module # #def kbfunc(): # return ord(msvcrt.getch()) if msvcrt.kbhit() else 0 # #def mount_windows(sqlfilename, driveletter): # sqfs = SqliteFS(sqlfilename) # from fs.expose import dokan # #mp = dokan.mount(sqfs,driveletter,foreground=True) # #mp.unmount() # sqfs.close() # #def run_tests(sqlfilename): # fs = SqliteFS(sqlfilename) # fs.makedir('/test') # f = fs.open('/test/test.txt', "w") # f.write("This is a test") # f.close() # f = fs.open('/test/test.txt', "r") # contents = f.read() # print contents # f.close() # print "testing file rename" # fs.rename('/test/test.txt', '/test/test1.txt') # f = fs.open('/test/test1.txt', "r") # print contents # f.close() # print "done testing file rename" # print "testing directory rename" # fs.rename('/test', '/test1') # f = fs.open('/test1/test1.txt', "r") # contents = f.read() # print contents # f.close() # print "done testing directory rename" # flist = fs.listdir('/', full=True,absolute=True,files_only=True) # print flist # fs.close() # #if __name__ == '__main__': # run_tests("sqfs.sqlite") # mount_windows("sqfs.sqlite", 'm') # # #fs.remove('/test1/test1.txt') # #try: # # f = fs.open('/test1/test1.txt', "r") # #except ResourceNotFoundError: # # print "Success : file doesnot exist" # #fs.browse() # fs-0.5.4/fs/tempfs.py0000664000175000017500000001022512512525115014351 0ustar willwill00000000000000""" fs.tempfs ========= Make a temporary file system that exists in a folder provided by the OS. All files contained in a TempFS are removed when the `close` method is called (or when the TempFS is cleaned up by Python). """ import os import os.path import time import tempfile from fs.base import synchronize from fs.osfs import OSFS from fs.errors import * from fs import _thread_synchronize_default class TempFS(OSFS): """Create a Filesystem in a temporary directory (with tempfile.mkdtemp), and removes it when the TempFS object is cleaned up.""" _meta = dict(OSFS._meta) _meta['pickle_contents'] = False _meta['network'] = False _meta['atomic.move'] = True _meta['atomic.copy'] = True def __init__(self, identifier=None, temp_dir=None, dir_mode=0700, thread_synchronize=_thread_synchronize_default): """Creates a temporary Filesystem identifier -- A string that is included in the name of the temporary directory, default uses "TempFS" """ self.identifier = identifier self.temp_dir = temp_dir self.dir_mode = dir_mode self._temp_dir = tempfile.mkdtemp(identifier or "TempFS", dir=temp_dir) self._cleaned = False super(TempFS, self).__init__(self._temp_dir, dir_mode=dir_mode, thread_synchronize=thread_synchronize) def __repr__(self): return '' % self._temp_dir __str__ = __repr__ def __unicode__(self): return u'' % self._temp_dir def __getstate__(self): # If we are picking a TempFS, we want to preserve its contents, # so we *don't* do the clean state = super(TempFS, self).__getstate__() self._cleaned = True return state def __setstate__(self, state): state = super(TempFS, self).__setstate__(state) self._cleaned = False #self._temp_dir = tempfile.mkdtemp(self.identifier or "TempFS", dir=self.temp_dir) #super(TempFS, self).__init__(self._temp_dir, # dir_mode=self.dir_mode, # thread_synchronize=self.thread_synchronize) @synchronize def close(self): """Removes the temporary directory. This will be called automatically when the object is cleaned up by Python, although it is advisable to call it manually. Note that once this method has been called, the FS object may no longer be used. """ super(TempFS, self).close() # Depending on how resources are freed by the OS, there could # be some transient errors when freeing a TempFS soon after it # was used. If they occur, do a small sleep and try again. try: self._close() except (ResourceLockedError, ResourceInvalidError): time.sleep(0.5) self._close() @convert_os_errors def _close(self): """Actual implementation of close(). This is a separate method so it can be re-tried in the face of transient errors. """ os_remove = convert_os_errors(os.remove) os_rmdir = convert_os_errors(os.rmdir) if not self._cleaned and self.exists("/"): self._lock.acquire() try: # shutil.rmtree doesn't handle long paths on win32, # so we walk the tree by hand. entries = os.walk(self.root_path, topdown=False) for (dir, dirnames, filenames) in entries: for filename in filenames: try: os_remove(os.path.join(dir, filename)) except ResourceNotFoundError: pass for dirname in dirnames: try: os_rmdir(os.path.join(dir, dirname)) except ResourceNotFoundError: pass try: os.rmdir(self.root_path) except OSError: pass self._cleaned = True finally: self._lock.release() super(TempFS, self).close() fs-0.5.4/fs/memoryfs.py0000664000175000017500000005640712512525115014730 0ustar willwill00000000000000#!/usr/bin/env python """ fs.memoryfs =========== A Filesystem that exists in memory only. Which makes them extremely fast, but non-permanent. If you open a file from a `memoryfs` you will get back a StringIO object from the standard library. """ import datetime import stat from fs.path import iteratepath, pathsplit, normpath from fs.base import * from fs.errors import * from fs import _thread_synchronize_default from fs.filelike import StringIO from fs import iotools from os import SEEK_END import threading import six from six import b def _check_mode(mode, mode_chars): for c in mode_chars: if c not in mode: return False return True class MemoryFile(object): def seek_and_lock(f): def deco(self, *args, **kwargs): try: self._lock.acquire() self.mem_file.seek(self.pos) ret = f(self, *args, **kwargs) self.pos = self.mem_file.tell() return ret finally: self._lock.release() return deco def __init__(self, path, memory_fs, mem_file, mode, lock): self.closed = False self.path = path self.memory_fs = memory_fs self.mem_file = mem_file self.mode = mode self._lock = lock self.pos = 0 if _check_mode(mode, 'a'): lock.acquire() try: self.mem_file.seek(0, SEEK_END) self.pos = self.mem_file.tell() finally: lock.release() elif _check_mode(mode, 'w'): lock.acquire() try: self.mem_file.seek(0) self.mem_file.truncate() finally: lock.release() assert self.mem_file is not None, "self.mem_file should have a value" def __str__(self): return "" % (self.memory_fs, self.path) def __repr__(self): return u"" % (self.memory_fs, self.path) def __unicode__(self): return u"" % (self.memory_fs, self.path) def __del__(self): if not self.closed: self.close() def flush(self): pass def __iter__(self): if 'r' not in self.mode and '+' not in self.mode: raise IOError("File not open for reading") self.mem_file.seek(self.pos) for line in self.mem_file: yield line @seek_and_lock def next(self): if 'r' not in self.mode and '+' not in self.mode: raise IOError("File not open for reading") return self.mem_file.next() @seek_and_lock def readline(self, *args, **kwargs): if 'r' not in self.mode and '+' not in self.mode: raise IOError("File not open for reading") return self.mem_file.readline(*args, **kwargs) def close(self): do_close = False self._lock.acquire() try: do_close = not self.closed and self.mem_file is not None if do_close: self.closed = True finally: self._lock.release() if do_close: self.memory_fs._on_close_memory_file(self, self.path) @seek_and_lock def read(self, size=None): if 'r' not in self.mode and '+' not in self.mode: raise IOError("File not open for reading") if size is None: size = -1 return self.mem_file.read(size) @seek_and_lock def seek(self, *args, **kwargs): return self.mem_file.seek(*args, **kwargs) @seek_and_lock def tell(self): return self.pos @seek_and_lock def truncate(self, *args, **kwargs): if 'r' in self.mode and '+' not in self.mode: raise IOError("File not open for writing") return self.mem_file.truncate(*args, **kwargs) #@seek_and_lock def write(self, data): if 'r' in self.mode and '+' not in self.mode: raise IOError("File not open for writing") self.memory_fs._on_modify_memory_file(self.path) self._lock.acquire() try: self.mem_file.seek(self.pos) self.mem_file.write(data) self.pos = self.mem_file.tell() finally: self._lock.release() @seek_and_lock def writelines(self, *args, **kwargs): return self.mem_file.writelines(*args, **kwargs) def __enter__(self): return self def __exit__(self, exc_type, exc_value, traceback): self.close() return False class DirEntry(object): def sync(f): def deco(self, *args, **kwargs): if self.lock is not None: try: self.lock.acquire() return f(self, *args, **kwargs) finally: self.lock.release() else: return f(self, *args, **kwargs) return deco def __init__(self, type, name, contents=None): assert type in ("dir", "file"), "Type must be dir or file!" self.type = type self.name = name if contents is None and type == "dir": contents = {} self.open_files = [] self.contents = contents self.mem_file = None self.created_time = datetime.datetime.now() self.modified_time = self.created_time self.accessed_time = self.created_time self.xattrs = {} self.lock = None if self.type == 'file': self.mem_file = StringIO() self.lock = threading.RLock() def get_value(self): self.lock.acquire() try: return self.mem_file.getvalue() finally: self.lock.release() data = property(get_value) def desc_contents(self): if self.isfile(): return "" % self.name elif self.isdir(): return "" % "".join("%s: %s" % (k, v.desc_contents()) for k, v in self.contents.iteritems()) def isdir(self): return self.type == "dir" def isfile(self): return self.type == "file" def __str__(self): return "%s: %s" % (self.name, self.desc_contents()) @sync def __getstate__(self): state = self.__dict__.copy() state.pop('lock') if self.mem_file is not None: state['mem_file'] = self.data return state def __setstate__(self, state): self.__dict__.update(state) if self.type == 'file': self.lock = threading.RLock() else: self.lock = None if self.mem_file is not None: data = self.mem_file self.mem_file = StringIO() self.mem_file.write(data) class MemoryFS(FS): """An in-memory filesystem. """ _meta = {'thread_safe': True, 'network': False, 'virtual': False, 'read_only': False, 'unicode_paths': True, 'case_insensitive_paths': False, 'atomic.move': False, 'atomic.copy': False, 'atomic.makedir': True, 'atomic.rename': True, 'atomic.setcontents': False} def _make_dir_entry(self, *args, **kwargs): return self.dir_entry_factory(*args, **kwargs) def __init__(self, file_factory=None): super(MemoryFS, self).__init__(thread_synchronize=_thread_synchronize_default) self.dir_entry_factory = DirEntry self.file_factory = file_factory or MemoryFile if not callable(self.file_factory): raise ValueError("file_factory should be callable") self.root = self._make_dir_entry('dir', 'root') def __str__(self): return "" def __repr__(self): return "MemoryFS()" def __unicode__(self): return "" @synchronize def _get_dir_entry(self, dirpath): dirpath = normpath(dirpath) current_dir = self.root for path_component in iteratepath(dirpath): if current_dir.contents is None: return None dir_entry = current_dir.contents.get(path_component, None) if dir_entry is None: return None current_dir = dir_entry return current_dir @synchronize def _dir_entry(self, path): dir_entry = self._get_dir_entry(path) if dir_entry is None: raise ResourceNotFoundError(path) return dir_entry @synchronize def desc(self, path): if self.isdir(path): return "Memory dir" elif self.isfile(path): return "Memory file object" else: return "No description available" @synchronize def isdir(self, path): path = normpath(path) if path in ('', '/'): return True dir_item = self._get_dir_entry(path) if dir_item is None: return False return dir_item.isdir() @synchronize def isfile(self, path): path = normpath(path) if path in ('', '/'): return False dir_item = self._get_dir_entry(path) if dir_item is None: return False return dir_item.isfile() @synchronize def exists(self, path): path = normpath(path) if path in ('', '/'): return True return self._get_dir_entry(path) is not None @synchronize def makedir(self, dirname, recursive=False, allow_recreate=False): if not dirname and not allow_recreate: raise PathError(dirname) fullpath = normpath(dirname) if fullpath in ('', '/'): if allow_recreate: return raise DestinationExistsError(dirname) dirpath, dirname = pathsplit(dirname.rstrip('/')) if recursive: parent_dir = self._get_dir_entry(dirpath) if parent_dir is not None: if parent_dir.isfile(): raise ResourceInvalidError(dirname, msg="Can not create a directory, because path references a file: %(path)s") else: if not allow_recreate: if dirname in parent_dir.contents: raise DestinationExistsError(dirname, msg="Can not create a directory that already exists (try allow_recreate=True): %(path)s") current_dir = self.root for path_component in iteratepath(dirpath)[:-1]: dir_item = current_dir.contents.get(path_component, None) if dir_item is None: break if not dir_item.isdir(): raise ResourceInvalidError(dirname, msg="Can not create a directory, because path references a file: %(path)s") current_dir = dir_item current_dir = self.root for path_component in iteratepath(dirpath): dir_item = current_dir.contents.get(path_component, None) if dir_item is None: new_dir = self._make_dir_entry("dir", path_component) current_dir.contents[path_component] = new_dir current_dir = new_dir else: current_dir = dir_item parent_dir = current_dir else: parent_dir = self._get_dir_entry(dirpath) if parent_dir is None: raise ParentDirectoryMissingError(dirname, msg="Could not make dir, as parent dir does not exist: %(path)s") dir_item = parent_dir.contents.get(dirname, None) if dir_item is not None: if dir_item.isdir(): if not allow_recreate: raise DestinationExistsError(dirname) else: raise ResourceInvalidError(dirname, msg="Can not create a directory, because path references a file: %(path)s") if dir_item is None: parent_dir.contents[dirname] = self._make_dir_entry("dir", dirname) #@synchronize #def _orphan_files(self, file_dir_entry): # for f in file_dir_entry.open_files[:]: # f.close() @synchronize @iotools.filelike_to_stream def open(self, path, mode='r', buffering=-1, encoding=None, errors=None, newline=None, line_buffering=False, **kwargs): path = normpath(path) filepath, filename = pathsplit(path) parent_dir_entry = self._get_dir_entry(filepath) if parent_dir_entry is None or not parent_dir_entry.isdir(): raise ResourceNotFoundError(path) if 'r' in mode or 'a' in mode: if filename not in parent_dir_entry.contents: raise ResourceNotFoundError(path) file_dir_entry = parent_dir_entry.contents[filename] if file_dir_entry.isdir(): raise ResourceInvalidError(path) file_dir_entry.accessed_time = datetime.datetime.now() mem_file = self.file_factory(path, self, file_dir_entry.mem_file, mode, file_dir_entry.lock) file_dir_entry.open_files.append(mem_file) return mem_file elif 'w' in mode: if filename not in parent_dir_entry.contents: file_dir_entry = self._make_dir_entry("file", filename) parent_dir_entry.contents[filename] = file_dir_entry else: file_dir_entry = parent_dir_entry.contents[filename] file_dir_entry.accessed_time = datetime.datetime.now() mem_file = self.file_factory(path, self, file_dir_entry.mem_file, mode, file_dir_entry.lock) file_dir_entry.open_files.append(mem_file) return mem_file if parent_dir_entry is None: raise ResourceNotFoundError(path) @synchronize def remove(self, path): dir_entry = self._get_dir_entry(path) if dir_entry is None: raise ResourceNotFoundError(path) if dir_entry.isdir(): raise ResourceInvalidError(path, msg="That's a directory, not a file: %(path)s") pathname, dirname = pathsplit(path) parent_dir = self._get_dir_entry(pathname) del parent_dir.contents[dirname] @synchronize def removedir(self, path, recursive=False, force=False): path = normpath(path) if path in ('', '/'): raise RemoveRootError(path) dir_entry = self._get_dir_entry(path) if dir_entry is None: raise ResourceNotFoundError(path) if not dir_entry.isdir(): raise ResourceInvalidError(path, msg="Can't remove resource, its not a directory: %(path)s" ) if dir_entry.contents and not force: raise DirectoryNotEmptyError(path) if recursive: rpathname = path while rpathname: rpathname, dirname = pathsplit(rpathname) parent_dir = self._get_dir_entry(rpathname) if not dirname: raise RemoveRootError(path) del parent_dir.contents[dirname] # stop recursing if the directory has other contents if parent_dir.contents: break else: pathname, dirname = pathsplit(path) parent_dir = self._get_dir_entry(pathname) if not dirname: raise RemoveRootError(path) del parent_dir.contents[dirname] @synchronize def rename(self, src, dst): src = normpath(src) dst = normpath(dst) src_dir, src_name = pathsplit(src) src_entry = self._get_dir_entry(src) if src_entry is None: raise ResourceNotFoundError(src) open_files = src_entry.open_files[:] for f in open_files: f.flush() f.path = dst dst_dir,dst_name = pathsplit(dst) dst_entry = self._get_dir_entry(dst) if dst_entry is not None: raise DestinationExistsError(dst) src_dir_entry = self._get_dir_entry(src_dir) src_xattrs = src_dir_entry.xattrs.copy() dst_dir_entry = self._get_dir_entry(dst_dir) if dst_dir_entry is None: raise ParentDirectoryMissingError(dst) dst_dir_entry.contents[dst_name] = src_dir_entry.contents[src_name] dst_dir_entry.contents[dst_name].name = dst_name dst_dir_entry.xattrs.update(src_xattrs) del src_dir_entry.contents[src_name] @synchronize def settimes(self, path, accessed_time=None, modified_time=None): now = datetime.datetime.now() if accessed_time is None: accessed_time = now if modified_time is None: modified_time = now dir_entry = self._get_dir_entry(path) if dir_entry is not None: dir_entry.accessed_time = accessed_time dir_entry.modified_time = modified_time return True return False @synchronize def _on_close_memory_file(self, open_file, path): dir_entry = self._get_dir_entry(path) if dir_entry is not None and open_file in dir_entry.open_files: dir_entry.open_files.remove(open_file) @synchronize def _on_modify_memory_file(self, path): dir_entry = self._get_dir_entry(path) if dir_entry is not None: dir_entry.modified_time = datetime.datetime.now() @synchronize def listdir(self, path="/", wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False): dir_entry = self._get_dir_entry(path) if dir_entry is None: raise ResourceNotFoundError(path) if dir_entry.isfile(): raise ResourceInvalidError(path, msg="not a directory: %(path)s") paths = dir_entry.contents.keys() for (i,p) in enumerate(paths): if not isinstance(p,unicode): paths[i] = unicode(p) return self._listdir_helper(path, paths, wildcard, full, absolute, dirs_only, files_only) @synchronize def getinfo(self, path): dir_entry = self._get_dir_entry(path) if dir_entry is None: raise ResourceNotFoundError(path) info = {} info['created_time'] = dir_entry.created_time info['modified_time'] = dir_entry.modified_time info['accessed_time'] = dir_entry.accessed_time if dir_entry.isdir(): info['st_mode'] = 0755 | stat.S_IFDIR else: info['size'] = len(dir_entry.data or b('')) info['st_mode'] = 0666 | stat.S_IFREG return info @synchronize def copydir(self, src, dst, overwrite=False, ignore_errors=False, chunk_size=1024*64): src_dir_entry = self._get_dir_entry(src) if src_dir_entry is None: raise ResourceNotFoundError(src) src_xattrs = src_dir_entry.xattrs.copy() super(MemoryFS, self).copydir(src, dst, overwrite, ignore_errors=ignore_errors, chunk_size=chunk_size) dst_dir_entry = self._get_dir_entry(dst) if dst_dir_entry is not None: dst_dir_entry.xattrs.update(src_xattrs) @synchronize def movedir(self, src, dst, overwrite=False, ignore_errors=False, chunk_size=1024*64): src_dir_entry = self._get_dir_entry(src) if src_dir_entry is None: raise ResourceNotFoundError(src) src_xattrs = src_dir_entry.xattrs.copy() super(MemoryFS, self).movedir(src, dst, overwrite, ignore_errors=ignore_errors, chunk_size=chunk_size) dst_dir_entry = self._get_dir_entry(dst) if dst_dir_entry is not None: dst_dir_entry.xattrs.update(src_xattrs) @synchronize def copy(self, src, dst, overwrite=False, chunk_size=1024*64): src_dir_entry = self._get_dir_entry(src) if src_dir_entry is None: raise ResourceNotFoundError(src) src_xattrs = src_dir_entry.xattrs.copy() super(MemoryFS, self).copy(src, dst, overwrite, chunk_size) dst_dir_entry = self._get_dir_entry(dst) if dst_dir_entry is not None: dst_dir_entry.xattrs.update(src_xattrs) @synchronize def move(self, src, dst, overwrite=False, chunk_size=1024*64): src_dir_entry = self._get_dir_entry(src) if src_dir_entry is None: raise ResourceNotFoundError(src) src_xattrs = src_dir_entry.xattrs.copy() super(MemoryFS, self).move(src, dst, overwrite, chunk_size) dst_dir_entry = self._get_dir_entry(dst) if dst_dir_entry is not None: dst_dir_entry.xattrs.update(src_xattrs) @synchronize def getcontents(self, path, mode="rb", encoding=None, errors=None, newline=None): dir_entry = self._get_dir_entry(path) if dir_entry is None: raise ResourceNotFoundError(path) if not dir_entry.isfile(): raise ResourceInvalidError(path, msg="not a file: %(path)s") data = dir_entry.data or b('') if 'b' not in mode: return iotools.decode_binary(data, encoding=encoding, errors=errors, newline=newline) return data @synchronize def setcontents(self, path, data=b'', encoding=None, errors=None, chunk_size=1024*64): if isinstance(data, six.binary_type): if not self.exists(path): self.open(path, 'wb').close() dir_entry = self._get_dir_entry(path) if not dir_entry.isfile(): raise ResourceInvalidError('Not a directory %(path)s', path) new_mem_file = StringIO() new_mem_file.write(data) dir_entry.mem_file = new_mem_file return len(data) return super(MemoryFS, self).setcontents(path, data=data, encoding=encoding, errors=errors, chunk_size=chunk_size) # if isinstance(data, six.text_type): # return super(MemoryFS, self).setcontents(path, data, encoding=encoding, errors=errors, chunk_size=chunk_size) # if not self.exists(path): # self.open(path, 'wb').close() # dir_entry = self._get_dir_entry(path) # if not dir_entry.isfile(): # raise ResourceInvalidError('Not a directory %(path)s', path) # new_mem_file = StringIO() # new_mem_file.write(data) # dir_entry.mem_file = new_mem_file @synchronize def setxattr(self, path, key, value): dir_entry = self._dir_entry(path) key = unicode(key) dir_entry.xattrs[key] = value @synchronize def getxattr(self, path, key, default=None): key = unicode(key) dir_entry = self._dir_entry(path) return dir_entry.xattrs.get(key, default) @synchronize def delxattr(self, path, key): dir_entry = self._dir_entry(path) try: del dir_entry.xattrs[key] except KeyError: pass @synchronize def listxattrs(self, path): dir_entry = self._dir_entry(path) return dir_entry.xattrs.keys() fs-0.5.4/fs/sftpfs.py0000664000175000017500000005527312512525115014374 0ustar willwill00000000000000""" fs.sftpfs ========= **Currently only avaiable on Python2 due to paramiko not being available for Python3** Filesystem accessing an SFTP server (via paramiko) """ import datetime import stat as statinfo import threading import os import paramiko from getpass import getuser import errno from fs.base import * from fs.path import * from fs.errors import * from fs.utils import isdir, isfile from fs import iotools ENOENT = errno.ENOENT class WrongHostKeyError(RemoteConnectionError): pass # SFTPClient appears to not be thread-safe, so we use an instance per thread if hasattr(threading, "local"): thread_local = threading.local #class TL(object): # pass #thread_local = TL else: class thread_local(object): def __init__(self): self._map = {} def __getattr__(self, attr): try: return self._map[(threading.currentThread().ident, attr)] except KeyError: raise AttributeError(attr) def __setattr__(self, attr, value): self._map[(threading.currentThread().ident, attr)] = value if not hasattr(paramiko.SFTPFile, "__enter__"): paramiko.SFTPFile.__enter__ = lambda self: self paramiko.SFTPFile.__exit__ = lambda self,et,ev,tb: self.close() and False class SFTPFS(FS): """A filesystem stored on a remote SFTP server. This is basically a compatibility wrapper for the excellent SFTPClient class in the paramiko module. """ _meta = { 'thread_safe' : True, 'virtual': False, 'read_only' : False, 'unicode_paths' : True, 'case_insensitive_paths' : False, 'network' : True, 'atomic.move' : True, 'atomic.copy' : True, 'atomic.makedir' : True, 'atomic.rename' : True, 'atomic.setcontents' : False } def __init__(self, connection, root_path="/", encoding=None, hostkey=None, username='', password=None, pkey=None, agent_auth=True, no_auth=False, look_for_keys=True): """SFTPFS constructor. The only required argument is 'connection', which must be something from which we can construct a paramiko.SFTPClient object. Possible values include: * a hostname string * a (hostname,port) tuple * a paramiko.Transport instance * a paramiko.Channel instance in "sftp" mode The keyword argument 'root_path' specifies the root directory on the remote machine - access to files outside this root will be prevented. :param connection: a connection string :param root_path: The root path to open :param encoding: String encoding of paths (defaults to UTF-8) :param hostkey: the host key expected from the server or None if you don't require server validation :param username: Name of SFTP user :param password: Password for SFTP user :param pkey: Public key :param agent_auth: attempt to authorize with the user's public keys :param no_auth: attempt to log in without any kind of authorization :param look_for_keys: Look for keys in the same locations as ssh, if other authentication is not succesful """ credentials = dict(username=username, password=password, pkey=pkey) self.credentials = credentials if encoding is None: encoding = "utf8" self.encoding = encoding self.closed = False self._owns_transport = False self._credentials = credentials self._tlocal = thread_local() self._transport = None self._client = None self.hostname = None if isinstance(connection, basestring): self.hostname = connection elif isinstance(connection, tuple): self.hostname = '%s:%s' % connection super(SFTPFS, self).__init__() self.root_path = abspath(normpath(root_path)) if isinstance(connection,paramiko.Channel): self._transport = None self._client = paramiko.SFTPClient(connection) else: if not isinstance(connection,paramiko.Transport): connection = paramiko.Transport(connection) connection.daemon = True self._owns_transport = True if hostkey is not None: key = self.get_remote_server_key() if hostkey != key: raise WrongHostKeyError('Host keys do not match') connection.start_client() if not connection.is_active(): raise RemoteConnectionError(msg='Unable to connect') if no_auth: try: connection.auth_none('') except paramiko.SSHException: pass elif not connection.is_authenticated(): if not username: username = getuser() try: if pkey: connection.auth_publickey(username, pkey) if not connection.is_authenticated() and password: connection.auth_password(username, password) if agent_auth and not connection.is_authenticated(): self._agent_auth(connection, username) if look_for_keys and not connection.is_authenticated(): self._userkeys_auth(connection, username, password) if not connection.is_authenticated(): try: connection.auth_none(username) except paramiko.BadAuthenticationType, e: self.close() allowed = ', '.join(e.allowed_types) raise RemoteConnectionError(msg='no auth - server requires one of the following: %s' % allowed, details=e) if not connection.is_authenticated(): self.close() raise RemoteConnectionError(msg='no auth') except paramiko.SSHException, e: self.close() raise RemoteConnectionError(msg='SSH exception (%s)' % str(e), details=e) self._transport = connection def __unicode__(self): return u'' % self.desc('/') @classmethod def _agent_auth(cls, transport, username): """ Attempt to authenticate to the given transport using any of the private keys available from an SSH agent. """ agent = paramiko.Agent() agent_keys = agent.get_keys() if not agent_keys: return None for key in agent_keys: try: transport.auth_publickey(username, key) return key except paramiko.SSHException: pass return None @classmethod def _userkeys_auth(cls, transport, username, password): """ Attempt to authenticate to the given transport using any of the private keys in the users ~/.ssh and ~/ssh dirs Derived from http://www.lag.net/paramiko/docs/paramiko.client-pysrc.html """ keyfiles = [] rsa_key = os.path.expanduser('~/.ssh/id_rsa') dsa_key = os.path.expanduser('~/.ssh/id_dsa') if os.path.isfile(rsa_key): keyfiles.append((paramiko.rsakey.RSAKey, rsa_key)) if os.path.isfile(dsa_key): keyfiles.append((paramiko.dsskey.DSSKey, dsa_key)) # look in ~/ssh/ for windows users: rsa_key = os.path.expanduser('~/ssh/id_rsa') dsa_key = os.path.expanduser('~/ssh/id_dsa') if os.path.isfile(rsa_key): keyfiles.append((paramiko.rsakey.RSAKey, rsa_key)) if os.path.isfile(dsa_key): keyfiles.append((paramiko.dsskey.DSSKey, dsa_key)) for pkey_class, filename in keyfiles: key = pkey_class.from_private_key_file(filename, password) try: transport.auth_publickey(username, key) return key except paramiko.SSHException: pass return None def __del__(self): self.close() @synchronize def __getstate__(self): state = super(SFTPFS,self).__getstate__() del state["_tlocal"] if self._owns_transport: state['_transport'] = self._transport.getpeername() return state def __setstate__(self,state): super(SFTPFS, self).__setstate__(state) #for (k,v) in state.iteritems(): # self.__dict__[k] = v #self._lock = threading.RLock() self._tlocal = thread_local() if self._owns_transport: self._transport = paramiko.Transport(self._transport) self._transport.connect(**self._credentials) @property @synchronize def client(self): if self.closed: return None client = getattr(self._tlocal, 'client', None) if client is None: if self._transport is None: return self._client client = paramiko.SFTPClient.from_transport(self._transport) self._tlocal.client = client return client # try: # return self._tlocal.client # except AttributeError: # #if self._transport is None: # # return self._client # client = paramiko.SFTPClient.from_transport(self._transport) # self._tlocal.client = client # return client @synchronize def close(self): """Close the connection to the remote server.""" if not self.closed: self._tlocal = None #if self.client: # self.client.close() if self._owns_transport and self._transport and self._transport.is_active: self._transport.close() self.closed = True def _normpath(self, path): if not isinstance(path, unicode): path = path.decode(self.encoding) npath = pathjoin(self.root_path, relpath(normpath(path))) if not isprefix(self.root_path, npath): raise PathError(path, msg="Path is outside root: %(path)s") return npath def getpathurl(self, path, allow_none=False): path = self._normpath(path) if self.hostname is None: if allow_none: return None raise NoPathURLError(path=path) username = self.credentials.get('username', '') or '' password = self.credentials.get('password', '') or '' credentials = ('%s:%s' % (username, password)).rstrip(':') if credentials: url = 'sftp://%s@%s%s' % (credentials, self.hostname.rstrip('/'), abspath(path)) else: url = 'sftp://%s%s' % (self.hostname.rstrip('/'), abspath(path)) return url @synchronize @convert_os_errors @iotools.filelike_to_stream def open(self, path, mode='r', buffering=-1, encoding=None, errors=None, newline=None, line_buffering=False, bufsize=-1, **kwargs): npath = self._normpath(path) if self.isdir(path): msg = "that's a directory: %(path)s" raise ResourceInvalidError(path, msg=msg) # paramiko implements its own buffering and write-back logic, # so we don't need to use a RemoteFileBuffer here. f = self.client.open(npath, mode, bufsize) # Unfortunately it has a broken truncate() method. # TODO: implement this as a wrapper old_truncate = f.truncate def new_truncate(size=None): if size is None: size = f.tell() return old_truncate(size) f.truncate = new_truncate return f @synchronize def desc(self, path): npath = self._normpath(path) if self.hostname: return u'sftp://%s%s' % (self.hostname, path) else: addr, port = self._transport.getpeername() return u'sftp://%s:%i%s' % (addr, port, self.client.normalize(npath)) @synchronize @convert_os_errors def exists(self, path): if path in ('', '/'): return True npath = self._normpath(path) try: self.client.stat(npath) except IOError, e: if getattr(e,"errno",None) == ENOENT: return False raise return True @synchronize @convert_os_errors def isdir(self,path): if normpath(path) in ('', '/'): return True npath = self._normpath(path) try: stat = self.client.stat(npath) except IOError, e: if getattr(e,"errno",None) == ENOENT: return False raise return statinfo.S_ISDIR(stat.st_mode) != 0 @synchronize @convert_os_errors def isfile(self,path): npath = self._normpath(path) try: stat = self.client.stat(npath) except IOError, e: if getattr(e,"errno",None) == ENOENT: return False raise return statinfo.S_ISREG(stat.st_mode) != 0 @synchronize @convert_os_errors def listdir(self,path="./",wildcard=None,full=False,absolute=False,dirs_only=False,files_only=False): npath = self._normpath(path) try: attrs_map = None if dirs_only or files_only: attrs = self.client.listdir_attr(npath) attrs_map = dict((a.filename, a) for a in attrs) paths = list(attrs_map.iterkeys()) else: paths = self.client.listdir(npath) except IOError, e: if getattr(e,"errno",None) == ENOENT: if self.isfile(path): raise ResourceInvalidError(path,msg="Can't list directory contents of a file: %(path)s") raise ResourceNotFoundError(path) elif self.isfile(path): raise ResourceInvalidError(path,msg="Can't list directory contents of a file: %(path)s") raise if attrs_map: if dirs_only: filter_paths = [] for apath, attr in attrs_map.iteritems(): if isdir(self, path, attr.__dict__): filter_paths.append(apath) paths = filter_paths elif files_only: filter_paths = [] for apath, attr in attrs_map.iteritems(): if isfile(self, apath, attr.__dict__): filter_paths.append(apath) paths = filter_paths for (i,p) in enumerate(paths): if not isinstance(p,unicode): paths[i] = p.decode(self.encoding) return self._listdir_helper(path, paths, wildcard, full, absolute, False, False) @synchronize @convert_os_errors def listdirinfo(self,path="./",wildcard=None,full=False,absolute=False,dirs_only=False,files_only=False): npath = self._normpath(path) try: attrs = self.client.listdir_attr(npath) attrs_map = dict((a.filename, a) for a in attrs) paths = attrs_map.keys() except IOError, e: if getattr(e,"errno",None) == ENOENT: if self.isfile(path): raise ResourceInvalidError(path,msg="Can't list directory contents of a file: %(path)s") raise ResourceNotFoundError(path) elif self.isfile(path): raise ResourceInvalidError(path,msg="Can't list directory contents of a file: %(path)s") raise if dirs_only: filter_paths = [] for path, attr in attrs_map.iteritems(): if isdir(self, path, attr.__dict__): filter_paths.append(path) paths = filter_paths elif files_only: filter_paths = [] for path, attr in attrs_map.iteritems(): if isfile(self, path, attr.__dict__): filter_paths.append(path) paths = filter_paths for (i, p) in enumerate(paths): if not isinstance(p, unicode): paths[i] = p.decode(self.encoding) def getinfo(p): resourcename = basename(p) info = attrs_map.get(resourcename) if info is None: return self.getinfo(pathjoin(path, p)) return self._extract_info(info.__dict__) return [(p, getinfo(p)) for p in self._listdir_helper(path, paths, wildcard, full, absolute, False, False)] @synchronize @convert_os_errors def makedir(self,path,recursive=False,allow_recreate=False): npath = self._normpath(path) try: self.client.mkdir(npath) except IOError, _e: # Error code is unreliable, try to figure out what went wrong try: stat = self.client.stat(npath) except IOError: if not self.isdir(dirname(path)): # Parent dir is missing if not recursive: raise ParentDirectoryMissingError(path) self.makedir(dirname(path),recursive=True) self.makedir(path,allow_recreate=allow_recreate) else: # Undetermined error, let the decorator handle it raise else: # Destination exists if statinfo.S_ISDIR(stat.st_mode): if not allow_recreate: raise DestinationExistsError(path,msg="Can't create a directory that already exists (try allow_recreate=True): %(path)s") else: raise ResourceInvalidError(path,msg="Can't create directory, there's already a file of that name: %(path)s") @synchronize @convert_os_errors def remove(self,path): npath = self._normpath(path) try: self.client.remove(npath) except IOError, e: if getattr(e,"errno",None) == ENOENT: raise ResourceNotFoundError(path) elif self.isdir(path): raise ResourceInvalidError(path,msg="Cannot use remove() on a directory: %(path)s") raise @synchronize @convert_os_errors def removedir(self,path,recursive=False,force=False): npath = self._normpath(path) if normpath(path) in ('', '/'): raise RemoveRootError(path) if force: for path2 in self.listdir(path,absolute=True): try: self.remove(path2) except ResourceInvalidError: self.removedir(path2,force=True) if not self.exists(path): raise ResourceNotFoundError(path) try: self.client.rmdir(npath) except IOError, e: if getattr(e,"errno",None) == ENOENT: if self.isfile(path): raise ResourceInvalidError(path,msg="Can't use removedir() on a file: %(path)s") raise ResourceNotFoundError(path) elif self.listdir(path): raise DirectoryNotEmptyError(path) raise if recursive: try: if dirname(path) not in ('', '/'): self.removedir(dirname(path),recursive=True) except DirectoryNotEmptyError: pass @synchronize @convert_os_errors def rename(self,src,dst): nsrc = self._normpath(src) ndst = self._normpath(dst) try: self.client.rename(nsrc,ndst) except IOError, e: if getattr(e,"errno",None) == ENOENT: raise ResourceNotFoundError(src) if not self.isdir(dirname(dst)): raise ParentDirectoryMissingError(dst) raise @synchronize @convert_os_errors def move(self,src,dst,overwrite=False,chunk_size=16384): nsrc = self._normpath(src) ndst = self._normpath(dst) if overwrite and self.isfile(dst): self.remove(dst) try: self.client.rename(nsrc,ndst) except IOError, e: if getattr(e,"errno",None) == ENOENT: raise ResourceNotFoundError(src) if self.exists(dst): raise DestinationExistsError(dst) if not self.isdir(dirname(dst)): raise ParentDirectoryMissingError(dst,msg="Destination directory does not exist: %(path)s") raise @synchronize @convert_os_errors def movedir(self,src,dst,overwrite=False,ignore_errors=False,chunk_size=16384): nsrc = self._normpath(src) ndst = self._normpath(dst) if overwrite and self.isdir(dst): self.removedir(dst) try: self.client.rename(nsrc,ndst) except IOError, e: if getattr(e,"errno",None) == ENOENT: raise ResourceNotFoundError(src) if self.exists(dst): raise DestinationExistsError(dst) if not self.isdir(dirname(dst)): raise ParentDirectoryMissingError(dst,msg="Destination directory does not exist: %(path)s") raise _info_vars = frozenset('st_size st_uid st_gid st_mode st_atime st_mtime'.split()) @classmethod def _extract_info(cls, stats): fromtimestamp = datetime.datetime.fromtimestamp info = dict((k, v) for k, v in stats.iteritems() if k in cls._info_vars and not k.startswith('_')) info['size'] = info['st_size'] ct = info.get('st_ctime') if ct is not None: info['created_time'] = fromtimestamp(ct) at = info.get('st_atime') if at is not None: info['accessed_time'] = fromtimestamp(at) mt = info.get('st_mtime') if mt is not None: info['modified_time'] = fromtimestamp(mt) return info @synchronize @convert_os_errors def getinfo(self, path): npath = self._normpath(path) stats = self.client.stat(npath) info = dict((k, getattr(stats, k)) for k in dir(stats) if not k.startswith('_')) info['size'] = info['st_size'] ct = info.get('st_ctime', None) if ct is not None: info['created_time'] = datetime.datetime.fromtimestamp(ct) at = info.get('st_atime', None) if at is not None: info['accessed_time'] = datetime.datetime.fromtimestamp(at) mt = info.get('st_mtime', None) if mt is not None: info['modified_time'] = datetime.datetime.fromtimestamp(mt) return info @synchronize @convert_os_errors def getsize(self, path): npath = self._normpath(path) stats = self.client.stat(npath) return stats.st_size fs-0.5.4/fs/expose/0000755000000000000000000000000012621617365014012 5ustar rootroot00000000000000fs-0.5.4/fs/expose/django_storage.py0000664000175000017500000000363412512525115017352 0ustar willwill00000000000000""" fs.expose.django ================ Use an FS object for Django File Storage This module exposes the class "FSStorage", a simple adapter for using FS objects as Django storage objects. Simply include the following lines in your settings.py:: DEFAULT_FILE_STORAGE = fs.expose.django_storage.FSStorage DEFAULT_FILE_STORAGE_FS = OSFS('foo/bar') # Or whatever FS """ from django.conf import settings from django.core.files.storage import Storage from django.core.files import File from fs.path import abspath, dirname from fs.errors import convert_fs_errors, ResourceNotFoundError class FSStorage(Storage): """Expose an FS object as a Django File Storage object.""" def __init__(self, fs=None, base_url=None): """ :param fs: an FS object :param base_url: The url to prepend to the path """ if fs is None: fs = settings.DEFAULT_FILE_STORAGE_FS if base_url is None: base_url = settings.MEDIA_URL base_url = base_url.rstrip('/') self.fs = fs self.base_url = base_url def exists(self, name): return self.fs.isfile(name) def path(self, name): path = self.fs.getsyspath(name) if path is None: raise NotImplementedError return path @convert_fs_errors def size(self, name): return self.fs.getsize(name) @convert_fs_errors def url(self, name): return self.base_url + abspath(name) @convert_fs_errors def _open(self, name, mode): return File(self.fs.open(name, mode)) @convert_fs_errors def _save(self, name, content): self.fs.makedir(dirname(name), allow_recreate=True, recursive=True) self.fs.setcontents(name, content) return name @convert_fs_errors def delete(self, name): try: self.fs.remove(name) except ResourceNotFoundError: pass fs-0.5.4/fs/expose/wsgi/0000755000000000000000000000000012621617365014763 5ustar rootroot00000000000000fs-0.5.4/fs/expose/wsgi/dirtemplate.py0000664000175000017500000000234112512525115017641 0ustar willwill00000000000000template = """ ${path}
% for i, entry in enumerate(dirlist): % endfor
File/Directory Size Created Date
${entry['name']} ${entry['size']} ${entry['created_time']}
"""fs-0.5.4/fs/expose/wsgi/wsgi.py0000664000175000017500000001117012512525115016300 0ustar willwill00000000000000 import urlparse import mimetypes from fs.errors import FSError from fs.path import basename, pathsplit from datetime import datetime try: from mako.template import Template except ImportError: print "Requires mako templates http://www.makotemplates.org/" raise class Request(object): """Very simple request object""" def __init__(self, environ, start_response): self.environ = environ self.start_response = start_response self.path = environ.get('PATH_INFO') class WSGIServer(object): """Light-weight WSGI server that exposes an FS""" def __init__(self, serve_fs, indexes=True, dir_template=None, chunk_size=16*1024*1024): if dir_template is None: from dirtemplate import template as dir_template self.serve_fs = serve_fs self.indexes = indexes self.chunk_size = chunk_size self.dir_template = Template(dir_template) def __call__(self, environ, start_response): request = Request(environ, start_response) if not self.serve_fs.exists(request.path): return self.serve_404(request) if self.serve_fs.isdir(request.path): if not self.indexes: return self.serve_404(request) return self.serve_dir(request) else: return self.serve_file(request) def serve_file(self, request): """Serve a file, guessing a mime-type""" path = request.path serving_file = None try: serving_file = self.serve_fs.open(path, 'rb') except Exception, e: if serving_file is not None: serving_file.close() return self.serve_500(request, str(e)) mime_type = mimetypes.guess_type(basename(path))[0] or b'text/plain' file_size = self.serve_fs.getsize(path) headers = [(b'Content-Type', bytes(mime_type)), (b'Content-Length', bytes(file_size))] def gen_file(): chunk_size = self.chunk_size read = serving_file.read try: while 1: data = read(chunk_size) if not data: break yield data finally: serving_file.close() request.start_response(b'200 OK', headers) return gen_file() def serve_dir(self, request): """Serve an index page""" fs = self.serve_fs isdir = fs.isdir path = request.path dirinfo = fs.listdirinfo(path, full=True, absolute=True) entries = [] for p, info in dirinfo: entry = {} entry['path'] = p entry['name'] = basename(p) entry['size'] = info.get('size', 'unknown') entry['created_time'] = info.get('created_time') if isdir(p): entry['type'] = 'dir' else: entry['type'] = 'file' entries.append(entry) # Put dirs first, and sort by reverse created time order no_time = datetime(1970, 1, 1, 1, 0) entries.sort(key=lambda k:(k['type'] == 'dir', k.get('created_time') or no_time), reverse=True) # Turn datetime to text and tweak names for entry in entries: t = entry.get('created_time') if t and hasattr(t, 'ctime'): entry['created_time'] = t.ctime() if entry['type'] == 'dir': entry['name'] += '/' # Add an up dir link for non-root if path not in ('', '/'): entries.insert(0, dict(name='../', path='../', type="dir", size='', created_time='..')) # Render the mako template html = self.dir_template.render(**dict(fs=self.serve_fs, path=path, dirlist=entries)).encode('utf-8') request.start_response(b'200 OK', [(b'Content-Type', b'text/html'), (b'Content-Length', b'%i' % len(html))]) return [html] def serve_404(self, request, msg='Not found'): """Serves a Not found page""" request.start_response(b'404 NOT FOUND', [(b'Content-Type', b'text/html')]) return [msg] def serve_500(self, request, msg='Unable to complete request'): """Serves an internal server error page""" request.start_response(b'500 INTERNAL SERVER ERROR', [(b'Content-Type', b'text/html')]) return [msg] def serve_fs(fs, indexes=True): """Serves an FS object via WSGI""" application = WSGIServer(fs, indexes) return application fs-0.5.4/fs/expose/wsgi/serve_home.py0000664000175000017500000000037412512525115017467 0ustar willwill00000000000000from wsgiref.simple_server import make_server from fs.osfs import OSFS from wsgi import serve_fs osfs = OSFS('~/') application = serve_fs(osfs) httpd = make_server('', 8000, application) print "Serving on http://127.0.0.1:8000" httpd.serve_forever() fs-0.5.4/fs/expose/wsgi/__init__.py0000664000175000017500000000003212512525115017061 0ustar willwill00000000000000from wsgi import serve_fs fs-0.5.4/fs/expose/fuse/0000755000000000000000000000000012621617365014754 5ustar rootroot00000000000000fs-0.5.4/fs/expose/fuse/fuse3.py0000664000175000017500000005315612512525115016357 0ustar willwill00000000000000# Copyright (c) 2008 Giorgos Verigakis # # Permission to use, copy, modify, and distribute this software for any # purpose with or without fee is hereby granted, provided that the above # copyright notice and this permission notice appear in all copies. # # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES # WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. from ctypes import * from ctypes.util import find_library from errno import * from functools import partial from platform import machine, system from stat import S_IFDIR from traceback import print_exc import logging class c_timespec(Structure): _fields_ = [('tv_sec', c_long), ('tv_nsec', c_long)] class c_utimbuf(Structure): _fields_ = [('actime', c_timespec), ('modtime', c_timespec)] class c_stat(Structure): pass # Platform dependent _system = system() if _system in ('Darwin', 'FreeBSD'): _libiconv = CDLL(find_library("iconv"), RTLD_GLOBAL) # libfuse dependency ENOTSUP = 45 c_dev_t = c_int32 c_fsblkcnt_t = c_ulong c_fsfilcnt_t = c_ulong c_gid_t = c_uint32 c_mode_t = c_uint16 c_off_t = c_int64 c_pid_t = c_int32 c_uid_t = c_uint32 setxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte), c_size_t, c_int, c_uint32) getxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte), c_size_t, c_uint32) c_stat._fields_ = [ ('st_dev', c_dev_t), ('st_ino', c_uint32), ('st_mode', c_mode_t), ('st_nlink', c_uint16), ('st_uid', c_uid_t), ('st_gid', c_gid_t), ('st_rdev', c_dev_t), ('st_atimespec', c_timespec), ('st_mtimespec', c_timespec), ('st_ctimespec', c_timespec), ('st_size', c_off_t), ('st_blocks', c_int64), ('st_blksize', c_int32)] elif _system == 'Linux': ENOTSUP = 95 c_dev_t = c_ulonglong c_fsblkcnt_t = c_ulonglong c_fsfilcnt_t = c_ulonglong c_gid_t = c_uint c_mode_t = c_uint c_off_t = c_longlong c_pid_t = c_int c_uid_t = c_uint setxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte), c_size_t, c_int) getxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte), c_size_t) _machine = machine() if _machine == 'x86_64': c_stat._fields_ = [ ('st_dev', c_dev_t), ('st_ino', c_ulong), ('st_nlink', c_ulong), ('st_mode', c_mode_t), ('st_uid', c_uid_t), ('st_gid', c_gid_t), ('__pad0', c_int), ('st_rdev', c_dev_t), ('st_size', c_off_t), ('st_blksize', c_long), ('st_blocks', c_long), ('st_atimespec', c_timespec), ('st_mtimespec', c_timespec), ('st_ctimespec', c_timespec)] elif _machine == 'ppc': c_stat._fields_ = [ ('st_dev', c_dev_t), ('st_ino', c_ulonglong), ('st_mode', c_mode_t), ('st_nlink', c_uint), ('st_uid', c_uid_t), ('st_gid', c_gid_t), ('st_rdev', c_dev_t), ('__pad2', c_ushort), ('st_size', c_off_t), ('st_blksize', c_long), ('st_blocks', c_longlong), ('st_atimespec', c_timespec), ('st_mtimespec', c_timespec), ('st_ctimespec', c_timespec)] else: # i686, use as fallback for everything else c_stat._fields_ = [ ('st_dev', c_dev_t), ('__pad1', c_ushort), ('__st_ino', c_ulong), ('st_mode', c_mode_t), ('st_nlink', c_uint), ('st_uid', c_uid_t), ('st_gid', c_gid_t), ('st_rdev', c_dev_t), ('__pad2', c_ushort), ('st_size', c_off_t), ('st_blksize', c_long), ('st_blocks', c_longlong), ('st_atimespec', c_timespec), ('st_mtimespec', c_timespec), ('st_ctimespec', c_timespec), ('st_ino', c_ulonglong)] else: raise NotImplementedError('%s is not supported.' % _system) class c_statvfs(Structure): _fields_ = [ ('f_bsize', c_ulong), ('f_frsize', c_ulong), ('f_blocks', c_fsblkcnt_t), ('f_bfree', c_fsblkcnt_t), ('f_bavail', c_fsblkcnt_t), ('f_files', c_fsfilcnt_t), ('f_ffree', c_fsfilcnt_t), ('f_favail', c_fsfilcnt_t)] if _system == 'FreeBSD': c_fsblkcnt_t = c_uint64 c_fsfilcnt_t = c_uint64 setxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte), c_size_t, c_int) getxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte), c_size_t) class c_statvfs(Structure): _fields_ = [ ('f_bavail', c_fsblkcnt_t), ('f_bfree', c_fsblkcnt_t), ('f_blocks', c_fsblkcnt_t), ('f_favail', c_fsfilcnt_t), ('f_ffree', c_fsfilcnt_t), ('f_files', c_fsfilcnt_t), ('f_bsize', c_ulong), ('f_flag', c_ulong), ('f_frsize', c_ulong)] class fuse_file_info(Structure): _fields_ = [ ('flags', c_int), ('fh_old', c_ulong), ('writepage', c_int), ('direct_io', c_uint, 1), ('keep_cache', c_uint, 1), ('flush', c_uint, 1), ('padding', c_uint, 29), ('fh', c_uint64), ('lock_owner', c_uint64)] class fuse_context(Structure): _fields_ = [ ('fuse', c_voidp), ('uid', c_uid_t), ('gid', c_gid_t), ('pid', c_pid_t), ('private_data', c_voidp)] class fuse_operations(Structure): _fields_ = [ ('getattr', CFUNCTYPE(c_int, c_char_p, POINTER(c_stat))), ('readlink', CFUNCTYPE(c_int, c_char_p, POINTER(c_byte), c_size_t)), ('getdir', c_voidp), # Deprecated, use readdir ('mknod', CFUNCTYPE(c_int, c_char_p, c_mode_t, c_dev_t)), ('mkdir', CFUNCTYPE(c_int, c_char_p, c_mode_t)), ('unlink', CFUNCTYPE(c_int, c_char_p)), ('rmdir', CFUNCTYPE(c_int, c_char_p)), ('symlink', CFUNCTYPE(c_int, c_char_p, c_char_p)), ('rename', CFUNCTYPE(c_int, c_char_p, c_char_p)), ('link', CFUNCTYPE(c_int, c_char_p, c_char_p)), ('chmod', CFUNCTYPE(c_int, c_char_p, c_mode_t)), ('chown', CFUNCTYPE(c_int, c_char_p, c_uid_t, c_gid_t)), ('truncate', CFUNCTYPE(c_int, c_char_p, c_off_t)), ('utime', c_voidp), # Deprecated, use utimens ('open', CFUNCTYPE(c_int, c_char_p, POINTER(fuse_file_info))), ('read', CFUNCTYPE(c_int, c_char_p, POINTER(c_byte), c_size_t, c_off_t, POINTER(fuse_file_info))), ('write', CFUNCTYPE(c_int, c_char_p, POINTER(c_byte), c_size_t, c_off_t, POINTER(fuse_file_info))), ('statfs', CFUNCTYPE(c_int, c_char_p, POINTER(c_statvfs))), ('flush', CFUNCTYPE(c_int, c_char_p, POINTER(fuse_file_info))), ('release', CFUNCTYPE(c_int, c_char_p, POINTER(fuse_file_info))), ('fsync', CFUNCTYPE(c_int, c_char_p, c_int, POINTER(fuse_file_info))), ('setxattr', setxattr_t), ('getxattr', getxattr_t), ('listxattr', CFUNCTYPE(c_int, c_char_p, POINTER(c_byte), c_size_t)), ('removexattr', CFUNCTYPE(c_int, c_char_p, c_char_p)), ('opendir', CFUNCTYPE(c_int, c_char_p, POINTER(fuse_file_info))), ('readdir', CFUNCTYPE(c_int, c_char_p, c_voidp, CFUNCTYPE(c_int, c_voidp, c_char_p, POINTER(c_stat), c_off_t), c_off_t, POINTER(fuse_file_info))), ('releasedir', CFUNCTYPE(c_int, c_char_p, POINTER(fuse_file_info))), ('fsyncdir', CFUNCTYPE(c_int, c_char_p, c_int, POINTER(fuse_file_info))), ('init', CFUNCTYPE(c_voidp, c_voidp)), ('destroy', CFUNCTYPE(c_voidp, c_voidp)), ('access', CFUNCTYPE(c_int, c_char_p, c_int)), ('create', CFUNCTYPE(c_int, c_char_p, c_mode_t, POINTER(fuse_file_info))), ('ftruncate', CFUNCTYPE(c_int, c_char_p, c_off_t, POINTER(fuse_file_info))), ('fgetattr', CFUNCTYPE(c_int, c_char_p, POINTER(c_stat), POINTER(fuse_file_info))), ('lock', CFUNCTYPE(c_int, c_char_p, POINTER(fuse_file_info), c_int, c_voidp)), ('utimens', CFUNCTYPE(c_int, c_char_p, POINTER(c_utimbuf))), ('bmap', CFUNCTYPE(c_int, c_char_p, c_size_t, POINTER(c_ulonglong)))] def time_of_timespec(ts): return ts.tv_sec + ts.tv_nsec / 10 ** 9 def set_st_attrs(st, attrs): for key, val in attrs.items(): if key in ('st_atime', 'st_mtime', 'st_ctime'): timespec = getattr(st, key + 'spec') timespec.tv_sec = int(val) timespec.tv_nsec = int((val - timespec.tv_sec) * 10 ** 9) elif hasattr(st, key): setattr(st, key, val) _libfuse_path = find_library('fuse') if not _libfuse_path: raise EnvironmentError('Unable to find libfuse') _libfuse = CDLL(_libfuse_path) _libfuse.fuse_get_context.restype = POINTER(fuse_context) def fuse_get_context(): """Returns a (uid, gid, pid) tuple""" ctxp = _libfuse.fuse_get_context() ctx = ctxp.contents return ctx.uid, ctx.gid, ctx.pid class FUSE(object): """This class is the lower level interface and should not be subclassed under normal use. Its methods are called by fuse. Assumes API version 2.6 or later.""" def __init__(self, operations, mountpoint, raw_fi=False, **kwargs): """Setting raw_fi to True will cause FUSE to pass the fuse_file_info class as is to Operations, instead of just the fh field. This gives you access to direct_io, keep_cache, etc.""" self.operations = operations self.raw_fi = raw_fi args = ['fuse'] if kwargs.pop('foreground', False): args.append('-f') if kwargs.pop('debug', False): args.append('-d') if kwargs.pop('nothreads', False): args.append('-s') kwargs.setdefault('fsname', operations.__class__.__name__) args.append('-o') args.append(','.join(key if val == True else '%s=%s' % (key, val) for key, val in kwargs.items())) args.append(mountpoint) argv = (c_char_p * len(args))(*args) fuse_ops = fuse_operations() for name, prototype in fuse_operations._fields_: if prototype != c_voidp and getattr(operations, name, None): op = partial(self._wrapper_, getattr(self, name)) setattr(fuse_ops, name, prototype(op)) _libfuse.fuse_main_real(len(args), argv, pointer(fuse_ops), sizeof(fuse_ops), None) del self.operations # Invoke the destructor def _wrapper_(self, func, *args, **kwargs): """Decorator for the methods that follow""" try: return func(*args, **kwargs) or 0 except OSError as e: return -(e.errno or EFAULT) except: print_exc() return -EFAULT def getattr(self, path, buf): return self.fgetattr(path, buf, None) def readlink(self, path, buf, bufsize): ret = self.operations('readlink', path).encode('utf-8') data = create_string_buffer(ret[:bufsize - 1]) memmove(buf, data, len(data)) return 0 def mknod(self, path, mode, dev): return self.operations('mknod', path, mode, dev) def mkdir(self, path, mode): return self.operations('mkdir', path, mode) def unlink(self, path): return self.operations('unlink', path) def rmdir(self, path): return self.operations('rmdir', path) def symlink(self, source, target): return self.operations('symlink', target, source) def rename(self, old, new): return self.operations('rename', old, new) def link(self, source, target): return self.operations('link', target, source) def chmod(self, path, mode): return self.operations('chmod', path, mode) def chown(self, path, uid, gid): return self.operations('chown', path, uid, gid) def truncate(self, path, length): return self.operations('truncate', path, length) def open(self, path, fip): fi = fip.contents if self.raw_fi: return self.operations('open', path, fi) else: fi.fh = self.operations('open', path, fi.flags) return 0 def read(self, path, buf, size, offset, fip): fh = fip.contents if self.raw_fi else fip.contents.fh ret = self.operations('read', path, size, offset, fh) if not ret: return 0 data = create_string_buffer(ret[:size], size) memmove(buf, data, size) return size def write(self, path, buf, size, offset, fip): data = string_at(buf, size) fh = fip.contents if self.raw_fi else fip.contents.fh return self.operations('write', path, data, offset, fh) def statfs(self, path, buf): stv = buf.contents attrs = self.operations('statfs', path) for key, val in attrs.items(): if hasattr(stv, key): setattr(stv, key, val) return 0 def flush(self, path, fip): fh = fip.contents if self.raw_fi else fip.contents.fh return self.operations('flush', path, fh) def release(self, path, fip): fh = fip.contents if self.raw_fi else fip.contents.fh return self.operations('release', path, fh) def fsync(self, path, datasync, fip): fh = fip.contents if self.raw_fi else fip.contents.fh return self.operations('fsync', path, datasync, fh) def setxattr(self, path, name, value, size, options, *args): data = string_at(value, size) return self.operations('setxattr', path, name, data, options, *args) def getxattr(self, path, name, value, size, *args): ret = self.operations('getxattr', path, name, *args) retsize = len(ret) buf = create_string_buffer(ret, retsize) # Does not add trailing 0 if bool(value): if retsize > size: return -ERANGE memmove(value, buf, retsize) return retsize def listxattr(self, path, namebuf, size): ret = self.operations('listxattr', path) buf = create_string_buffer('\x00'.join(ret)) if ret else '' bufsize = len(buf) if bool(namebuf): if bufsize > size: return -ERANGE memmove(namebuf, buf, bufsize) return bufsize def removexattr(self, path, name): return self.operations('removexattr', path, name) def opendir(self, path, fip): # Ignore raw_fi fip.contents.fh = self.operations('opendir', path) return 0 def readdir(self, path, buf, filler, offset, fip): # Ignore raw_fi for item in self.operations('readdir', path, fip.contents.fh): if isinstance(item, str): name, st, offset = item, None, 0 name = name.encode('utf-8') else: name, attrs, offset = item if attrs: st = c_stat() set_st_attrs(st, attrs) else: st = None if filler(buf, name, st, offset) != 0: break return 0 def releasedir(self, path, fip): # Ignore raw_fi return self.operations('releasedir', path, fip.contents.fh) def fsyncdir(self, path, datasync, fip): # Ignore raw_fi return self.operations('fsyncdir', path, datasync, fip.contents.fh) def init(self, conn): return self.operations('init', '/') def destroy(self, private_data): return self.operations('destroy', '/') def access(self, path, amode): return self.operations('access', path, amode) def create(self, path, mode, fip): fi = fip.contents if self.raw_fi: return self.operations('create', path, mode, fi) else: fi.fh = self.operations('create', path, mode) return 0 def ftruncate(self, path, length, fip): fh = fip.contents if self.raw_fi else fip.contents.fh return self.operations('truncate', path, length, fh) def fgetattr(self, path, buf, fip): memset(buf, 0, sizeof(c_stat)) st = buf.contents fh = fip and (fip.contents if self.raw_fi else fip.contents.fh) attrs = self.operations('getattr', path, fh) set_st_attrs(st, attrs) return 0 def lock(self, path, fip, cmd, lock): fh = fip.contents if self.raw_fi else fip.contents.fh return self.operations('lock', path, fh, cmd, lock) def utimens(self, path, buf): if buf: atime = time_of_timespec(buf.contents.actime) mtime = time_of_timespec(buf.contents.modtime) times = (atime, mtime) else: times = None return self.operations('utimens', path, times) def bmap(self, path, blocksize, idx): return self.operations('bmap', path, blocksize, idx) class Operations(object): """This class should be subclassed and passed as an argument to FUSE on initialization. All operations should raise an OSError exception on error. When in doubt of what an operation should do, check the FUSE header file or the corresponding system call man page.""" def __call__(self, op, *args): if not hasattr(self, op): raise OSError(EFAULT, '') return getattr(self, op)(*args) def access(self, path, amode): return 0 bmap = None def chmod(self, path, mode): raise OSError(EROFS, '') def chown(self, path, uid, gid): raise OSError(EROFS, '') def create(self, path, mode, fi=None): """When raw_fi is False (default case), fi is None and create should return a numerical file handle. When raw_fi is True the file handle should be set directly by create and return 0.""" raise OSError(EROFS, '') def destroy(self, path): """Called on filesystem destruction. Path is always /""" pass def flush(self, path, fh): return 0 def fsync(self, path, datasync, fh): return 0 def fsyncdir(self, path, datasync, fh): return 0 def getattr(self, path, fh=None): """Returns a dictionary with keys identical to the stat C structure of stat(2). st_atime, st_mtime and st_ctime should be floats. NOTE: There is an incombatibility between Linux and Mac OS X concerning st_nlink of directories. Mac OS X counts all files inside the directory, while Linux counts only the subdirectories.""" if path != '/': raise OSError(ENOENT, '') return dict(st_mode=(S_IFDIR | 0o755), st_nlink=2) def getxattr(self, path, name, position=0): raise OSError(ENOTSUP, '') def init(self, path): """Called on filesystem initialization. Path is always / Use it instead of __init__ if you start threads on initialization.""" pass def link(self, target, source): raise OSError(EROFS, '') def listxattr(self, path): return [] lock = None def mkdir(self, path, mode): raise OSError(EROFS, '') def mknod(self, path, mode, dev): raise OSError(EROFS, '') def open(self, path, flags): """When raw_fi is False (default case), open should return a numerical file handle. When raw_fi is True the signature of open becomes: open(self, path, fi) and the file handle should be set directly.""" return 0 def opendir(self, path): """Returns a numerical file handle.""" return 0 def read(self, path, size, offset, fh): """Returns a string containing the data requested.""" raise OSError(ENOENT, '') def readdir(self, path, fh): """Can return either a list of names, or a list of (name, attrs, offset) tuples. attrs is a dict as in getattr.""" return ['.', '..'] def readlink(self, path): raise OSError(ENOENT, '') def release(self, path, fh): return 0 def releasedir(self, path, fh): return 0 def removexattr(self, path, name): raise OSError(ENOTSUP, '') def rename(self, old, new): raise OSError(EROFS, '') def rmdir(self, path): raise OSError(EROFS, '') def setxattr(self, path, name, value, options, position=0): raise OSError(ENOTSUP, '') def statfs(self, path): """Returns a dictionary with keys identical to the statvfs C structure of statvfs(3). On Mac OS X f_bsize and f_frsize must be a power of 2 (minimum 512).""" return {} def symlink(self, target, source): raise OSError(EROFS, '') def truncate(self, path, length, fh=None): raise OSError(EROFS, '') def unlink(self, path): raise OSError(EROFS, '') def utimens(self, path, times=None): """Times is a (atime, mtime) tuple. If None use current time.""" return 0 def write(self, path, data, offset, fh): raise OSError(EROFS, '') class LoggingMixIn: def __call__(self, op, path, *args): logging.debug('-> %s %s %s', op, path, repr(args)) ret = '[Unknown Error]' try: ret = getattr(self, op)(path, *args) return ret except OSError as e: ret = str(e) raise finally: logging.debug('<- %s %s', op, repr(ret)) fs-0.5.4/fs/expose/fuse/fuse_ctypes.py0000664000175000017500000005645012512525115017663 0ustar willwill00000000000000# Copyright (c) 2008 Giorgos Verigakis # # Permission to use, copy, modify, and distribute this software for any # purpose with or without fee is hereby granted, provided that the above # copyright notice and this permission notice appear in all copies. # # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES # WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. from __future__ import division from ctypes import * from ctypes.util import find_library from errno import * from functools import partial from platform import machine, system from stat import S_IFDIR from traceback import print_exc _system = system() _machine = machine() # Locate the fuse shared library. # On OSX this can be provided by a number of different packages # with slightly incompatible interfaces. if _system == 'Darwin': _libfuse_path = find_library('fuse4x') or find_library('fuse') else: _libfuse_path = find_library('fuse') if not _libfuse_path: raise EnvironmentError('Unable to find libfuse') if _system == 'Darwin': _libiconv = CDLL(find_library('iconv'), RTLD_GLOBAL) # libfuse dependency _libfuse = CDLL(_libfuse_path) # Check whether OSX is using the legacy "macfuse" system. # This has a different struct layout than the newer fuse4x system. if _system == 'Darwin' and hasattr(_libfuse, 'macfuse_version'): _system = 'Darwin-MacFuse' class c_timespec(Structure): _fields_ = [('tv_sec', c_long), ('tv_nsec', c_long)] class c_utimbuf(Structure): _fields_ = [('actime', c_timespec), ('modtime', c_timespec)] class c_stat(Structure): pass # Platform dependent if _system in ('Darwin', 'Darwin-MacFuse', 'FreeBSD'): ENOTSUP = 45 c_dev_t = c_int32 c_fsblkcnt_t = c_ulong c_fsfilcnt_t = c_ulong c_gid_t = c_uint32 c_mode_t = c_uint16 c_off_t = c_int64 c_pid_t = c_int32 c_uid_t = c_uint32 setxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte), c_size_t, c_int, c_uint32) getxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte), c_size_t, c_uint32) # OSX with fuse4x uses 64-bit inodes and so has a different # struct layout. Other darwinish platforms use 32-bit inodes. if _system == 'Darwin': c_stat._fields_ = [ ('st_dev', c_dev_t), ('st_mode', c_mode_t), ('st_nlink', c_uint16), ('st_ino', c_uint64), ('st_uid', c_uid_t), ('st_gid', c_gid_t), ('st_rdev', c_dev_t), ('st_atimespec', c_timespec), ('st_mtimespec', c_timespec), ('st_ctimespec', c_timespec), ('st_birthtimespec', c_timespec), ('st_size', c_off_t), ('st_blocks', c_int64), ('st_blksize', c_int32), ('st_flags', c_int32), ('st_gen', c_int32), ('st_lspare', c_int32), ('st_qspare', c_int64)] else: c_stat._fields_ = [ ('st_dev', c_dev_t), ('st_ino', c_uint32), ('st_mode', c_mode_t), ('st_nlink', c_uint16), ('st_uid', c_uid_t), ('st_gid', c_gid_t), ('st_rdev', c_dev_t), ('st_atimespec', c_timespec), ('st_mtimespec', c_timespec), ('st_ctimespec', c_timespec), ('st_size', c_off_t), ('st_blocks', c_int64), ('st_blksize', c_int32)] elif _system == 'Linux': ENOTSUP = 95 c_dev_t = c_ulonglong c_fsblkcnt_t = c_ulonglong c_fsfilcnt_t = c_ulonglong c_gid_t = c_uint c_mode_t = c_uint c_off_t = c_longlong c_pid_t = c_int c_uid_t = c_uint setxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte), c_size_t, c_int) getxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte), c_size_t) _machine = machine() if _machine == 'x86_64': c_stat._fields_ = [ ('st_dev', c_dev_t), ('st_ino', c_ulong), ('st_nlink', c_ulong), ('st_mode', c_mode_t), ('st_uid', c_uid_t), ('st_gid', c_gid_t), ('__pad0', c_int), ('st_rdev', c_dev_t), ('st_size', c_off_t), ('st_blksize', c_long), ('st_blocks', c_long), ('st_atimespec', c_timespec), ('st_mtimespec', c_timespec), ('st_ctimespec', c_timespec)] elif _machine == 'ppc': c_stat._fields_ = [ ('st_dev', c_dev_t), ('st_ino', c_ulonglong), ('st_mode', c_mode_t), ('st_nlink', c_uint), ('st_uid', c_uid_t), ('st_gid', c_gid_t), ('st_rdev', c_dev_t), ('__pad2', c_ushort), ('st_size', c_off_t), ('st_blksize', c_long), ('st_blocks', c_longlong), ('st_atimespec', c_timespec), ('st_mtimespec', c_timespec), ('st_ctimespec', c_timespec)] else: # i686, use as fallback for everything else c_stat._fields_ = [ ('st_dev', c_dev_t), ('__pad1', c_ushort), ('__st_ino', c_ulong), ('st_mode', c_mode_t), ('st_nlink', c_uint), ('st_uid', c_uid_t), ('st_gid', c_gid_t), ('st_rdev', c_dev_t), ('__pad2', c_ushort), ('st_size', c_off_t), ('st_blksize', c_long), ('st_blocks', c_longlong), ('st_atimespec', c_timespec), ('st_mtimespec', c_timespec), ('st_ctimespec', c_timespec), ('st_ino', c_ulonglong)] else: raise NotImplementedError('%s is not supported.' % _system) class c_statvfs(Structure): _fields_ = [ ('f_bsize', c_ulong), ('f_frsize', c_ulong), ('f_blocks', c_fsblkcnt_t), ('f_bfree', c_fsblkcnt_t), ('f_bavail', c_fsblkcnt_t), ('f_files', c_fsfilcnt_t), ('f_ffree', c_fsfilcnt_t), ('f_favail', c_fsfilcnt_t)] if _system == 'FreeBSD': c_fsblkcnt_t = c_uint64 c_fsfilcnt_t = c_uint64 setxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte), c_size_t, c_int) getxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte), c_size_t) class c_statvfs(Structure): _fields_ = [ ('f_bavail', c_fsblkcnt_t), ('f_bfree', c_fsblkcnt_t), ('f_blocks', c_fsblkcnt_t), ('f_favail', c_fsfilcnt_t), ('f_ffree', c_fsfilcnt_t), ('f_files', c_fsfilcnt_t), ('f_bsize', c_ulong), ('f_flag', c_ulong), ('f_frsize', c_ulong)] class fuse_file_info(Structure): _fields_ = [ ('flags', c_int), ('fh_old', c_ulong), ('writepage', c_int), ('direct_io', c_uint, 1), ('keep_cache', c_uint, 1), ('flush', c_uint, 1), ('padding', c_uint, 29), ('fh', c_uint64), ('lock_owner', c_uint64)] class fuse_context(Structure): _fields_ = [ ('fuse', c_voidp), ('uid', c_uid_t), ('gid', c_gid_t), ('pid', c_pid_t), ('private_data', c_voidp)] class fuse_operations(Structure): _fields_ = [ ('getattr', CFUNCTYPE(c_int, c_char_p, POINTER(c_stat))), ('readlink', CFUNCTYPE(c_int, c_char_p, POINTER(c_byte), c_size_t)), ('getdir', c_voidp), # Deprecated, use readdir ('mknod', CFUNCTYPE(c_int, c_char_p, c_mode_t, c_dev_t)), ('mkdir', CFUNCTYPE(c_int, c_char_p, c_mode_t)), ('unlink', CFUNCTYPE(c_int, c_char_p)), ('rmdir', CFUNCTYPE(c_int, c_char_p)), ('symlink', CFUNCTYPE(c_int, c_char_p, c_char_p)), ('rename', CFUNCTYPE(c_int, c_char_p, c_char_p)), ('link', CFUNCTYPE(c_int, c_char_p, c_char_p)), ('chmod', CFUNCTYPE(c_int, c_char_p, c_mode_t)), ('chown', CFUNCTYPE(c_int, c_char_p, c_uid_t, c_gid_t)), ('truncate', CFUNCTYPE(c_int, c_char_p, c_off_t)), ('utime', c_voidp), # Deprecated, use utimens ('open', CFUNCTYPE(c_int, c_char_p, POINTER(fuse_file_info))), ('read', CFUNCTYPE(c_int, c_char_p, POINTER(c_byte), c_size_t, c_off_t, POINTER(fuse_file_info))), ('write', CFUNCTYPE(c_int, c_char_p, POINTER(c_byte), c_size_t, c_off_t, POINTER(fuse_file_info))), ('statfs', CFUNCTYPE(c_int, c_char_p, POINTER(c_statvfs))), ('flush', CFUNCTYPE(c_int, c_char_p, POINTER(fuse_file_info))), ('release', CFUNCTYPE(c_int, c_char_p, POINTER(fuse_file_info))), ('fsync', CFUNCTYPE(c_int, c_char_p, c_int, POINTER(fuse_file_info))), ('setxattr', setxattr_t), ('getxattr', getxattr_t), ('listxattr', CFUNCTYPE(c_int, c_char_p, POINTER(c_byte), c_size_t)), ('removexattr', CFUNCTYPE(c_int, c_char_p, c_char_p)), ('opendir', CFUNCTYPE(c_int, c_char_p, POINTER(fuse_file_info))), ('readdir', CFUNCTYPE(c_int, c_char_p, c_voidp, CFUNCTYPE(c_int, c_voidp, c_char_p, POINTER(c_stat), c_off_t), c_off_t, POINTER(fuse_file_info))), ('releasedir', CFUNCTYPE(c_int, c_char_p, POINTER(fuse_file_info))), ('fsyncdir', CFUNCTYPE(c_int, c_char_p, c_int, POINTER(fuse_file_info))), ('init', CFUNCTYPE(c_voidp, c_voidp)), ('destroy', CFUNCTYPE(c_voidp, c_voidp)), ('access', CFUNCTYPE(c_int, c_char_p, c_int)), ('create', CFUNCTYPE(c_int, c_char_p, c_mode_t, POINTER(fuse_file_info))), ('ftruncate', CFUNCTYPE(c_int, c_char_p, c_off_t, POINTER(fuse_file_info))), ('fgetattr', CFUNCTYPE(c_int, c_char_p, POINTER(c_stat), POINTER(fuse_file_info))), ('lock', CFUNCTYPE(c_int, c_char_p, POINTER(fuse_file_info), c_int, c_voidp)), ('utimens', CFUNCTYPE(c_int, c_char_p, POINTER(c_utimbuf))), ('bmap', CFUNCTYPE(c_int, c_char_p, c_size_t, POINTER(c_ulonglong)))] def time_of_timespec(ts): return ts.tv_sec + ts.tv_nsec / 10 ** 9 def set_st_attrs(st, attrs): for key, val in attrs.items(): if key in ('st_atime', 'st_mtime', 'st_ctime'): timespec = getattr(st, key + 'spec') timespec.tv_sec = int(val) timespec.tv_nsec = int((val - timespec.tv_sec) * 10 ** 9) elif hasattr(st, key): setattr(st, key, val) _libfuse.fuse_get_context.restype = POINTER(fuse_context) def fuse_get_context(): """Returns a (uid, gid, pid) tuple""" ctxp = _libfuse.fuse_get_context() ctx = ctxp.contents return ctx.uid, ctx.gid, ctx.pid class FUSE(object): """This class is the lower level interface and should not be subclassed under normal use. Its methods are called by fuse. Assumes API version 2.6 or later.""" def __init__(self, operations, mountpoint, raw_fi=False, **kwargs): """Setting raw_fi to True will cause FUSE to pass the fuse_file_info class as is to Operations, instead of just the fh field. This gives you access to direct_io, keep_cache, etc.""" self.operations = operations self.raw_fi = raw_fi args = ['fuse'] if kwargs.pop('foreground', False): args.append('-f') if kwargs.pop('debug', False): args.append('-d') if kwargs.pop('nothreads', False): args.append('-s') kwargs.setdefault('fsname', operations.__class__.__name__) args.append('-o') args.append(','.join(key if val == True else '%s=%s' % (key, val) for key, val in kwargs.items())) args.append(mountpoint) argv = (c_char_p * len(args))(*args) fuse_ops = fuse_operations() for name, prototype in fuse_operations._fields_: if prototype != c_voidp and getattr(operations, name, None): op = partial(self._wrapper_, getattr(self, name)) setattr(fuse_ops, name, prototype(op)) _libfuse.fuse_main_real(len(args), argv, pointer(fuse_ops), sizeof(fuse_ops), None) del self.operations # Invoke the destructor def _wrapper_(self, func, *args, **kwargs): """Decorator for the methods that follow""" try: return func(*args, **kwargs) or 0 except OSError, e: return -(e.errno or EFAULT) except: print_exc() return -EFAULT def getattr(self, path, buf): return self.fgetattr(path, buf, None) def readlink(self, path, buf, bufsize): ret = self.operations('readlink', path) data = create_string_buffer(ret[:bufsize - 1]) memmove(buf, data, len(data)) return 0 def mknod(self, path, mode, dev): return self.operations('mknod', path, mode, dev) def mkdir(self, path, mode): return self.operations('mkdir', path, mode) def unlink(self, path): return self.operations('unlink', path) def rmdir(self, path): return self.operations('rmdir', path) def symlink(self, source, target): return self.operations('symlink', target, source) def rename(self, old, new): return self.operations('rename', old, new) def link(self, source, target): return self.operations('link', target, source) def chmod(self, path, mode): return self.operations('chmod', path, mode) def chown(self, path, uid, gid): return self.operations('chown', path, uid, gid) def truncate(self, path, length): return self.operations('truncate', path, length) def open(self, path, fip): fi = fip.contents if self.raw_fi: return self.operations('open', path, fi) else: fi.fh = self.operations('open', path, fi.flags) return 0 def read(self, path, buf, size, offset, fip): fh = fip.contents if self.raw_fi else fip.contents.fh ret = self.operations('read', path, size, offset, fh) if ret: strbuf = create_string_buffer(ret) memmove(buf, strbuf, len(strbuf)) return len(ret) def write(self, path, buf, size, offset, fip): data = string_at(buf, size) fh = fip.contents if self.raw_fi else fip.contents.fh return self.operations('write', path, data, offset, fh) def statfs(self, path, buf): stv = buf.contents attrs = self.operations('statfs', path) for key, val in attrs.items(): if hasattr(stv, key): setattr(stv, key, val) return 0 def flush(self, path, fip): fh = fip.contents if self.raw_fi else fip.contents.fh return self.operations('flush', path, fh) def release(self, path, fip): fh = fip.contents if self.raw_fi else fip.contents.fh return self.operations('release', path, fh) def fsync(self, path, datasync, fip): fh = fip.contents if self.raw_fi else fip.contents.fh return self.operations('fsync', path, datasync, fh) def setxattr(self, path, name, value, size, options, *args): data = string_at(value, size) return self.operations('setxattr', path, name, data, options, *args) def getxattr(self, path, name, value, size, *args): ret = self.operations('getxattr', path, name, *args) retsize = len(ret) buf = create_string_buffer(ret, retsize) # Does not add trailing 0 if bool(value): if retsize > size: return -ERANGE memmove(value, buf, retsize) return retsize def listxattr(self, path, namebuf, size): ret = self.operations('listxattr', path) if ret: buf = create_string_buffer('\x00'.join(ret)) else: buf = '' bufsize = len(buf) if bool(namebuf): if bufsize > size: return -ERANGE memmove(namebuf, buf, bufsize) return bufsize def removexattr(self, path, name): return self.operations('removexattr', path, name) def opendir(self, path, fip): # Ignore raw_fi fip.contents.fh = self.operations('opendir', path) return 0 def readdir(self, path, buf, filler, offset, fip): # Ignore raw_fi for item in self.operations('readdir', path, fip.contents.fh): if isinstance(item, str): name, st, offset = item, None, 0 else: name, attrs, offset = item if attrs: st = c_stat() set_st_attrs(st, attrs) else: st = None if filler(buf, name, st, offset) != 0: break return 0 def releasedir(self, path, fip): # Ignore raw_fi return self.operations('releasedir', path, fip.contents.fh) def fsyncdir(self, path, datasync, fip): # Ignore raw_fi return self.operations('fsyncdir', path, datasync, fip.contents.fh) def init(self, conn): return self.operations('init', '/') def destroy(self, private_data): return self.operations('destroy', '/') def access(self, path, amode): return self.operations('access', path, amode) def create(self, path, mode, fip): fi = fip.contents if self.raw_fi: return self.operations('create', path, mode, fi) else: fi.fh = self.operations('create', path, mode) return 0 def ftruncate(self, path, length, fip): fh = fip.contents if self.raw_fi else fip.contents.fh return self.operations('truncate', path, length, fh) def fgetattr(self, path, buf, fip): memset(buf, 0, sizeof(c_stat)) st = buf.contents fh = fip and (fip.contents if self.raw_fi else fip.contents.fh) attrs = self.operations('getattr', path, fh) set_st_attrs(st, attrs) return 0 def lock(self, path, fip, cmd, lock): fh = fip.contents if self.raw_fi else fip.contents.fh return self.operations('lock', path, fh, cmd, lock) def utimens(self, path, buf): if buf: atime = time_of_timespec(buf.contents.actime) mtime = time_of_timespec(buf.contents.modtime) times = (atime, mtime) else: times = None return self.operations('utimens', path, times) def bmap(self, path, blocksize, idx): return self.operations('bmap', path, blocksize, idx) class Operations(object): """This class should be subclassed and passed as an argument to FUSE on initialization. All operations should raise an OSError exception on error. When in doubt of what an operation should do, check the FUSE header file or the corresponding system call man page.""" def __call__(self, op, *args): if not hasattr(self, op): raise OSError(EFAULT, '') return getattr(self, op)(*args) def access(self, path, amode): return 0 bmap = None def chmod(self, path, mode): raise OSError(EROFS, '') def chown(self, path, uid, gid): raise OSError(EROFS, '') def create(self, path, mode, fi=None): """When raw_fi is False (default case), fi is None and create should return a numerical file handle. When raw_fi is True the file handle should be set directly by create and return 0.""" raise OSError(EROFS, '') def destroy(self, path): """Called on filesystem destruction. Path is always /""" pass def flush(self, path, fh): return 0 def fsync(self, path, datasync, fh): return 0 def fsyncdir(self, path, datasync, fh): return 0 def getattr(self, path, fh=None): """Returns a dictionary with keys identical to the stat C structure of stat(2). st_atime, st_mtime and st_ctime should be floats. NOTE: There is an incombatibility between Linux and Mac OS X concerning st_nlink of directories. Mac OS X counts all files inside the directory, while Linux counts only the subdirectories.""" if path != '/': raise OSError(ENOENT, '') return dict(st_mode=(S_IFDIR | 0755), st_nlink=2) def getxattr(self, path, name, position=0): raise OSError(ENOTSUP, '') def init(self, path): """Called on filesystem initialization. Path is always / Use it instead of __init__ if you start threads on initialization.""" pass def link(self, target, source): raise OSError(EROFS, '') def listxattr(self, path): return [] lock = None def mkdir(self, path, mode): raise OSError(EROFS, '') def mknod(self, path, mode, dev): raise OSError(EROFS, '') def open(self, path, flags): """When raw_fi is False (default case), open should return a numerical file handle. When raw_fi is True the signature of open becomes: open(self, path, fi) and the file handle should be set directly.""" return 0 def opendir(self, path): """Returns a numerical file handle.""" return 0 def read(self, path, size, offset, fh): """Returns a string containing the data requested.""" raise OSError(ENOENT, '') def readdir(self, path, fh): """Can return either a list of names, or a list of (name, attrs, offset) tuples. attrs is a dict as in getattr.""" return ['.', '..'] def readlink(self, path): raise OSError(ENOENT, '') def release(self, path, fh): return 0 def releasedir(self, path, fh): return 0 def removexattr(self, path, name): raise OSError(ENOTSUP, '') def rename(self, old, new): raise OSError(EROFS, '') def rmdir(self, path): raise OSError(EROFS, '') def setxattr(self, path, name, value, options, position=0): raise OSError(ENOTSUP, '') def statfs(self, path): """Returns a dictionary with keys identical to the statvfs C structure of statvfs(3). On Mac OS X f_bsize and f_frsize must be a power of 2 (minimum 512).""" return {} def symlink(self, target, source): raise OSError(EROFS, '') def truncate(self, path, length, fh=None): raise OSError(EROFS, '') def unlink(self, path): raise OSError(EROFS, '') def utimens(self, path, times=None): """Times is a (atime, mtime) tuple. If None use current time.""" return 0 def write(self, path, data, offset, fh): raise OSError(EROFS, '') class LoggingMixIn: def __call__(self, op, path, *args): print '->', op, path, repr(args) ret = '[Unknown Error]' try: ret = getattr(self, op)(path, *args) return ret except OSError, e: ret = str(e) raise finally: print '<-', op, repr(ret) fs-0.5.4/fs/expose/fuse/fuse.py0000664000175000017500000005636012512525115016274 0ustar willwill00000000000000# Copyright (c) 2008 Giorgos Verigakis # # Permission to use, copy, modify, and distribute this software for any # purpose with or without fee is hereby granted, provided that the above # copyright notice and this permission notice appear in all copies. # # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES # WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. from __future__ import division from ctypes import * from ctypes.util import find_library from errno import * from functools import partial from os import strerror from platform import machine, system from stat import S_IFDIR from traceback import print_exc _system = system() _machine = machine() if _system == 'Darwin': _libfuse_path = find_library('fuse4x') or find_library('fuse') else: _libfuse_path = find_library('fuse') if not _libfuse_path: raise EnvironmentError('Unable to find libfuse') if _system == 'Darwin': _libiconv = CDLL(find_library('iconv'), RTLD_GLOBAL) # libfuse dependency _libfuse = CDLL(_libfuse_path) if _system == 'Darwin' and hasattr(_libfuse, 'macfuse_version'): _system = 'Darwin-MacFuse' class c_timespec(Structure): _fields_ = [('tv_sec', c_long), ('tv_nsec', c_long)] class c_utimbuf(Structure): _fields_ = [('actime', c_timespec), ('modtime', c_timespec)] class c_stat(Structure): pass # Platform dependent if _system in ('Darwin', 'Darwin-MacFuse', 'FreeBSD'): ENOTSUP = 45 c_dev_t = c_int32 c_fsblkcnt_t = c_ulong c_fsfilcnt_t = c_ulong c_gid_t = c_uint32 c_mode_t = c_uint16 c_off_t = c_int64 c_pid_t = c_int32 c_uid_t = c_uint32 setxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte), c_size_t, c_int, c_uint32) getxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte), c_size_t, c_uint32) if _system == 'Darwin': c_stat._fields_ = [ ('st_dev', c_dev_t), ('st_mode', c_mode_t), ('st_nlink', c_uint16), ('st_ino', c_uint64), ('st_uid', c_uid_t), ('st_gid', c_gid_t), ('st_rdev', c_dev_t), ('st_atimespec', c_timespec), ('st_mtimespec', c_timespec), ('st_ctimespec', c_timespec), ('st_birthtimespec', c_timespec), ('st_size', c_off_t), ('st_blocks', c_int64), ('st_blksize', c_int32), ('st_flags', c_int32), ('st_gen', c_int32), ('st_lspare', c_int32), ('st_qspare', c_int64)] else: c_stat._fields_ = [ ('st_dev', c_dev_t), ('st_ino', c_uint32), ('st_mode', c_mode_t), ('st_nlink', c_uint16), ('st_uid', c_uid_t), ('st_gid', c_gid_t), ('st_rdev', c_dev_t), ('st_atimespec', c_timespec), ('st_mtimespec', c_timespec), ('st_ctimespec', c_timespec), ('st_size', c_off_t), ('st_blocks', c_int64), ('st_blksize', c_int32)] elif _system == 'Linux': ENOTSUP = 95 c_dev_t = c_ulonglong c_fsblkcnt_t = c_ulonglong c_fsfilcnt_t = c_ulonglong c_gid_t = c_uint c_mode_t = c_uint c_off_t = c_longlong c_pid_t = c_int c_uid_t = c_uint setxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte), c_size_t, c_int) getxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte), c_size_t) if _machine == 'x86_64': c_stat._fields_ = [ ('st_dev', c_dev_t), ('st_ino', c_ulong), ('st_nlink', c_ulong), ('st_mode', c_mode_t), ('st_uid', c_uid_t), ('st_gid', c_gid_t), ('__pad0', c_int), ('st_rdev', c_dev_t), ('st_size', c_off_t), ('st_blksize', c_long), ('st_blocks', c_long), ('st_atimespec', c_timespec), ('st_mtimespec', c_timespec), ('st_ctimespec', c_timespec)] elif _machine == 'ppc': c_stat._fields_ = [ ('st_dev', c_dev_t), ('st_ino', c_ulonglong), ('st_mode', c_mode_t), ('st_nlink', c_uint), ('st_uid', c_uid_t), ('st_gid', c_gid_t), ('st_rdev', c_dev_t), ('__pad2', c_ushort), ('st_size', c_off_t), ('st_blksize', c_long), ('st_blocks', c_longlong), ('st_atimespec', c_timespec), ('st_mtimespec', c_timespec), ('st_ctimespec', c_timespec)] else: # i686, use as fallback for everything else c_stat._fields_ = [ ('st_dev', c_dev_t), ('__pad1', c_ushort), ('__st_ino', c_ulong), ('st_mode', c_mode_t), ('st_nlink', c_uint), ('st_uid', c_uid_t), ('st_gid', c_gid_t), ('st_rdev', c_dev_t), ('__pad2', c_ushort), ('st_size', c_off_t), ('st_blksize', c_long), ('st_blocks', c_longlong), ('st_atimespec', c_timespec), ('st_mtimespec', c_timespec), ('st_ctimespec', c_timespec), ('st_ino', c_ulonglong)] else: raise NotImplementedError('%s is not supported.' % _system) class c_statvfs(Structure): _fields_ = [ ('f_bsize', c_ulong), ('f_frsize', c_ulong), ('f_blocks', c_fsblkcnt_t), ('f_bfree', c_fsblkcnt_t), ('f_bavail', c_fsblkcnt_t), ('f_files', c_fsfilcnt_t), ('f_ffree', c_fsfilcnt_t), ('f_favail', c_fsfilcnt_t)] if _system == 'FreeBSD': c_fsblkcnt_t = c_uint64 c_fsfilcnt_t = c_uint64 setxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte), c_size_t, c_int) getxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte), c_size_t) class c_statvfs(Structure): _fields_ = [ ('f_bavail', c_fsblkcnt_t), ('f_bfree', c_fsblkcnt_t), ('f_blocks', c_fsblkcnt_t), ('f_favail', c_fsfilcnt_t), ('f_ffree', c_fsfilcnt_t), ('f_files', c_fsfilcnt_t), ('f_bsize', c_ulong), ('f_flag', c_ulong), ('f_frsize', c_ulong)] class fuse_file_info(Structure): _fields_ = [ ('flags', c_int), ('fh_old', c_ulong), ('writepage', c_int), ('direct_io', c_uint, 1), ('keep_cache', c_uint, 1), ('flush', c_uint, 1), ('padding', c_uint, 29), ('fh', c_uint64), ('lock_owner', c_uint64)] class fuse_context(Structure): _fields_ = [ ('fuse', c_voidp), ('uid', c_uid_t), ('gid', c_gid_t), ('pid', c_pid_t), ('private_data', c_voidp)] _libfuse.fuse_get_context.restype = POINTER(fuse_context) class fuse_operations(Structure): _fields_ = [ ('getattr', CFUNCTYPE(c_int, c_char_p, POINTER(c_stat))), ('readlink', CFUNCTYPE(c_int, c_char_p, POINTER(c_byte), c_size_t)), ('getdir', c_voidp), # Deprecated, use readdir ('mknod', CFUNCTYPE(c_int, c_char_p, c_mode_t, c_dev_t)), ('mkdir', CFUNCTYPE(c_int, c_char_p, c_mode_t)), ('unlink', CFUNCTYPE(c_int, c_char_p)), ('rmdir', CFUNCTYPE(c_int, c_char_p)), ('symlink', CFUNCTYPE(c_int, c_char_p, c_char_p)), ('rename', CFUNCTYPE(c_int, c_char_p, c_char_p)), ('link', CFUNCTYPE(c_int, c_char_p, c_char_p)), ('chmod', CFUNCTYPE(c_int, c_char_p, c_mode_t)), ('chown', CFUNCTYPE(c_int, c_char_p, c_uid_t, c_gid_t)), ('truncate', CFUNCTYPE(c_int, c_char_p, c_off_t)), ('utime', c_voidp), # Deprecated, use utimens ('open', CFUNCTYPE(c_int, c_char_p, POINTER(fuse_file_info))), ('read', CFUNCTYPE(c_int, c_char_p, POINTER(c_byte), c_size_t, c_off_t, POINTER(fuse_file_info))), ('write', CFUNCTYPE(c_int, c_char_p, POINTER(c_byte), c_size_t, c_off_t, POINTER(fuse_file_info))), ('statfs', CFUNCTYPE(c_int, c_char_p, POINTER(c_statvfs))), ('flush', CFUNCTYPE(c_int, c_char_p, POINTER(fuse_file_info))), ('release', CFUNCTYPE(c_int, c_char_p, POINTER(fuse_file_info))), ('fsync', CFUNCTYPE(c_int, c_char_p, c_int, POINTER(fuse_file_info))), ('setxattr', setxattr_t), ('getxattr', getxattr_t), ('listxattr', CFUNCTYPE(c_int, c_char_p, POINTER(c_byte), c_size_t)), ('removexattr', CFUNCTYPE(c_int, c_char_p, c_char_p)), ('opendir', CFUNCTYPE(c_int, c_char_p, POINTER(fuse_file_info))), ('readdir', CFUNCTYPE(c_int, c_char_p, c_voidp, CFUNCTYPE(c_int, c_voidp, c_char_p, POINTER(c_stat), c_off_t), c_off_t, POINTER(fuse_file_info))), ('releasedir', CFUNCTYPE(c_int, c_char_p, POINTER(fuse_file_info))), ('fsyncdir', CFUNCTYPE(c_int, c_char_p, c_int, POINTER(fuse_file_info))), ('init', CFUNCTYPE(c_voidp, c_voidp)), ('destroy', CFUNCTYPE(c_voidp, c_voidp)), ('access', CFUNCTYPE(c_int, c_char_p, c_int)), ('create', CFUNCTYPE(c_int, c_char_p, c_mode_t, POINTER(fuse_file_info))), ('ftruncate', CFUNCTYPE(c_int, c_char_p, c_off_t, POINTER(fuse_file_info))), ('fgetattr', CFUNCTYPE(c_int, c_char_p, POINTER(c_stat), POINTER(fuse_file_info))), ('lock', CFUNCTYPE(c_int, c_char_p, POINTER(fuse_file_info), c_int, c_voidp)), ('utimens', CFUNCTYPE(c_int, c_char_p, POINTER(c_utimbuf))), ('bmap', CFUNCTYPE(c_int, c_char_p, c_size_t, POINTER(c_ulonglong)))] def time_of_timespec(ts): return ts.tv_sec + ts.tv_nsec / 10 ** 9 def set_st_attrs(st, attrs): for key, val in attrs.items(): if key in ('st_atime', 'st_mtime', 'st_ctime'): timespec = getattr(st, key + 'spec') timespec.tv_sec = int(val) timespec.tv_nsec = int((val - timespec.tv_sec) * 10 ** 9) elif hasattr(st, key): setattr(st, key, val) def fuse_get_context(): """Returns a (uid, gid, pid) tuple""" ctxp = _libfuse.fuse_get_context() ctx = ctxp.contents return ctx.uid, ctx.gid, ctx.pid class FuseOSError(OSError): def __init__(self, errno): super(FuseOSError, self).__init__(errno, strerror(errno)) class FUSE(object): """This class is the lower level interface and should not be subclassed under normal use. Its methods are called by fuse. Assumes API version 2.6 or later.""" def __init__(self, operations, mountpoint, raw_fi=False, **kwargs): """Setting raw_fi to True will cause FUSE to pass the fuse_file_info class as is to Operations, instead of just the fh field. This gives you access to direct_io, keep_cache, etc.""" self.operations = operations self.raw_fi = raw_fi args = ['fuse'] if kwargs.pop('foreground', False): args.append('-f') if kwargs.pop('debug', False): args.append('-d') if kwargs.pop('nothreads', False): args.append('-s') kwargs.setdefault('fsname', operations.__class__.__name__) args.append('-o') args.append(','.join(key if val == True else '%s=%s' % (key, val) for key, val in kwargs.items())) args.append(mountpoint) argv = (c_char_p * len(args))(*args) fuse_ops = fuse_operations() for name, prototype in fuse_operations._fields_: if prototype != c_voidp and getattr(operations, name, None): op = partial(self._wrapper_, getattr(self, name)) setattr(fuse_ops, name, prototype(op)) err = _libfuse.fuse_main_real(len(args), argv, pointer(fuse_ops), sizeof(fuse_ops), None) del self.operations # Invoke the destructor if err: raise RuntimeError(err) def _wrapper_(self, func, *args, **kwargs): """Decorator for the methods that follow""" try: return func(*args, **kwargs) or 0 except OSError, e: return -(e.errno or EFAULT) except: print_exc() return -EFAULT def getattr(self, path, buf): return self.fgetattr(path, buf, None) def readlink(self, path, buf, bufsize): ret = self.operations('readlink', path) data = create_string_buffer(ret[:bufsize - 1]) memmove(buf, data, len(data)) return 0 def mknod(self, path, mode, dev): return self.operations('mknod', path, mode, dev) def mkdir(self, path, mode): return self.operations('mkdir', path, mode) def unlink(self, path): return self.operations('unlink', path) def rmdir(self, path): return self.operations('rmdir', path) def symlink(self, source, target): return self.operations('symlink', target, source) def rename(self, old, new): return self.operations('rename', old, new) def link(self, source, target): return self.operations('link', target, source) def chmod(self, path, mode): return self.operations('chmod', path, mode) def chown(self, path, uid, gid): # Check if any of the arguments is a -1 that has overflowed if c_uid_t(uid + 1).value == 0: uid = -1 if c_gid_t(gid + 1).value == 0: gid = -1 return self.operations('chown', path, uid, gid) def truncate(self, path, length): return self.operations('truncate', path, length) def open(self, path, fip): fi = fip.contents if self.raw_fi: return self.operations('open', path, fi) else: fi.fh = self.operations('open', path, fi.flags) return 0 def read(self, path, buf, size, offset, fip): fh = fip.contents if self.raw_fi else fip.contents.fh ret = self.operations('read', path, size, offset, fh) if not ret: return 0 data = create_string_buffer(ret[:size], size) memmove(buf, data, size) return size def write(self, path, buf, size, offset, fip): data = string_at(buf, size) fh = fip.contents if self.raw_fi else fip.contents.fh return self.operations('write', path, data, offset, fh) def statfs(self, path, buf): stv = buf.contents attrs = self.operations('statfs', path) for key, val in attrs.items(): if hasattr(stv, key): setattr(stv, key, val) return 0 def flush(self, path, fip): fh = fip.contents if self.raw_fi else fip.contents.fh return self.operations('flush', path, fh) def release(self, path, fip): fh = fip.contents if self.raw_fi else fip.contents.fh return self.operations('release', path, fh) def fsync(self, path, datasync, fip): fh = fip.contents if self.raw_fi else fip.contents.fh return self.operations('fsync', path, datasync, fh) def setxattr(self, path, name, value, size, options, *args): data = string_at(value, size) return self.operations('setxattr', path, name, data, options, *args) def getxattr(self, path, name, value, size, *args): ret = self.operations('getxattr', path, name, *args) retsize = len(ret) buf = create_string_buffer(ret, retsize) # Does not add trailing 0 if bool(value): if retsize > size: return -ERANGE memmove(value, buf, retsize) return retsize def listxattr(self, path, namebuf, size): ret = self.operations('listxattr', path) buf = create_string_buffer('\x00'.join(ret)) if ret else '' bufsize = len(buf) if bool(namebuf): if bufsize > size: return -ERANGE memmove(namebuf, buf, bufsize) return bufsize def removexattr(self, path, name): return self.operations('removexattr', path, name) def opendir(self, path, fip): # Ignore raw_fi fip.contents.fh = self.operations('opendir', path) return 0 def readdir(self, path, buf, filler, offset, fip): # Ignore raw_fi for item in self.operations('readdir', path, fip.contents.fh): if isinstance(item, str): name, st, offset = item, None, 0 else: name, attrs, offset = item if attrs: st = c_stat() set_st_attrs(st, attrs) else: st = None if filler(buf, name, st, offset) != 0: break return 0 def releasedir(self, path, fip): # Ignore raw_fi return self.operations('releasedir', path, fip.contents.fh) def fsyncdir(self, path, datasync, fip): # Ignore raw_fi return self.operations('fsyncdir', path, datasync, fip.contents.fh) def init(self, conn): return self.operations('init', '/') def destroy(self, private_data): return self.operations('destroy', '/') def access(self, path, amode): return self.operations('access', path, amode) def create(self, path, mode, fip): fi = fip.contents if self.raw_fi: return self.operations('create', path, mode, fi) else: fi.fh = self.operations('create', path, mode) return 0 def ftruncate(self, path, length, fip): fh = fip.contents if self.raw_fi else fip.contents.fh return self.operations('truncate', path, length, fh) def fgetattr(self, path, buf, fip): memset(buf, 0, sizeof(c_stat)) st = buf.contents fh = fip and (fip.contents if self.raw_fi else fip.contents.fh) attrs = self.operations('getattr', path, fh) set_st_attrs(st, attrs) return 0 def lock(self, path, fip, cmd, lock): fh = fip.contents if self.raw_fi else fip.contents.fh return self.operations('lock', path, fh, cmd, lock) def utimens(self, path, buf): if buf: atime = time_of_timespec(buf.contents.actime) mtime = time_of_timespec(buf.contents.modtime) times = (atime, mtime) else: times = None return self.operations('utimens', path, times) def bmap(self, path, blocksize, idx): return self.operations('bmap', path, blocksize, idx) class Operations(object): """This class should be subclassed and passed as an argument to FUSE on initialization. All operations should raise a FuseOSError exception on error. When in doubt of what an operation should do, check the FUSE header file or the corresponding system call man page.""" def __call__(self, op, *args): if not hasattr(self, op): raise FuseOSError(EFAULT) return getattr(self, op)(*args) def access(self, path, amode): return 0 bmap = None def chmod(self, path, mode): raise FuseOSError(EROFS) def chown(self, path, uid, gid): raise FuseOSError(EROFS) def create(self, path, mode, fi=None): """When raw_fi is False (default case), fi is None and create should return a numerical file handle. When raw_fi is True the file handle should be set directly by create and return 0.""" raise FuseOSError(EROFS) def destroy(self, path): """Called on filesystem destruction. Path is always /""" pass def flush(self, path, fh): return 0 def fsync(self, path, datasync, fh): return 0 def fsyncdir(self, path, datasync, fh): return 0 def getattr(self, path, fh=None): """Returns a dictionary with keys identical to the stat C structure of stat(2). st_atime, st_mtime and st_ctime should be floats. NOTE: There is an incombatibility between Linux and Mac OS X concerning st_nlink of directories. Mac OS X counts all files inside the directory, while Linux counts only the subdirectories.""" if path != '/': raise FuseOSError(ENOENT) return dict(st_mode=(S_IFDIR | 0755), st_nlink=2) def getxattr(self, path, name, position=0): raise FuseOSError(ENOTSUP) def init(self, path): """Called on filesystem initialization. Path is always / Use it instead of __init__ if you start threads on initialization.""" pass def link(self, target, source): raise FuseOSError(EROFS) def listxattr(self, path): return [] lock = None def mkdir(self, path, mode): raise FuseOSError(EROFS) def mknod(self, path, mode, dev): raise FuseOSError(EROFS) def open(self, path, flags): """When raw_fi is False (default case), open should return a numerical file handle. When raw_fi is True the signature of open becomes: open(self, path, fi) and the file handle should be set directly.""" return 0 def opendir(self, path): """Returns a numerical file handle.""" return 0 def read(self, path, size, offset, fh): """Returns a string containing the data requested.""" raise FuseOSError(EIO) def readdir(self, path, fh): """Can return either a list of names, or a list of (name, attrs, offset) tuples. attrs is a dict as in getattr.""" return ['.', '..'] def readlink(self, path): raise FuseOSError(ENOENT) def release(self, path, fh): return 0 def releasedir(self, path, fh): return 0 def removexattr(self, path, name): raise FuseOSError(ENOTSUP) def rename(self, old, new): raise FuseOSError(EROFS) def rmdir(self, path): raise FuseOSError(EROFS) def setxattr(self, path, name, value, options, position=0): raise FuseOSError(ENOTSUP) def statfs(self, path): """Returns a dictionary with keys identical to the statvfs C structure of statvfs(3). On Mac OS X f_bsize and f_frsize must be a power of 2 (minimum 512).""" return {} def symlink(self, target, source): raise FuseOSError(EROFS) def truncate(self, path, length, fh=None): raise FuseOSError(EROFS) def unlink(self, path): raise FuseOSError(EROFS) def utimens(self, path, times=None): """Times is a (atime, mtime) tuple. If None use current time.""" return 0 def write(self, path, data, offset, fh): raise FuseOSError(EROFS) class LoggingMixIn: def __call__(self, op, path, *args): print '->', op, path, repr(args) ret = '[Unhandled Exception]' try: ret = getattr(self, op)(path, *args) return ret except OSError, e: ret = str(e) raise finally: print '<-', op, repr(ret) fs-0.5.4/fs/expose/fuse/__init__.py0000664000175000017500000005276512512525115017076 0ustar willwill00000000000000""" fs.expose.fuse ============== Expose an FS object to the native filesystem via FUSE This module provides the necessary interfaces to mount an FS object into the local filesystem via FUSE:: http://fuse.sourceforge.net/ For simple usage, the function 'mount' takes an FS object and a local path, and exposes the given FS at that path:: >>> from fs.memoryfs import MemoryFS >>> from fs.expose import fuse >>> fs = MemoryFS() >>> mp = fuse.mount(fs,"/mnt/my-memory-fs") >>> mp.path '/mnt/my-memory-fs' >>> mp.unmount() The above spawns a new background process to manage the FUSE event loop, which can be controlled through the returned subprocess.Popen object. To avoid spawning a new process, set the 'foreground' option:: >>> # This will block until the filesystem is unmounted >>> fuse.mount(fs,"/mnt/my-memory-fs",foreground=True) Any additional options for the FUSE process can be passed as keyword arguments to the 'mount' function. If you require finer control over the creation of the FUSE process, you can instantiate the MountProcess class directly. It accepts all options available to subprocess.Popen:: >>> from subprocess import PIPE >>> mp = fuse.MountProcess(fs,"/mnt/my-memory-fs",stderr=PIPE) >>> fuse_errors = mp.communicate()[1] The binding to FUSE is created via ctypes, using a custom version of the fuse.py code from Giorgos Verigakis: http://code.google.com/p/fusepy/ """ import sys if sys.platform == "win32": raise ImportError("FUSE is not available on win32") import datetime import os import signal import errno import time import stat as statinfo import subprocess import cPickle import logging logger = logging.getLogger("fs.expose.fuse") from fs.base import flags_to_mode, threading from fs.errors import * from fs.path import * from fs.local_functools import wraps from six import PY3 from six import b try: if PY3: from fs.expose.fuse import fuse_ctypes as fuse else: from fs.expose.fuse import fuse3 as fuse except NotImplementedError: raise ImportError("FUSE found but not usable") try: fuse._libfuse.fuse_get_context except AttributeError: raise ImportError("could not locate FUSE library") FUSE = fuse.FUSE Operations = fuse.Operations fuse_get_context = fuse.fuse_get_context STARTUP_TIME = time.time() NATIVE_ENCODING = sys.getfilesystemencoding() def handle_fs_errors(func): """Method decorator to report FS errors in the appropriate way. This decorator catches all FS errors and translates them into an equivalent OSError. It also makes the function return zero instead of None as an indication of successful execution. """ name = func.__name__ func = convert_fs_errors(func) @wraps(func) def wrapper(*args,**kwds): #logger.debug("CALL %r %s",name,repr(args)) try: res = func(*args,**kwds) except Exception: #logger.exception("ERR %r %s",name,repr(args)) raise else: #logger.exception("OK %r %s %r",name,repr(args),res) if res is None: return 0 return res return wrapper class FSOperations(Operations): """FUSE Operations interface delegating all activities to an FS object.""" def __init__(self, fs, on_init=None, on_destroy=None): self.fs = fs self._on_init = on_init self._on_destroy = on_destroy self._files_by_handle = {} self._files_lock = threading.Lock() self._next_handle = 1 # FUSE expects a succesful write() to be reflected in the file's # reported size, but the FS might buffer writes and prevent this. # We explicitly keep track of the size FUSE expects a file to be. # This dict is indexed by path, then file handle. self._files_size_written = {} def _get_file(self, fh): try: return self._files_by_handle[fh.fh] except KeyError: raise FSError("invalid file handle") def _reg_file(self, f, path): self._files_lock.acquire() try: fh = self._next_handle self._next_handle += 1 lock = threading.Lock() self._files_by_handle[fh] = (f,path,lock) if path not in self._files_size_written: self._files_size_written[path] = {} self._files_size_written[path][fh] = 0 return fh finally: self._files_lock.release() def _del_file(self, fh): self._files_lock.acquire() try: (f,path,lock) = self._files_by_handle.pop(fh.fh) del self._files_size_written[path][fh.fh] if not self._files_size_written[path]: del self._files_size_written[path] finally: self._files_lock.release() def init(self, conn): if self._on_init: self._on_init() def destroy(self, data): if self._on_destroy: self._on_destroy() @handle_fs_errors def chmod(self, path, mode): raise UnsupportedError("chmod") @handle_fs_errors def chown(self, path, uid, gid): raise UnsupportedError("chown") @handle_fs_errors def create(self, path, mode, fi): path = path.decode(NATIVE_ENCODING) # FUSE doesn't seem to pass correct mode information here - at least, # I haven't figured out how to distinguish between "w" and "w+". # Go with the most permissive option. mode = flags_to_mode(fi.flags) fh = self._reg_file(self.fs.open(path, mode), path) fi.fh = fh fi.keep_cache = 0 @handle_fs_errors def flush(self, path, fh): (file, _, lock) = self._get_file(fh) lock.acquire() try: file.flush() finally: lock.release() @handle_fs_errors def getattr(self, path, fh=None): attrs = self._get_stat_dict(path.decode(NATIVE_ENCODING)) return attrs @handle_fs_errors def getxattr(self, path, name, position=0): path = path.decode(NATIVE_ENCODING) name = name.decode(NATIVE_ENCODING) try: value = self.fs.getxattr(path, name) except AttributeError: raise UnsupportedError("getxattr") else: if value is None: raise OSError(errno.ENODATA, "no attribute '%s'" % (name,)) return value @handle_fs_errors def link(self, target, souce): raise UnsupportedError("link") @handle_fs_errors def listxattr(self, path): path = path.decode(NATIVE_ENCODING) try: return self.fs.listxattrs(path) except AttributeError: raise UnsupportedError("listxattrs") @handle_fs_errors def mkdir(self, path, mode): path = path.decode(NATIVE_ENCODING) try: self.fs.makedir(path, recursive=True) except TypeError: self.fs.makedir(path) @handle_fs_errors def mknod(self, path, mode, dev): raise UnsupportedError("mknod") @handle_fs_errors def open(self, path, fi): path = path.decode(NATIVE_ENCODING) mode = flags_to_mode(fi.flags) fi.fh = self._reg_file(self.fs.open(path, mode), path) fi.keep_cache = 0 return 0 @handle_fs_errors def read(self, path, size, offset, fh): (file, _, lock) = self._get_file(fh) lock.acquire() try: file.seek(offset) data = file.read(size) return data finally: lock.release() @handle_fs_errors def readdir(self, path, fh=None): path = path.decode(NATIVE_ENCODING) entries = ['.', '..'] for (nm, info) in self.fs.listdirinfo(path): self._fill_stat_dict(pathjoin(path, nm), info) entries.append((nm.encode(NATIVE_ENCODING), info, 0)) return entries @handle_fs_errors def readlink(self, path): raise UnsupportedError("readlink") @handle_fs_errors def release(self, path, fh): (file, _, lock) = self._get_file(fh) lock.acquire() try: file.close() self._del_file(fh) finally: lock.release() @handle_fs_errors def removexattr(self, path, name): path = path.decode(NATIVE_ENCODING) name = name.decode(NATIVE_ENCODING) try: return self.fs.delxattr(path, name) except AttributeError: raise UnsupportedError("removexattr") @handle_fs_errors def rename(self, old, new): old = old.decode(NATIVE_ENCODING) new = new.decode(NATIVE_ENCODING) try: self.fs.rename(old, new) except FSError: if self.fs.isdir(old): self.fs.movedir(old, new) else: self.fs.move(old, new) @handle_fs_errors def rmdir(self, path): path = path.decode(NATIVE_ENCODING) self.fs.removedir(path) @handle_fs_errors def setxattr(self, path, name, value, options, position=0): path = path.decode(NATIVE_ENCODING) name = name.decode(NATIVE_ENCODING) try: return self.fs.setxattr(path, name, value) except AttributeError: raise UnsupportedError("setxattr") @handle_fs_errors def symlink(self, target, source): raise UnsupportedError("symlink") @handle_fs_errors def truncate(self, path, length, fh=None): path = path.decode(NATIVE_ENCODING) if fh is None and length == 0: self.fs.open(path, "wb").close() else: if fh is None: f = self.fs.open(path, "rb+") if not hasattr(f, "truncate"): raise UnsupportedError("truncate") f.truncate(length) else: (file, _, lock) = self._get_file(fh) lock.acquire() try: if not hasattr(file, "truncate"): raise UnsupportedError("truncate") file.truncate(length) finally: lock.release() self._files_lock.acquire() try: try: size_written = self._files_size_written[path] except KeyError: pass else: for k in size_written: size_written[k] = length finally: self._files_lock.release() @handle_fs_errors def unlink(self, path): path = path.decode(NATIVE_ENCODING) self.fs.remove(path) @handle_fs_errors def utimens(self, path, times=None): path = path.decode(NATIVE_ENCODING) accessed_time, modified_time = times if accessed_time is not None: accessed_time = datetime.datetime.fromtimestamp(accessed_time) if modified_time is not None: modified_time = datetime.datetime.fromtimestamp(modified_time) self.fs.settimes(path, accessed_time, modified_time) @handle_fs_errors def write(self, path, data, offset, fh): (file, path, lock) = self._get_file(fh) lock.acquire() try: file.seek(offset) file.write(data) if self._files_size_written[path][fh.fh] < offset + len(data): self._files_size_written[path][fh.fh] = offset + len(data) return len(data) finally: lock.release() def _get_stat_dict(self, path): """Build a 'stat' dictionary for the given file.""" info = self.fs.getinfo(path) self._fill_stat_dict(path, info) return info def _fill_stat_dict(self, path, info): """Fill default values in the stat dict.""" uid, gid, pid = fuse_get_context() private_keys = [k for k in info if k.startswith("_")] for k in private_keys: del info[k] # Basic stuff that is constant for all paths info.setdefault("st_ino", 0) info.setdefault("st_dev", 0) info.setdefault("st_uid", uid) info.setdefault("st_gid", gid) info.setdefault("st_rdev", 0) info.setdefault("st_blksize", 1024) info.setdefault("st_blocks", 1) # The interesting stuff if 'st_mode' not in info: if self.fs.isdir(path): info['st_mode'] = 0755 else: info['st_mode'] = 0666 mode = info['st_mode'] if not statinfo.S_ISDIR(mode) and not statinfo.S_ISREG(mode): if self.fs.isdir(path): info["st_mode"] = mode | statinfo.S_IFDIR info.setdefault("st_nlink", 2) else: info["st_mode"] = mode | statinfo.S_IFREG info.setdefault("st_nlink", 1) for (key1, key2) in [("st_atime", "accessed_time"), ("st_mtime", "modified_time"), ("st_ctime", "created_time")]: if key1 not in info: if key2 in info: info[key1] = time.mktime(info[key2].timetuple()) else: info[key1] = STARTUP_TIME # Ensure the reported size reflects any writes performed, even if # they haven't been flushed to the filesystem yet. if "size" in info: info["st_size"] = info["size"] elif "st_size" not in info: info["st_size"] = 0 try: written_sizes = self._files_size_written[path] except KeyError: pass else: info["st_size"] = max(written_sizes.values() + [info["st_size"]]) return info def mount(fs, path, foreground=False, ready_callback=None, unmount_callback=None, **kwds): """Mount the given FS at the given path, using FUSE. By default, this function spawns a new background process to manage the FUSE event loop. The return value in this case is an instance of the 'MountProcess' class, a subprocess.Popen subclass. If the keyword argument 'foreground' is given, we instead run the FUSE main loop in the current process. In this case the function will block until the filesystem is unmounted, then return None. If the keyword argument 'ready_callback' is provided, it will be called when the filesystem has been mounted and is ready for use. Any additional keyword arguments will be passed through as options to the underlying FUSE class. Some interesting options include: * nothreads Switch off threading in the FUSE event loop * fsname Name to display in the mount info table """ path = os.path.expanduser(path) if foreground: op = FSOperations(fs, on_init=ready_callback, on_destroy=unmount_callback) return FUSE(op, path, raw_fi=True, foreground=foreground, **kwds) else: mp = MountProcess(fs, path, kwds) if ready_callback: ready_callback() if unmount_callback: orig_unmount = mp.unmount def new_unmount(): orig_unmount() unmount_callback() mp.unmount = new_unmount return mp def unmount(path): """Unmount the given mount point. This function shells out to the 'fusermount' program to unmount a FUSE filesystem. It works, but it would probably be better to use the 'unmount' method on the MountProcess class if you have it. On darwin, "diskutil umount " is called On freebsd, "umount " is called """ if sys.platform == "darwin": args = ["diskutil", "umount", path] elif "freebsd" in sys.platform: args = ["umount", path] else: args = ["fusermount", "-u", path] for num_tries in xrange(3): p = subprocess.Popen(args, stderr=subprocess.PIPE, stdout=subprocess.PIPE) (stdout, stderr) = p.communicate() if p.returncode == 0: return if "not mounted" in stderr: return if "not found" in stderr: return raise OSError("filesystem could not be unmounted: %s (%s) " % (path, str(stderr).rstrip(),)) class MountProcess(subprocess.Popen): """subprocess.Popen subclass managing a FUSE mount. This is a subclass of subprocess.Popen, designed for easy management of a FUSE mount in a background process. Rather than specifying the command to execute, pass in the FS object to be mounted, the target mount point and a dictionary of options for the underlying FUSE class. In order to be passed successfully to the new process, the FS object must be pickleable. This restriction may be lifted in the future. This class has an extra attribute 'path' giving the path to the mounted filesystem, and an extra method 'unmount' that will cleanly unmount it and terminate the process. By default, the spawning process will block until it receives notification that the filesystem has been mounted. Since this notification is sent by writing to a pipe, using the 'close_fds' option on this class will prevent it from being sent. You can also pass in the keyword argument 'nowait' to continue without waiting for notification. """ # This works by spawning a new python interpreter and passing it the # pickled (fs,path,opts) tuple on the command-line. Something like this: # # python -c "import MountProcess; MountProcess._do_mount('..data..') # # It would be more efficient to do a straight os.fork() here, and would # remove the need to pickle the FS. But API wise, I think it's much # better for mount() to return a Popen instance than just a pid. # # In the future this class could implement its own forking logic and # just copy the relevant bits of the Popen interface. For now, this # spawn-a-new-interpreter solution is the easiest to get up and running. unmount_timeout = 5 def __init__(self, fs, path, fuse_opts={}, nowait=False, **kwds): self.path = path if nowait or kwds.get("close_fds", False): if PY3: cmd = "from pickle import loads;" else: cmd = "from cPickle import loads;" #cmd = 'import cPickle; ' cmd = cmd + 'data = loads(%s); ' cmd = cmd + 'from fs.expose.fuse import MountProcess; ' cmd = cmd + 'MountProcess._do_mount_nowait(data)' cmd = cmd % (repr(cPickle.dumps((fs, path, fuse_opts), -1)),) cmd = [sys.executable, "-c", cmd] super(MountProcess, self).__init__(cmd, **kwds) else: (r, w) = os.pipe() if PY3: cmd = "from pickle import loads;" else: cmd = "from cPickle import loads;" #cmd = 'import cPickle; ' cmd = cmd + 'data = loads(%s); ' cmd = cmd + 'from fs.expose.fuse import MountProcess; ' cmd = cmd + 'MountProcess._do_mount_wait(data)' cmd = cmd % (repr(cPickle.dumps((fs, path, fuse_opts, r, w), -1)),) cmd = [sys.executable, "-c", cmd] super(MountProcess, self).__init__(cmd, **kwds) os.close(w) byte = os.read(r, 1) if byte != b("S"): err_text = os.read(r, 20) self.terminate() if hasattr(err_text, 'decode'): err_text = err_text.decode(NATIVE_ENCODING) raise RuntimeError("FUSE error: " + err_text) def unmount(self): """Cleanly unmount the FUSE filesystem, terminating this subprocess.""" self.terminate() def killme(): self.kill() time.sleep(0.1) try: unmount(self.path) except OSError: pass tmr = threading.Timer(self.unmount_timeout, killme) tmr.start() self.wait() tmr.cancel() if not hasattr(subprocess.Popen, "terminate"): def terminate(self): """Gracefully terminate the subprocess.""" os.kill(self.pid, signal.SIGTERM) if not hasattr(subprocess.Popen, "kill"): def kill(self): """Forcibly terminate the subprocess.""" os.kill(self.pid, signal.SIGKILL) @staticmethod def _do_mount_nowait(data): """Perform the specified mount, return without waiting.""" fs, path, opts = data opts["foreground"] = True def unmount_callback(): fs.close() opts["unmount_callback"] = unmount_callback mount(fs, path, *opts) @staticmethod def _do_mount_wait(data): """Perform the specified mount, signalling when ready.""" fs, path, opts, r, w = data os.close(r) opts["foreground"] = True successful = [] def ready_callback(): successful.append(True) os.write(w, b("S")) os.close(w) opts["ready_callback"] = ready_callback def unmount_callback(): fs.close() opts["unmount_callback"] = unmount_callback try: mount(fs, path, **opts) except Exception, e: os.write(w, b("E") + unicode(e).encode('ascii', errors='replace')) os.close(w) if not successful: os.write(w, b("EMount unsuccessful")) os.close(w) if __name__ == "__main__": import os import os.path from fs.tempfs import TempFS mount_point = os.path.join(os.environ["HOME"], "fs.expose.fuse") if not os.path.exists(mount_point): os.makedirs(mount_point) def ready_callback(): print "READY" mount(TempFS(), mount_point, foreground=True, ready_callback=ready_callback) fs-0.5.4/fs/expose/sftp.py0000664000175000017500000003241112512525115015333 0ustar willwill00000000000000""" fs.expose.sftp ============== Expose an FS object over SFTP (via paramiko). This module provides the necessary interfaces to expose an FS object over SFTP, plugging into the infrastructure provided by the 'paramiko' module. For simple usage, the class 'BaseSFTPServer' provides an all-in-one server class based on the standard SocketServer module. Use it like so:: server = BaseSFTPServer((hostname,port),fs) server.serve_forever() Note that the base class allows UNAUTHENTICATED ACCESS by default. For more serious work you will probably want to subclass it and override methods such as check_auth_password() and get_allowed_auths(). To integrate this module into an existing server framework based on paramiko, the 'SFTPServerInterface' class provides a concrete implementation of the paramiko.SFTPServerInterface protocol. If you don't understand what this is, you probably don't want to use it. """ from __future__ import with_statement import os import stat as statinfo import time import SocketServer import threading import paramiko from fs.base import flags_to_mode from fs.path import * from fs.errors import * from fs.local_functools import wraps from fs.filelike import StringIO from fs.utils import isdir # Default host key used by BaseSFTPServer # DEFAULT_HOST_KEY = paramiko.RSAKey.from_private_key(StringIO( "-----BEGIN RSA PRIVATE KEY-----\n" \ "MIICXgIBAAKCAIEAl7sAF0x2O/HwLhG68b1uG8KHSOTqe3Cdlj5i/1RhO7E2BJ4B\n" \ "3jhKYDYtupRnMFbpu7fb21A24w3Y3W5gXzywBxR6dP2HgiSDVecoDg2uSYPjnlDk\n" \ "HrRuviSBG3XpJ/awn1DObxRIvJP4/sCqcMY8Ro/3qfmid5WmMpdCZ3EBeC0CAwEA\n" \ "AQKCAIBSGefUs5UOnr190C49/GiGMN6PPP78SFWdJKjgzEHI0P0PxofwPLlSEj7w\n" \ "RLkJWR4kazpWE7N/bNC6EK2pGueMN9Ag2GxdIRC5r1y8pdYbAkuFFwq9Tqa6j5B0\n" \ "GkkwEhrcFNBGx8UfzHESXe/uE16F+e8l6xBMcXLMJVo9Xjui6QJBAL9MsJEx93iO\n" \ "zwjoRpSNzWyZFhiHbcGJ0NahWzc3wASRU6L9M3JZ1VkabRuWwKNuEzEHNK8cLbRl\n" \ "TyH0mceWXcsCQQDLDEuWcOeoDteEpNhVJFkXJJfwZ4Rlxu42MDsQQ/paJCjt2ONU\n" \ "WBn/P6iYDTvxrt/8+CtLfYc+QQkrTnKn3cLnAkEAk3ixXR0h46Rj4j/9uSOfyyow\n" \ "qHQunlZ50hvNz8GAm4TU7v82m96449nFZtFObC69SLx/VsboTPsUh96idgRrBQJA\n" \ "QBfGeFt1VGAy+YTLYLzTfnGnoFQcv7+2i9ZXnn/Gs9N8M+/lekdBFYgzoKN0y4pG\n" \ "2+Q+Tlr2aNlAmrHtkT13+wJAJVgZATPI5X3UO0Wdf24f/w9+OY+QxKGl86tTQXzE\n" \ "4bwvYtUGufMIHiNeWP66i6fYCucXCMYtx6Xgu2hpdZZpFw==\n" \ "-----END RSA PRIVATE KEY-----\n" )) def report_sftp_errors(func): """Decorator to catch and report FS errors as SFTP error codes. Any FSError exceptions are caught and translated into an appropriate return code, while other exceptions are passed through untouched. """ @wraps(func) def wrapper(*args,**kwds): try: return func(*args, **kwds) except ResourceNotFoundError, e: return paramiko.SFTP_NO_SUCH_FILE except UnsupportedError, e: return paramiko.SFTP_OP_UNSUPPORTED except FSError, e: return paramiko.SFTP_FAILURE return wrapper class SFTPServerInterface(paramiko.SFTPServerInterface): """SFTPServerInterface implementation that exposes an FS object. This SFTPServerInterface subclass expects a single additional argument, the fs object to be exposed. Use it to set up a transport subsystem handler like so:: t.set_subsystem_handler("sftp",SFTPServer,SFTPServerInterface,fs) If this all looks too complicated, you might consider the BaseSFTPServer class also provided by this module - it automatically creates the enclosing paramiko server infrastructure. """ def __init__(self, server, fs, encoding=None, *args, **kwds): self.fs = fs if encoding is None: encoding = "utf8" self.encoding = encoding super(SFTPServerInterface,self).__init__(server, *args, **kwds) def close(self): # Close the pyfs file system and dereference it. self.fs.close() self.fs = None @report_sftp_errors def open(self, path, flags, attr): return SFTPHandle(self, path, flags) @report_sftp_errors def list_folder(self, path): if not isinstance(path, unicode): path = path.decode(self.encoding) stats = [] for entry in self.fs.listdir(path, absolute=True): stat = self.stat(entry) if not isinstance(stat, int): stats.append(stat) return stats @report_sftp_errors def stat(self, path): if not isinstance(path, unicode): path = path.decode(self.encoding) info = self.fs.getinfo(path) stat = paramiko.SFTPAttributes() stat.filename = basename(path).encode(self.encoding) stat.st_size = info.get("size") if 'st_atime' in info: stat.st_atime = info.get('st_atime') elif 'accessed_time' in info: stat.st_atime = time.mktime(info.get("accessed_time").timetuple()) if 'st_mtime' in info: stat.st_mtime = info.get('st_mtime') else: if 'modified_time' in info: stat.st_mtime = time.mktime(info.get("modified_time").timetuple()) if isdir(self.fs, path, info): stat.st_mode = 0777 | statinfo.S_IFDIR else: stat.st_mode = 0777 | statinfo.S_IFREG return stat def lstat(self, path): return self.stat(path) @report_sftp_errors def remove(self, path): if not isinstance(path, unicode): path = path.decode(self.encoding) self.fs.remove(path) return paramiko.SFTP_OK @report_sftp_errors def rename(self, oldpath, newpath): if not isinstance(oldpath, unicode): oldpath = oldpath.decode(self.encoding) if not isinstance(newpath, unicode): newpath = newpath.decode(self.encoding) if self.fs.isfile(oldpath): self.fs.move(oldpath, newpath) else: self.fs.movedir(oldpath, newpath) return paramiko.SFTP_OK @report_sftp_errors def mkdir(self, path, attr): if not isinstance(path, unicode): path = path.decode(self.encoding) self.fs.makedir(path) return paramiko.SFTP_OK @report_sftp_errors def rmdir(self, path): if not isinstance(path, unicode): path = path.decode(self.encoding) self.fs.removedir(path) return paramiko.SFTP_OK def canonicalize(self, path): try: return abspath(normpath(path)).encode(self.encoding) except BackReferenceError: # If the client tries to use backrefs to escape root, gently # nudge them back to /. return '/' @report_sftp_errors def chattr(self, path, attr): # f.truncate() is implemented by setting the size attr. # Any other attr requests fail out. if attr._flags: if attr._flags != attr.FLAG_SIZE: raise UnsupportedError with self.fs.open(path,"r+") as f: f.truncate(attr.st_size) return paramiko.SFTP_OK def readlink(self, path): return paramiko.SFTP_OP_UNSUPPORTED def symlink(self, path): return paramiko.SFTP_OP_UNSUPPORTED class SFTPHandle(paramiko.SFTPHandle): """SFTP file handler pointing to a file in an FS object. This is a simple file wrapper for SFTPServerInterface, passing read and write requests directly through the to underlying file from the FS. """ def __init__(self, owner, path, flags): super(SFTPHandle, self).__init__(flags) mode = flags_to_mode(flags) self.owner = owner if not isinstance(path, unicode): path = path.decode(self.owner.encoding) self.path = path self._file = owner.fs.open(path, mode) @report_sftp_errors def close(self): self._file.close() return paramiko.SFTP_OK @report_sftp_errors def read(self, offset, length): self._file.seek(offset) return self._file.read(length) @report_sftp_errors def write(self, offset, data): self._file.seek(offset) self._file.write(data) return paramiko.SFTP_OK def stat(self): return self.owner.stat(self.path) def chattr(self,attr): return self.owner.chattr(self.path, attr) class SFTPServer(paramiko.SFTPServer): """ An SFTPServer class that closes the filesystem when done. """ def finish_subsystem(self): # Close the SFTPServerInterface, it will close the pyfs file system. self.server.close() super(SFTPServer, self).finish_subsystem() class SFTPRequestHandler(SocketServer.BaseRequestHandler): """SocketServer RequestHandler subclass for BaseSFTPServer. This RequestHandler subclass creates a paramiko Transport, sets up the sftp subsystem, and hands off to the transport's own request handling thread. """ timeout = 60 auth_timeout = 60 def setup(self): """ Creates the SSH transport. Sets security options. """ self.transport = paramiko.Transport(self.request) self.transport.load_server_moduli() so = self.transport.get_security_options() so.digests = ('hmac-sha1', ) so.compression = ('zlib@openssh.com', 'none') self.transport.add_server_key(self.server.host_key) self.transport.set_subsystem_handler("sftp", SFTPServer, SFTPServerInterface, self.server.fs, encoding=self.server.encoding) def handle(self): """ Start the paramiko server, this will start a thread to handle the connection. """ self.transport.start_server(server=BaseServerInterface()) # TODO: I like the code below _in theory_ but it does not work as I expected. # Figure out how to actually time out a new client if they fail to auth in a # certain amount of time. #chan = self.transport.accept(self.auth_timeout) #if chan is None: # self.transport.close() def handle_timeout(self): try: self.transport.close() finally: super(SFTPRequestHandler, self).handle_timeout() class ThreadedTCPServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer): pass class BaseSFTPServer(ThreadedTCPServer): """SocketServer.TCPServer subclass exposing an FS via SFTP. Operation is in the standard SocketServer style. The target FS object can be passed into the constructor, or set as an attribute on the server:: server = BaseSFTPServer((hostname,port),fs) server.serve_forever() It is also possible to specify the host key used by the sever by setting the 'host_key' attribute. If this is not specified, it will default to the key found in the DEFAULT_HOST_KEY variable. """ # If the server stops/starts quickly, don't fail because of # "port in use" error. allow_reuse_address = True def __init__(self, address, fs=None, encoding=None, host_key=None, RequestHandlerClass=None): self.fs = fs self.encoding = encoding if host_key is None: host_key = DEFAULT_HOST_KEY self.host_key = host_key if RequestHandlerClass is None: RequestHandlerClass = SFTPRequestHandler SocketServer.TCPServer.__init__(self, address, RequestHandlerClass) def shutdown_request(self, request): # Prevent TCPServer from closing the connection prematurely return def close_request(self, request): # Prevent TCPServer from closing the connection prematurely return class BaseServerInterface(paramiko.ServerInterface): """ Paramiko ServerInterface implementation that performs user authentication. Note that this base class allows UNAUTHENTICATED ACCESS to the exposed FS. This is intentional, since we can't guess what your authentication needs are. To protect the exposed FS, override the following methods: * get_allowed_auths Determine the allowed auth modes * check_auth_none Check auth with no credentials * check_auth_password Check auth with a password * check_auth_publickey Check auth with a public key """ def check_channel_request(self, kind, chanid): if kind == 'session': return paramiko.OPEN_SUCCEEDED return paramiko.OPEN_FAILED_ADMINISTRATIVELY_PROHIBITED def check_auth_none(self, username): """Check whether the user can proceed without authentication.""" return paramiko.AUTH_SUCCESSFUL def check_auth_publickey(self, username,key): """Check whether the given public key is valid for authentication.""" return paramiko.AUTH_FAILED def check_auth_password(self, username, password): """Check whether the given password is valid for authentication.""" return paramiko.AUTH_FAILED def get_allowed_auths(self,username): """Return string containing a comma separated list of allowed auth modes. The available modes are "node", "password" and "publickey". """ return "none" # When called from the command-line, expose a TempFS for testing purposes if __name__ == "__main__": from fs.tempfs import TempFS server = BaseSFTPServer(("localhost",8022),TempFS()) try: #import rpdb2; rpdb2.start_embedded_debugger('password') server.serve_forever() except (SystemExit,KeyboardInterrupt): server.server_close() fs-0.5.4/fs/expose/dokan/0000755000000000000000000000000012621617365015106 5ustar rootroot00000000000000fs-0.5.4/fs/expose/dokan/libdokan.py0000664000175000017500000001475212512525115017246 0ustar willwill00000000000000# Copyright (c) 2009-2010, Cloud Matrix Pty. Ltd. # All rights reserved; available under the terms of the MIT License. """ fs.expose.dokan.libdokan: low-level ctypes interface to Dokan """ from ctypes import * try: DokanMain = windll.Dokan.DokanMain DokanVersion = windll.Dokan.DokanVersion except AttributeError: raise ImportError("Dokan DLL not found") from ctypes.wintypes import * ULONG64 = c_ulonglong ULONGLONG = c_ulonglong PULONGLONG = POINTER(ULONGLONG) UCHAR = c_ubyte LPDWORD = POINTER(DWORD) LONGLONG = c_longlong try: USHORT = USHORT except NameError: # Not available in older python versions USHORT = c_ushort DokanVersion.restype = ULONG DokanVersion.argtypes = () if DokanVersion() < 392: # ths is release 0.5.3 raise ImportError("Dokan DLL is too old") MAX_PATH = 260 class FILETIME(Structure): _fields_ = [ ("dwLowDateTime", DWORD), ("dwHighDateTime", DWORD), ] class WIN32_FIND_DATAW(Structure): _fields_ = [ ("dwFileAttributes", DWORD), ("ftCreationTime", FILETIME), ("ftLastAccessTime", FILETIME), ("ftLastWriteTime", FILETIME), ("nFileSizeHigh", DWORD), ("nFileSizeLow", DWORD), ("dwReserved0", DWORD), ("dwReserved1", DWORD), ("cFileName", WCHAR * MAX_PATH), ("cAlternateFileName", WCHAR * 14), ] class BY_HANDLE_FILE_INFORMATION(Structure): _fields_ = [ ('dwFileAttributes', DWORD), ('ftCreationTime', FILETIME), ('ftLastAccessTime', FILETIME), ('ftLastWriteTime', FILETIME), ('dwVolumeSerialNumber', DWORD), ('nFileSizeHigh', DWORD), ('nFileSizeLow', DWORD), ('nNumberOfLinks', DWORD), ('nFileIndexHigh', DWORD), ('nFileIndexLow', DWORD), ] class DOKAN_OPTIONS(Structure): _fields_ = [ ("DriveLetter", WCHAR), ("ThreadCount", USHORT), ("Options", ULONG), ("GlobalContext", ULONG64), ] class DOKAN_FILE_INFO(Structure): _fields_ = [ ("Context", ULONG64), ("DokanContext", ULONG64), ("DokanOptions", POINTER(DOKAN_OPTIONS)), ("ProcessId", ULONG), ("IsDirectory", UCHAR), ("DeleteOnClose", UCHAR), ("PagingIO", UCHAR), ("SyncronousIo", UCHAR), ("Nocache", UCHAR), ("WriteToEndOfFile", UCHAR), ] PDOKAN_FILE_INFO = POINTER(DOKAN_FILE_INFO) PFillFindData = WINFUNCTYPE(c_int,POINTER(WIN32_FIND_DATAW),PDOKAN_FILE_INFO) class DOKAN_OPERATIONS(Structure): _fields_ = [ ("CreateFile", CFUNCTYPE(c_int, LPCWSTR, # FileName DWORD, # DesiredAccess DWORD, # ShareMode DWORD, # CreationDisposition DWORD, # FlagsAndAttributes PDOKAN_FILE_INFO)), ("OpenDirectory", CFUNCTYPE(c_int, LPCWSTR, # FileName PDOKAN_FILE_INFO)), ("CreateDirectory", CFUNCTYPE(c_int, LPCWSTR, # FileName PDOKAN_FILE_INFO)), ("Cleanup", CFUNCTYPE(c_int, LPCWSTR, # FileName PDOKAN_FILE_INFO)), ("CloseFile", CFUNCTYPE(c_int, LPCWSTR, # FileName PDOKAN_FILE_INFO)), ("ReadFile", CFUNCTYPE(c_int, LPCWSTR, # FileName POINTER(c_char), # Buffer DWORD, # NumberOfBytesToRead LPDWORD, # NumberOfBytesRead LONGLONG, # Offset PDOKAN_FILE_INFO)), ("WriteFile", CFUNCTYPE(c_int, LPCWSTR, # FileName POINTER(c_char), # Buffer DWORD, # NumberOfBytesToWrite LPDWORD, # NumberOfBytesWritten LONGLONG, # Offset PDOKAN_FILE_INFO)), ("FlushFileBuffers", CFUNCTYPE(c_int, LPCWSTR, # FileName PDOKAN_FILE_INFO)), ("GetFileInformation", CFUNCTYPE(c_int, LPCWSTR, # FileName POINTER(BY_HANDLE_FILE_INFORMATION), # Buffer PDOKAN_FILE_INFO)), ("FindFiles", CFUNCTYPE(c_int, LPCWSTR, # PathName PFillFindData, # call this function with PWIN32_FIND_DATAW PDOKAN_FILE_INFO)), ("FindFilesWithPattern", CFUNCTYPE(c_int, LPCWSTR, # PathName LPCWSTR, # SearchPattern PFillFindData, #call this function with PWIN32_FIND_DATAW PDOKAN_FILE_INFO)), ("SetFileAttributes", CFUNCTYPE(c_int, LPCWSTR, # FileName DWORD, # FileAttributes PDOKAN_FILE_INFO)), ("SetFileTime", CFUNCTYPE(c_int, LPCWSTR, # FileName POINTER(FILETIME), # CreationTime POINTER(FILETIME), # LastAccessTime POINTER(FILETIME), # LastWriteTime PDOKAN_FILE_INFO)), ("DeleteFile", CFUNCTYPE(c_int, LPCWSTR, # FileName PDOKAN_FILE_INFO)), ("DeleteDirectory", CFUNCTYPE(c_int, LPCWSTR, # FileName PDOKAN_FILE_INFO)), ("MoveFile", CFUNCTYPE(c_int, LPCWSTR, # ExistingFileName LPCWSTR, # NewFileName BOOL, # ReplaceExisiting PDOKAN_FILE_INFO)), ("SetEndOfFile", CFUNCTYPE(c_int, LPCWSTR, # FileName LONGLONG, # Length PDOKAN_FILE_INFO)), ("SetAllocationSize", CFUNCTYPE(c_int, LPCWSTR, # FileName LONGLONG, # Length PDOKAN_FILE_INFO)), ("LockFile", CFUNCTYPE(c_int, LPCWSTR, # FileName LONGLONG, # ByteOffset LONGLONG, # Length PDOKAN_FILE_INFO)), ("UnlockFile", CFUNCTYPE(c_int, LPCWSTR, # FileName LONGLONG, # ByteOffset LONGLONG, # Length PDOKAN_FILE_INFO)), ("GetDiskFreeSpaceEx", CFUNCTYPE(c_int, PULONGLONG, # FreeBytesAvailable PULONGLONG, # TotalNumberOfBytes PULONGLONG, # TotalNumberOfFreeBytes PDOKAN_FILE_INFO)), ("GetVolumeInformation", CFUNCTYPE(c_int, POINTER(c_wchar), # VolumeNameBuffer DWORD, # VolumeNameSize in num of chars LPDWORD, # VolumeSerialNumber LPDWORD, # MaximumComponentLength in num of chars LPDWORD, # FileSystemFlags POINTER(c_wchar), # FileSystemNameBuffer DWORD, # FileSystemNameSize in num of chars PDOKAN_FILE_INFO)), ("Unmount", CFUNCTYPE(c_int, PDOKAN_FILE_INFO)), ] DokanMain.restype = c_int DokanMain.argtypes = ( POINTER(DOKAN_OPTIONS), POINTER(DOKAN_OPERATIONS), ) DokanUnmount = windll.Dokan.DokanUnmount DokanUnmount.restype = BOOL DokanUnmount.argtypes = ( WCHAR, ) DokanIsNameInExpression = windll.Dokan.DokanIsNameInExpression DokanIsNameInExpression.restype = BOOL DokanIsNameInExpression.argtypes = ( LPCWSTR, # pattern LPCWSTR, # name BOOL, # ignore case ) DokanDriverVersion = windll.Dokan.DokanDriverVersion DokanDriverVersion.restype = ULONG DokanDriverVersion.argtypes = ( ) DokanResetTimeout = windll.Dokan.DokanResetTimeout DokanResetTimeout.restype = BOOL DokanResetTimeout.argtypes = ( ULONG, #timeout PDOKAN_FILE_INFO, # file info pointer ) fs-0.5.4/fs/expose/dokan/__init__.py0000664000175000017500000011212312512525115017211 0ustar willwill00000000000000""" fs.expose.dokan =============== Expose an FS object to the native filesystem via Dokan. This module provides the necessary interfaces to mount an FS object into the local filesystem using Dokan on win32:: http://dokan-dev.net/en/ For simple usage, the function 'mount' takes an FS object and a drive letter, and exposes the given FS as that drive:: >>> from fs.memoryfs import MemoryFS >>> from fs.expose import dokan >>> fs = MemoryFS() >>> mp = dokan.mount(fs,"Q") >>> mp.drive 'Q' >>> mp.path 'Q:\\' >>> mp.unmount() The above spawns a new background process to manage the Dokan event loop, which can be controlled through the returned subprocess.Popen object. To avoid spawning a new process, set the 'foreground' option:: >>> # This will block until the filesystem is unmounted >>> dokan.mount(fs,"Q",foreground=True) Any additional options for the Dokan process can be passed as keyword arguments to the 'mount' function. If you require finer control over the creation of the Dokan process, you can instantiate the MountProcess class directly. It accepts all options available to subprocess.Popen:: >>> from subprocess import PIPE >>> mp = dokan.MountProcess(fs,"Q",stderr=PIPE) >>> dokan_errors = mp.communicate()[1] If you are exposing an untrusted filesystem, you may like to apply the wrapper class Win32SafetyFS before passing it into dokan. This will take a number of steps to avoid suspicious operations on windows, such as hiding autorun files. The binding to Dokan is created via ctypes. Due to the very stable ABI of win32, this should work without further configuration on just about all systems with Dokan installed. """ # Copyright (c) 2009-2010, Cloud Matrix Pty. Ltd. # All rights reserved; available under the terms of the MIT License. from __future__ import with_statement import sys import os import signal import errno import time import stat as statinfo import subprocess import cPickle import datetime import ctypes from collections import deque from fs.base import threading from fs.errors import * from fs.path import * from fs.local_functools import wraps from fs.wrapfs import WrapFS try: import libdokan except (NotImplementedError, EnvironmentError, ImportError, NameError,): is_available = False sys.modules.pop("fs.expose.dokan.libdokan", None) libdokan = None else: is_available = True from ctypes.wintypes import LPCWSTR, WCHAR kernel32 = ctypes.windll.kernel32 import logging logger = logging.getLogger("fs.expose.dokan") # Options controlling the behavior of the Dokan filesystem DOKAN_OPTION_DEBUG = 1 DOKAN_OPTION_STDERR = 2 DOKAN_OPTION_ALT_STREAM = 4 DOKAN_OPTION_KEEP_ALIVE = 8 DOKAN_OPTION_NETWORK = 16 DOKAN_OPTION_REMOVABLE = 32 # Error codes returned by DokanMain DOKAN_SUCCESS = 0 DOKAN_ERROR = -1 DOKAN_DRIVE_LETTER_ERROR = -2 DOKAN_DRIVER_INSTALL_ERROR = -3 DOKAN_START_ERROR = -4 DOKAN_MOUNT_ERROR = -5 # Misc windows constants FILE_LIST_DIRECTORY = 0x01 FILE_SHARE_READ = 0x01 FILE_SHARE_WRITE = 0x02 FILE_FLAG_BACKUP_SEMANTICS = 0x02000000 FILE_FLAG_OVERLAPPED = 0x40000000 FILE_ATTRIBUTE_ARCHIVE = 32 FILE_ATTRIBUTE_COMPRESSED = 2048 FILE_ATTRIBUTE_DIRECTORY = 16 FILE_ATTRIBUTE_HIDDEN = 2 FILE_ATTRIBUTE_NORMAL = 128 FILE_ATTRIBUTE_OFFLINE = 4096 FILE_ATTRIBUTE_READONLY = 1 FILE_ATTRIBUTE_SYSTEM = 4 FILE_ATTRIBUTE_TEMPORARY = 4 CREATE_NEW = 1 CREATE_ALWAYS = 2 OPEN_EXISTING = 3 OPEN_ALWAYS = 4 TRUNCATE_EXISTING = 5 FILE_GENERIC_READ = 1179785 FILE_GENERIC_WRITE = 1179926 REQ_GENERIC_READ = 0x80 | 0x08 | 0x01 REQ_GENERIC_WRITE = 0x004 | 0x0100 | 0x002 | 0x0010 ERROR_ACCESS_DENIED = 5 ERROR_LOCK_VIOLATION = 33 ERROR_NOT_SUPPORTED = 50 ERROR_FILE_EXISTS = 80 ERROR_DIR_NOT_EMPTY = 145 ERROR_NOT_LOCKED = 158 ERROR_LOCK_FAILED = 167 ERROR_ALREADY_EXISTS = 183 ERROR_LOCKED = 212 ERROR_INVALID_LOCK_RANGE = 306 # Some useful per-process global information NATIVE_ENCODING = sys.getfilesystemencoding() DATETIME_ZERO = datetime.datetime(1,1,1,0,0,0) DATETIME_STARTUP = datetime.datetime.utcnow() FILETIME_UNIX_EPOCH = 116444736000000000 def handle_fs_errors(func): """Method decorator to report FS errors in the appropriate way. This decorator catches all FS errors and translates them into an equivalent OSError, then returns the negated error number. It also makes the function return zero instead of None as an indication of successful execution. """ name = func.__name__ func = convert_fs_errors(func) @wraps(func) def wrapper(*args,**kwds): try: res = func(*args,**kwds) except OSError, e: if e.errno: res = -1 * _errno2syserrcode(e.errno) else: res = -1 except Exception, e: raise else: if res is None: res = 0 return res return wrapper # During long-running operations, Dokan requires that the DokanResetTimeout # function be called periodically to indicate the progress is still being # made. Unfortunately we don't have any facility for the underlying FS # to make these calls for us, so we have to hack around it. # # The idea is to use a single background thread to monitor all active Dokan # method calls, resetting the timeout until they have completed. Note that # this completely undermines the point of DokanResetTimeout as it's now # possible for a deadlock to hang the entire filesystem. _TIMEOUT_PROTECT_THREAD = None _TIMEOUT_PROTECT_LOCK = threading.Lock() _TIMEOUT_PROTECT_COND = threading.Condition(_TIMEOUT_PROTECT_LOCK) _TIMEOUT_PROTECT_QUEUE = deque() _TIMEOUT_PROTECT_WAIT_TIME = 4 * 60 _TIMEOUT_PROTECT_RESET_TIME = 5 * 60 * 1000 def _start_timeout_protect_thread(): """Start the background thread used to protect dokan from timeouts. This function starts the background thread that monitors calls into the dokan API and resets their timeouts. It's safe to call this more than once, only a single thread will be started. """ global _TIMEOUT_PROTECT_THREAD with _TIMEOUT_PROTECT_LOCK: if _TIMEOUT_PROTECT_THREAD is None: target = _run_timeout_protect_thread _TIMEOUT_PROTECT_THREAD = threading.Thread(target=target) _TIMEOUT_PROTECT_THREAD.daemon = True _TIMEOUT_PROTECT_THREAD.start() def _run_timeout_protect_thread(): while True: with _TIMEOUT_PROTECT_COND: try: (when,info,finished) = _TIMEOUT_PROTECT_QUEUE.popleft() except IndexError: _TIMEOUT_PROTECT_COND.wait() continue if finished: continue now = time.time() wait_time = max(0,_TIMEOUT_PROTECT_WAIT_TIME - now + when) time.sleep(wait_time) with _TIMEOUT_PROTECT_LOCK: if finished: continue libdokan.DokanResetTimeout(_TIMEOUT_PROTECT_RESET_TIME,info) _TIMEOUT_PROTECT_QUEUE.append((now+wait_time,info,finished)) def timeout_protect(func): """Method decorator to enable timeout protection during call. This decorator adds an entry to the timeout protect queue before executing the function, and marks it as finished when the function exits. """ @wraps(func) def wrapper(self,*args): if _TIMEOUT_PROTECT_THREAD is None: _start_timeout_protect_thread() info = args[-1] finished = [] try: with _TIMEOUT_PROTECT_COND: _TIMEOUT_PROTECT_QUEUE.append((time.time(),info,finished)) _TIMEOUT_PROTECT_COND.notify() return func(self,*args) finally: with _TIMEOUT_PROTECT_LOCK: finished.append(True) return wrapper MIN_FH = 100 class FSOperations(object): """Object delegating all DOKAN_OPERATIONS pointers to an FS object.""" def __init__(self, fs, fsname="Dokan FS", volname="Dokan Volume"): if libdokan is None: msg = "dokan library (http://dokan-dev.net/en/) is not available" raise OSError(msg) self.fs = fs self.fsname = fsname self.volname = volname self._files_by_handle = {} self._files_lock = threading.Lock() self._next_handle = MIN_FH # Windows requires us to implement a kind of "lazy deletion", where # a handle is marked for deletion but this is not actually done # until the handle is closed. This set monitors pending deletes. self._pending_delete = set() # Since pyfilesystem has no locking API, we manage file locks # in memory. This maps paths to a list of current locks. self._active_locks = PathMap() # Dokan expects a succesful write() to be reflected in the file's # reported size, but the FS might buffer writes and prevent this. # We explicitly keep track of the size Dokan expects a file to be. # This dict is indexed by path, then file handle. self._files_size_written = PathMap() def get_ops_struct(self): """Get a DOKAN_OPERATIONS struct mapping to our methods.""" struct = libdokan.DOKAN_OPERATIONS() for (nm,typ) in libdokan.DOKAN_OPERATIONS._fields_: setattr(struct,nm,typ(getattr(self,nm))) return struct def _get_file(self, fh): """Get the information associated with the given file handle.""" try: return self._files_by_handle[fh] except KeyError: raise FSError("invalid file handle") def _reg_file(self, f, path): """Register a new file handle for the given file and path.""" self._files_lock.acquire() try: fh = self._next_handle self._next_handle += 1 lock = threading.Lock() self._files_by_handle[fh] = (f,path,lock) if path not in self._files_size_written: self._files_size_written[path] = {} self._files_size_written[path][fh] = 0 return fh finally: self._files_lock.release() def _rereg_file(self, fh, f): """Re-register the file handle for the given file. This might be necessary if we are required to write to a file after its handle was closed (e.g. to complete an async write). """ self._files_lock.acquire() try: (f2, path, lock) = self._files_by_handle[fh] assert f2.closed self._files_by_handle[fh] = (f, path, lock) return fh finally: self._files_lock.release() def _del_file(self, fh): """Unregister the given file handle.""" self._files_lock.acquire() try: (f, path, lock) = self._files_by_handle.pop(fh) del self._files_size_written[path][fh] if not self._files_size_written[path]: del self._files_size_written[path] finally: self._files_lock.release() def _is_pending_delete(self, path): """Check if the given path is pending deletion. This is true if the path or any of its parents have been marked as pending deletion, false otherwise. """ for ppath in recursepath(path): if ppath in self._pending_delete: return True return False def _check_lock(self, path, offset, length, info, locks=None): """Check whether the given file range is locked. This method implements basic lock checking. It checks all the locks held against the given file, and if any overlap the given byte range then it returns -ERROR_LOCKED. If the range is not locked, it will return zero. """ if locks is None: with self._files_lock: try: locks = self._active_locks[path] except KeyError: return 0 for (lh, lstart, lend) in locks: if info is not None and info.contents.Context == lh: continue if lstart >= offset + length: continue if lend <= offset: continue return -ERROR_LOCKED return 0 @timeout_protect @handle_fs_errors def CreateFile(self, path, access, sharing, disposition, flags, info): path = normpath(path) # Can't open files that are pending delete. if self._is_pending_delete(path): return -ERROR_ACCESS_DENIED # If no access rights are requestsed, only basic metadata is queried. if not access: if self.fs.isdir(path): info.contents.IsDirectory = True elif not self.fs.exists(path): raise ResourceNotFoundError(path) return # This is where we'd convert the access mask into an appropriate # mode string. Unfortunately, I can't seem to work out all the # details. I swear MS Word is trying to write to files that it # opens without asking for write permission. # For now, just set the mode based on disposition flag. retcode = 0 if disposition == CREATE_ALWAYS: if self.fs.exists(path): retcode = ERROR_ALREADY_EXISTS mode = "w+b" elif disposition == OPEN_ALWAYS: if not self.fs.exists(path): mode = "w+b" else: retcode = ERROR_ALREADY_EXISTS mode = "r+b" elif disposition == OPEN_EXISTING: mode = "r+b" elif disposition == TRUNCATE_EXISTING: if not self.fs.exists(path): raise ResourceNotFoundError(path) mode = "w+b" elif disposition == CREATE_NEW: if self.fs.exists(path): return -ERROR_ALREADY_EXISTS mode = "w+b" else: mode = "r+b" # Try to open the requested file. It may actually be a directory. info.contents.Context = 1 try: f = self.fs.open(path, mode) print path, mode, repr(f) except ResourceInvalidError: info.contents.IsDirectory = True except FSError: # Sadly, win32 OSFS will raise all kinds of strange errors # if you try to open() a directory. Need to check by hand. if self.fs.isdir(path): info.contents.IsDirectory = True else: raise else: info.contents.Context = self._reg_file(f, path) return retcode @timeout_protect @handle_fs_errors def OpenDirectory(self, path, info): path = normpath(path) if self._is_pending_delete(path): raise ResourceNotFoundError(path) if not self.fs.isdir(path): if not self.fs.exists(path): raise ResourceNotFoundError(path) else: raise ResourceInvalidError(path) info.contents.IsDirectory = True @timeout_protect @handle_fs_errors def CreateDirectory(self, path, info): path = normpath(path) if self._is_pending_delete(path): return -ERROR_ACCESS_DENIED self.fs.makedir(path) info.contents.IsDirectory = True @timeout_protect @handle_fs_errors def Cleanup(self, path, info): path = normpath(path) if info.contents.IsDirectory: if info.contents.DeleteOnClose: self.fs.removedir(path) self._pending_delete.remove(path) else: (file, _, lock) = self._get_file(info.contents.Context) lock.acquire() try: file.close() if info.contents.DeleteOnClose: self.fs.remove(path) self._pending_delete.remove(path) self._del_file(info.contents.Context) info.contents.Context = 0 finally: lock.release() @timeout_protect @handle_fs_errors def CloseFile(self, path, info): if info.contents.Context >= MIN_FH: (file, _, lock) = self._get_file(info.contents.Context) lock.acquire() try: file.close() self._del_file(info.contents.Context) finally: lock.release() info.contents.Context = 0 @timeout_protect @handle_fs_errors def ReadFile(self, path, buffer, nBytesToRead, nBytesRead, offset, info): path = normpath(path) (file, _, lock) = self._get_file(info.contents.Context) lock.acquire() try: errno = self._check_lock(path, offset, nBytesToRead, info) if errno: return errno # This may be called after Cleanup, meaning we # need to re-open the file. if file.closed: file = self.fs.open(path, file.mode) self._rereg_file(info.contents.Context, file) file.seek(offset) data = file.read(nBytesToRead) ctypes.memmove(buffer, ctypes.create_string_buffer(data), len(data)) nBytesRead[0] = len(data) finally: lock.release() @timeout_protect @handle_fs_errors def WriteFile(self, path, buffer, nBytesToWrite, nBytesWritten, offset, info): path = normpath(path) fh = info.contents.Context (file, _, lock) = self._get_file(fh) lock.acquire() try: errno = self._check_lock(path, offset, nBytesToWrite, info) if errno: return errno # This may be called after Cleanup, meaning we # need to re-open the file. if file.closed: file = self.fs.open(path, file.mode) self._rereg_file(info.contents.Context, file) if info.contents.WriteToEndOfFile: file.seek(0, os.SEEK_END) else: file.seek(offset) data = ctypes.create_string_buffer(nBytesToWrite) ctypes.memmove(data, buffer, nBytesToWrite) file.write(data.raw) nBytesWritten[0] = len(data.raw) try: size_written = self._files_size_written[path][fh] except KeyError: pass else: if offset + nBytesWritten[0] > size_written: new_size_written = offset + nBytesWritten[0] self._files_size_written[path][fh] = new_size_written finally: lock.release() @timeout_protect @handle_fs_errors def FlushFileBuffers(self, path, info): path = normpath(path) (file, _, lock) = self._get_file(info.contents.Context) lock.acquire() try: file.flush() finally: lock.release() @timeout_protect @handle_fs_errors def GetFileInformation(self, path, buffer, info): path = normpath(path) finfo = self.fs.getinfo(path) data = buffer.contents self._info2finddataw(path, finfo, data, info) try: written_size = max(self._files_size_written[path].values()) except KeyError: pass else: reported_size = (data.nFileSizeHigh << 32) + data.nFileSizeLow if written_size > reported_size: data.nFileSizeHigh = written_size >> 32 data.nFileSizeLow = written_size & 0xffffffff data.nNumberOfLinks = 1 @timeout_protect @handle_fs_errors def FindFiles(self, path, fillFindData, info): path = normpath(path) for (nm, finfo) in self.fs.listdirinfo(path): fpath = pathjoin(path, nm) if self._is_pending_delete(fpath): continue data = self._info2finddataw(fpath, finfo) fillFindData(ctypes.byref(data), info) @timeout_protect @handle_fs_errors def FindFilesWithPattern(self, path, pattern, fillFindData, info): path = normpath(path) for (nm, finfo) in self.fs.listdirinfo(path): fpath = pathjoin(path, nm) if self._is_pending_delete(fpath): continue if not libdokan.DokanIsNameInExpression(pattern, nm, True): continue data = self._info2finddataw(fpath, finfo, None) fillFindData(ctypes.byref(data), info) @timeout_protect @handle_fs_errors def SetFileAttributes(self, path, attrs, info): path = normpath(path) # TODO: decode various file attributes @timeout_protect @handle_fs_errors def SetFileTime(self, path, ctime, atime, mtime, info): path = normpath(path) # setting ctime is not supported if atime is not None: try: atime = _filetime2datetime(atime.contents) except ValueError: atime = None if mtime is not None: try: mtime = _filetime2datetime(mtime.contents) except ValueError: mtime = None # some programs demand this succeed; fake it try: self.fs.settimes(path, atime, mtime) except UnsupportedError: pass @timeout_protect @handle_fs_errors def DeleteFile(self, path, info): path = normpath(path) if not self.fs.isfile(path): if not self.fs.exists(path): raise ResourceNotFoundError(path) else: raise ResourceInvalidError(path) self._pending_delete.add(path) # the actual delete takes place in self.CloseFile() @timeout_protect @handle_fs_errors def DeleteDirectory(self, path, info): path = normpath(path) for nm in self.fs.listdir(path): if not self._is_pending_delete(pathjoin(path, nm)): raise DirectoryNotEmptyError(path) self._pending_delete.add(path) # the actual delete takes place in self.CloseFile() @timeout_protect @handle_fs_errors def MoveFile(self, src, dst, overwrite, info): # Close the file if we have an open handle to it. if info.contents.Context >= MIN_FH: (file, _, lock) = self._get_file(info.contents.Context) lock.acquire() try: file.close() self._del_file(info.contents.Context) finally: lock.release() src = normpath(src) dst = normpath(dst) if info.contents.IsDirectory: self.fs.movedir(src, dst, overwrite=overwrite) else: self.fs.move(src, dst, overwrite=overwrite) @timeout_protect @handle_fs_errors def SetEndOfFile(self, path, length, info): path = normpath(path) (file, _, lock) = self._get_file(info.contents.Context) lock.acquire() try: pos = file.tell() if length != pos: file.seek(length) file.truncate() if pos < length: file.seek(min(pos, length)) finally: lock.release() @handle_fs_errors def GetDiskFreeSpaceEx(self, nBytesAvail, nBytesTotal, nBytesFree, info): # This returns a stupidly large number if not info is available. # It's better to pretend an operation is possible and have it fail # than to pretend an operation will fail when it's actually possible. large_amount = 100 * 1024*1024*1024 nBytesFree[0] = self.fs.getmeta("free_space", large_amount) nBytesTotal[0] = self.fs.getmeta("total_space", 2 * large_amount) nBytesAvail[0] = nBytesFree[0] @handle_fs_errors def GetVolumeInformation(self, vnmBuf, vnmSz, sNum, maxLen, flags, fnmBuf, fnmSz, info): nm = ctypes.create_unicode_buffer(self.volname[:vnmSz-1]) sz = (len(nm.value) + 1) * ctypes.sizeof(ctypes.c_wchar) ctypes.memmove(vnmBuf, nm, sz) if sNum: sNum[0] = 0 if maxLen: maxLen[0] = 255 if flags: flags[0] = 0 nm = ctypes.create_unicode_buffer(self.fsname[:fnmSz-1]) sz = (len(nm.value) + 1) * ctypes.sizeof(ctypes.c_wchar) ctypes.memmove(fnmBuf, nm, sz) @timeout_protect @handle_fs_errors def SetAllocationSize(self, path, length, info): # I think this is supposed to reserve space for the file # but *not* actually move the end-of-file marker. # No way to do that in pyfs. return 0 @timeout_protect @handle_fs_errors def LockFile(self, path, offset, length, info): end = offset + length with self._files_lock: try: locks = self._active_locks[path] except KeyError: locks = self._active_locks[path] = [] else: errno = self._check_lock(path, offset, length, None, locks) if errno: return errno locks.append((info.contents.Context, offset, end)) return 0 @timeout_protect @handle_fs_errors def UnlockFile(self, path, offset, length, info): end = offset + length with self._files_lock: try: locks = self._active_locks[path] except KeyError: return -ERROR_NOT_LOCKED todel = [] for i, (lh, lstart, lend) in enumerate(locks): if info.contents.Context == lh: if lstart == offset: if lend == offset + length: todel.append(i) if not todel: return -ERROR_NOT_LOCKED for i in reversed(todel): del locks[i] return 0 @handle_fs_errors def Unmount(self, info): pass def _info2attrmask(self, path, info, hinfo=None): """Convert a file/directory info dict to a win32 file attribute mask.""" attrs = 0 st_mode = info.get("st_mode", None) if st_mode: if statinfo.S_ISDIR(st_mode): attrs |= FILE_ATTRIBUTE_DIRECTORY elif statinfo.S_ISREG(st_mode): attrs |= FILE_ATTRIBUTE_NORMAL if not attrs and hinfo: if hinfo.contents.IsDirectory: attrs |= FILE_ATTRIBUTE_DIRECTORY else: attrs |= FILE_ATTRIBUTE_NORMAL if not attrs: if self.fs.isdir(path): attrs |= FILE_ATTRIBUTE_DIRECTORY else: attrs |= FILE_ATTRIBUTE_NORMAL return attrs def _info2finddataw(self,path,info,data=None,hinfo=None): """Convert a file/directory info dict into a WIN32_FIND_DATAW struct.""" if data is None: data = libdokan.WIN32_FIND_DATAW() data.dwFileAttributes = self._info2attrmask(path,info,hinfo) data.ftCreationTime = _datetime2filetime(info.get("created_time",None)) data.ftLastAccessTime = _datetime2filetime(info.get("accessed_time",None)) data.ftLastWriteTime = _datetime2filetime(info.get("modified_time",None)) data.nFileSizeHigh = info.get("size",0) >> 32 data.nFileSizeLow = info.get("size",0) & 0xffffffff data.cFileName = basename(path) data.cAlternateFileName = "" return data def _datetime2timestamp(dtime): """Convert a datetime object to a unix timestamp.""" t = time.mktime(dtime.timetuple()) t += dtime.microsecond / 1000000.0 return t DATETIME_LOCAL_TO_UTC = _datetime2timestamp(datetime.datetime.utcnow()) - _datetime2timestamp(datetime.datetime.now()) def _timestamp2datetime(tstamp): """Convert a unix timestamp to a datetime object.""" return datetime.datetime.fromtimestamp(tstamp) def _timestamp2filetime(tstamp): f = FILETIME_UNIX_EPOCH + int(tstamp * 10000000) return libdokan.FILETIME(f & 0xffffffff,f >> 32) def _filetime2timestamp(ftime): f = ftime.dwLowDateTime | (ftime.dwHighDateTime << 32) return (f - FILETIME_UNIX_EPOCH) / 10000000.0 def _filetime2datetime(ftime): """Convert a FILETIME struct info datetime.datetime object.""" if ftime is None: return DATETIME_ZERO if ftime.dwLowDateTime == 0 and ftime.dwHighDateTime == 0: return DATETIME_ZERO return _timestamp2datetime(_filetime2timestamp(ftime)) def _datetime2filetime(dtime): """Convert a FILETIME struct info datetime.datetime object.""" if dtime is None: return libdokan.FILETIME(0,0) if dtime == DATETIME_ZERO: return libdokan.FILETIME(0,0) return _timestamp2filetime(_datetime2timestamp(dtime)) def _errno2syserrcode(eno): """Convert an errno into a win32 system error code.""" if eno == errno.EEXIST: return ERROR_FILE_EXISTS if eno == errno.ENOTEMPTY: return ERROR_DIR_NOT_EMPTY if eno == errno.ENOSYS: return ERROR_NOT_SUPPORTED if eno == errno.EACCES: return ERROR_ACCESS_DENIED return eno def _normalise_drive_string(drive): """Normalise drive string to a single letter.""" if not drive: raise ValueError("invalid drive letter: %r" % (drive,)) if len(drive) > 3: raise ValueError("invalid drive letter: %r" % (drive,)) if not drive[0].isalpha(): raise ValueError("invalid drive letter: %r" % (drive,)) if not ":\\".startswith(drive[1:]): raise ValueError("invalid drive letter: %r" % (drive,)) return drive[0].upper() def mount(fs, drive, foreground=False, ready_callback=None, unmount_callback=None, **kwds): """Mount the given FS at the given drive letter, using Dokan. By default, this function spawns a new background process to manage the Dokan event loop. The return value in this case is an instance of the 'MountProcess' class, a subprocess.Popen subclass. If the keyword argument 'foreground' is given, we instead run the Dokan main loop in the current process. In this case the function will block until the filesystem is unmounted, then return None. If the keyword argument 'ready_callback' is provided, it will be called when the filesystem has been mounted and is ready for use. Any additional keyword arguments control the behavior of the final dokan mount point. Some interesting options include: * numthreads: number of threads to use for handling Dokan requests * fsname: name to display in explorer etc * flags: DOKAN_OPTIONS bitmask * FSOperationsClass: custom FSOperations subclass to use """ if libdokan is None: raise OSError("the dokan library is not available") drive = _normalise_drive_string(drive) # This function captures the logic of checking whether the Dokan mount # is up and running. Unfortunately I can't find a way to get this # via a callback in the Dokan API. Instead we just check for the drive # in a loop, polling the mount proc to make sure it hasn't died. def check_alive(mp): if mp and mp.poll() != None: raise OSError("dokan mount process exited prematurely") def check_ready(mp=None): if ready_callback is not False: check_alive(mp) for _ in xrange(100): try: os.stat(drive+":\\") except EnvironmentError, e: check_alive(mp) time.sleep(0.05) else: check_alive(mp) if ready_callback: return ready_callback() else: return None else: check_alive(mp) raise OSError("dokan mount process seems to be hung") # Running the the foreground is the final endpoint for the mount # operation, it's where we call DokanMain(). if foreground: numthreads = kwds.pop("numthreads",0) flags = kwds.pop("flags",0) FSOperationsClass = kwds.pop("FSOperationsClass",FSOperations) opts = libdokan.DOKAN_OPTIONS(drive[:1], numthreads, flags) ops = FSOperationsClass(fs, **kwds) if ready_callback: check_thread = threading.Thread(target=check_ready) check_thread.daemon = True check_thread.start() opstruct = ops.get_ops_struct() res = libdokan.DokanMain(ctypes.byref(opts),ctypes.byref(opstruct)) if res != DOKAN_SUCCESS: raise OSError("Dokan failed with error: %d" % (res,)) if unmount_callback: unmount_callback() # Running the background, spawn a subprocess and wait for it # to be ready before returning. else: mp = MountProcess(fs, drive, kwds) check_ready(mp) if unmount_callback: orig_unmount = mp.unmount def new_unmount(): orig_unmount() unmount_callback() mp.unmount = new_unmount return mp def unmount(drive): """Unmount the given drive. This function unmounts the dokan drive mounted at the given drive letter. It works but may leave dangling processes; its better to use the "unmount" method on the MountProcess class if you have one. """ drive = _normalise_drive_string(drive) if not libdokan.DokanUnmount(drive): raise OSError("filesystem could not be unmounted: %s" % (drive,)) class MountProcess(subprocess.Popen): """subprocess.Popen subclass managing a Dokan mount. This is a subclass of subprocess.Popen, designed for easy management of a Dokan mount in a background process. Rather than specifying the command to execute, pass in the FS object to be mounted, the target drive letter and a dictionary of options for the Dokan process. In order to be passed successfully to the new process, the FS object must be pickleable. Since win32 has no fork() this restriction is not likely to be lifted (see also the "multiprocessing" module) This class has an extra attribute 'drive' giving the drive of the mounted filesystem, and an extra method 'unmount' that will cleanly unmount it and terminate the process. """ # This works by spawning a new python interpreter and passing it the # pickled (fs,path,opts) tuple on the command-line. Something like this: # # python -c "import MountProcess; MountProcess._do_mount('..data..') # unmount_timeout = 5 def __init__(self, fs, drive, dokan_opts={}, nowait=False, **kwds): if libdokan is None: raise OSError("the dokan library is not available") self.drive = _normalise_drive_string(drive) self.path = self.drive + ":\\" cmd = "import cPickle; " cmd = cmd + "data = cPickle.loads(%s); " cmd = cmd + "from fs.expose.dokan import MountProcess; " cmd = cmd + "MountProcess._do_mount(data)" cmd = cmd % (repr(cPickle.dumps((fs,drive,dokan_opts,nowait),-1)),) cmd = [sys.executable,"-c",cmd] super(MountProcess,self).__init__(cmd,**kwds) def unmount(self): """Cleanly unmount the Dokan filesystem, terminating this subprocess.""" if not libdokan.DokanUnmount(self.drive): raise OSError("the filesystem could not be unmounted: %s" %(self.drive,)) self.terminate() if not hasattr(subprocess.Popen, "terminate"): def terminate(self): """Gracefully terminate the subprocess.""" kernel32.TerminateProcess(int(self._handle),-1) if not hasattr(subprocess.Popen, "kill"): def kill(self): """Forcibly terminate the subprocess.""" kernel32.TerminateProcess(int(self._handle),-1) @staticmethod def _do_mount(data): """Perform the specified mount.""" (fs,drive,opts,nowait) = data opts["foreground"] = True def unmount_callback(): fs.close() opts["unmount_callback"] = unmount_callback if nowait: opts["ready_callback"] = False mount(fs,drive,**opts) class Win32SafetyFS(WrapFS): """FS wrapper for extra safety when mounting on win32. This wrapper class provides some safety features when mounting untrusted filesystems on win32. Specifically: * hiding autorun files * removing colons from paths """ def __init__(self,wrapped_fs,allow_autorun=False): self.allow_autorun = allow_autorun super(Win32SafetyFS,self).__init__(wrapped_fs) def _encode(self,path): path = relpath(normpath(path)) path = path.replace(":","__colon__") if not self.allow_autorun: if path.lower().startswith("_autorun."): path = path[1:] return path def _decode(self,path): path = relpath(normpath(path)) path = path.replace("__colon__",":") if not self.allow_autorun: if path.lower().startswith("autorun."): path = "_" + path return path if __name__ == "__main__": import os, os.path import tempfile from fs.osfs import OSFS from fs.memoryfs import MemoryFS from shutil import rmtree from six import b path = tempfile.mkdtemp() try: fs = OSFS(path) #fs = MemoryFS() fs.setcontents("test1.txt",b("test one")) flags = DOKAN_OPTION_DEBUG|DOKAN_OPTION_STDERR|DOKAN_OPTION_REMOVABLE mount(fs, "Q", foreground=True, numthreads=1, flags=flags) fs.close() finally: rmtree(path) fs-0.5.4/fs/expose/ftp.py0000664000175000017500000002146012512525115015152 0ustar willwill00000000000000""" fs.expose.ftp ============== Expose an FS object over FTP (via pyftpdlib). This module provides the necessary interfaces to expose an FS object over FTP, plugging into the infrastructure provided by the 'pyftpdlib' module. To use this in combination with fsserve, do the following: $ fsserve -t 'ftp' $HOME The above will serve your home directory in read-only mode via anonymous FTP on the loopback address. """ import os import stat import time import errno from functools import wraps from pyftpdlib import ftpserver from fs.path import * from fs.osfs import OSFS from fs.errors import convert_fs_errors from fs import iotools from six import text_type as unicode # Get these once so we can reuse them: UID = os.getuid() GID = os.getgid() def decode_args(f): """ Decodes string arguments using the decoding defined on the method's class. This decorator is for use on methods (functions which take a class or instance as the first parameter). Pyftpdlib (as of 0.7.0) uses str internally, so this decoding is necessary. """ @wraps(f) def wrapper(self, *args): encoded = [] for arg in args: if isinstance(arg, str): arg = arg.decode(self.encoding) encoded.append(arg) return f(self, *encoded) return wrapper class FakeStat(object): """ Pyftpdlib uses stat inside the library. This class emulates the standard os.stat_result class to make pyftpdlib happy. Think of it as a stat-like object ;-). """ def __init__(self, **kwargs): for attr in dir(stat): if not attr.startswith('ST_'): continue attr = attr.lower() value = kwargs.get(attr, 0) setattr(self, attr, value) class FTPFS(ftpserver.AbstractedFS): """ The basic FTP Filesystem. This is a bridge between a pyfs filesystem and pyftpdlib's AbstractedFS. This class will cause the FTP server to serve the given fs instance. """ encoding = 'utf8' "Sets the encoding to use for paths." def __init__(self, fs, root, cmd_channel, encoding=None): self.fs = fs if encoding is not None: self.encoding = encoding super(FTPFS, self).__init__(root, cmd_channel) def close(self): # Close and dereference the pyfs file system. if self.fs: self.fs.close() self.fs = None def validpath(self, path): try: normpath(path) return True except: return False @convert_fs_errors @decode_args @iotools.filelike_to_stream def open(self, path, mode, **kwargs): return self.fs.open(path, mode, **kwargs) @convert_fs_errors def chdir(self, path): # We dont' use the decorator here, we actually decode a version of the # path for use with pyfs, but keep the original for use with pyftpdlib. if not isinstance(path, unicode): # pyftpdlib 0.7.x unipath = unicode(path, self.encoding) else: # pyftpdlib 1.x unipath = path # TODO: can the following conditional checks be farmed out to the fs? # If we don't raise an error here for files, then the FTP server will # happily allow the client to CWD into a file. We really only want to # allow that for directories. if self.fs.isfile(unipath): raise OSError(errno.ENOTDIR, 'Not a directory') # similarly, if we don't check for existence, the FTP server will allow # the client to CWD into a non-existent directory. if not self.fs.exists(unipath): raise OSError(errno.ENOENT, 'Does not exist') # We use the original path here, so we don't corrupt self._cwd self._cwd = self.ftp2fs(path) @convert_fs_errors @decode_args def mkdir(self, path): self.fs.makedir(path) @convert_fs_errors @decode_args def listdir(self, path): return map(lambda x: x.encode(self.encoding), self.fs.listdir(path)) @convert_fs_errors @decode_args def rmdir(self, path): self.fs.removedir(path) @convert_fs_errors @decode_args def remove(self, path): self.fs.remove(path) @convert_fs_errors @decode_args def rename(self, src, dst): self.fs.rename(src, dst) @convert_fs_errors @decode_args def chmod(self, path, mode): return @convert_fs_errors @decode_args def stat(self, path): info = self.fs.getinfo(path) kwargs = { 'st_size': info.get('size'), } # Give the fs a chance to provide the uid/gid. Otherwise echo the current # uid/gid. kwargs['st_uid'] = info.get('st_uid', UID) kwargs['st_gid'] = info.get('st_gid', GID) if 'st_atime' in info: kwargs['st_atime'] = info['st_atime'] elif 'accessed_time' in info: kwargs['st_atime'] = time.mktime(info["accessed_time"].timetuple()) if 'st_mtime' in info: kwargs['st_mtime'] = info.get('st_mtime') elif 'modified_time' in info: kwargs['st_mtime'] = time.mktime(info["modified_time"].timetuple()) # Pyftpdlib uses st_ctime on Windows platform, try to provide it. if 'st_ctime' in info: kwargs['st_ctime'] = info['st_ctime'] elif 'created_time' in info: kwargs['st_ctime'] = time.mktime(info["created_time"].timetuple()) elif 'st_mtime' in kwargs: # As a last resort, just copy the modified time. kwargs['st_ctime'] = kwargs['st_mtime'] # Try to use existing mode. if 'st_mode' in info: kwargs['st_mode'] = info['st_mode'] elif 'mode' in info: kwargs['st_mode'] = info['mode'] else: # Otherwise, build one. Not executable by default. mode = 0660 # Merge in the type (dir or file). File is tested first, some file systems # such as ArchiveMountFS treat archive files as directories too. By checking # file first, any such files will be only files (not directories). if self.fs.isfile(path): mode |= stat.S_IFREG elif self.fs.isdir(path): mode |= stat.S_IFDIR mode |= 0110 # Merge in exec bit to signal dir is listable kwargs['st_mode'] = mode return FakeStat(**kwargs) # No link support... lstat = stat @convert_fs_errors @decode_args def isfile(self, path): return self.fs.isfile(path) @convert_fs_errors @decode_args def isdir(self, path): return self.fs.isdir(path) @convert_fs_errors @decode_args def getsize(self, path): return self.fs.getsize(path) @convert_fs_errors @decode_args def getmtime(self, path): return self.stat(path).st_mtime def realpath(self, path): return path def lexists(self, path): return True class FTPFSHandler(ftpserver.FTPHandler): """ An FTPHandler class that closes the filesystem when done. """ def close(self): # Close the FTPFS instance, it will close the pyfs file system. if self.fs: self.fs.close() super(FTPFSHandler, self).close() class FTPFSFactory(object): """ A factory class which can hold a reference to a file system object and encoding, then later pass it along to an FTPFS instance. An instance of this object allows multiple FTPFS instances to be created by pyftpdlib while sharing the same fs. """ def __init__(self, fs, encoding=None): """ Initializes the factory with an fs instance. """ self.fs = fs self.encoding = encoding def __call__(self, root, cmd_channel): """ This is the entry point of pyftpdlib. We will pass along the two parameters as well as the previously provided fs instance and encoding. """ return FTPFS(self.fs, root, cmd_channel, encoding=self.encoding) class HomeFTPFS(FTPFS): """ A file system which serves a user's home directory. """ def __init__(self, root, cmd_channel): """ Use the provided user's home directory to create an FTPFS that serves an OSFS rooted at the home directory. """ super(DemoFS, self).__init__(OSFS(root_path=root), '/', cmd_channel) def serve_fs(fs, addr, port): """ Creates a basic anonymous FTP server serving the given FS on the given address/port combo. """ from pyftpdlib.contrib.authorizers import UnixAuthorizer ftp_handler = FTPFSHandler ftp_handler.authorizer = ftpserver.DummyAuthorizer() ftp_handler.authorizer.add_anonymous('/') ftp_handler.abstracted_fs = FTPFSFactory(fs) s = ftpserver.FTPServer((addr, port), ftp_handler) s.serve_forever() fs-0.5.4/fs/expose/xmlrpc.py0000664000175000017500000001607312512525115015672 0ustar willwill00000000000000""" fs.expose.xmlrpc ================ Server to expose an FS via XML-RPC This module provides the necessary infrastructure to expose an FS object over XML-RPC. The main class is 'RPCFSServer', a SimpleXMLRPCServer subclass designed to expose an underlying FS. If you need to use a more powerful server than SimpleXMLRPCServer, you can use the RPCFSInterface class to provide an XML-RPC-compatible wrapper around an FS object, which can then be exposed using whatever server you choose (e.g. Twisted's XML-RPC server). """ import xmlrpclib from SimpleXMLRPCServer import SimpleXMLRPCServer from datetime import datetime import base64 import six from six import PY3 class RPCFSInterface(object): """Wrapper to expose an FS via a XML-RPC compatible interface. The only real trick is using xmlrpclib.Binary objects to transport the contents of files. """ # info keys are restricted to a subset known to work over xmlrpc # This fixes an issue with transporting Longs on Py3 _allowed_info = ["size", "created_time", "modified_time", "accessed_time", "st_size", "st_mode", "type"] def __init__(self, fs): super(RPCFSInterface, self).__init__() self.fs = fs def encode_path(self, path): """Encode a filesystem path for sending over the wire. Unfortunately XMLRPC only supports ASCII strings, so this method must return something that can be represented in ASCII. The default is base64-encoded UTF-8. """ #return path return six.text_type(base64.b64encode(path.encode("utf8")), 'ascii') def decode_path(self, path): """Decode paths arriving over the wire.""" return six.text_type(base64.b64decode(path.encode('ascii')), 'utf8') def getmeta(self, meta_name): meta = self.fs.getmeta(meta_name) if isinstance(meta, basestring): meta = self.decode_path(meta) return meta def getmeta_default(self, meta_name, default): meta = self.fs.getmeta(meta_name, default) if isinstance(meta, basestring): meta = self.decode_path(meta) return meta def hasmeta(self, meta_name): return self.fs.hasmeta(meta_name) def get_contents(self, path, mode="rb"): path = self.decode_path(path) data = self.fs.getcontents(path, mode) return xmlrpclib.Binary(data) def set_contents(self, path, data): path = self.decode_path(path) self.fs.setcontents(path, data.data) def exists(self, path): path = self.decode_path(path) return self.fs.exists(path) def isdir(self, path): path = self.decode_path(path) return self.fs.isdir(path) def isfile(self, path): path = self.decode_path(path) return self.fs.isfile(path) def listdir(self, path="./", wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False): path = self.decode_path(path) entries = self.fs.listdir(path, wildcard, full, absolute, dirs_only, files_only) return [self.encode_path(e) for e in entries] def makedir(self, path, recursive=False, allow_recreate=False): path = self.decode_path(path) return self.fs.makedir(path, recursive, allow_recreate) def remove(self, path): path = self.decode_path(path) return self.fs.remove(path) def removedir(self, path, recursive=False, force=False): path = self.decode_path(path) return self.fs.removedir(path, recursive, force) def rename(self, src, dst): src = self.decode_path(src) dst = self.decode_path(dst) return self.fs.rename(src, dst) def settimes(self, path, accessed_time, modified_time): path = self.decode_path(path) if isinstance(accessed_time, xmlrpclib.DateTime): accessed_time = datetime.strptime(accessed_time.value, "%Y%m%dT%H:%M:%S") if isinstance(modified_time, xmlrpclib.DateTime): modified_time = datetime.strptime(modified_time.value, "%Y%m%dT%H:%M:%S") return self.fs.settimes(path, accessed_time, modified_time) def getinfo(self, path): path = self.decode_path(path) info = self.fs.getinfo(path) info = dict((k, v) for k, v in info.iteritems() if k in self._allowed_info) return info def desc(self, path): path = self.decode_path(path) return self.fs.desc(path) def getxattr(self, path, attr, default=None): path = self.decode_path(path) attr = self.decode_path(attr) return self.fs.getxattr(path, attr, default) def setxattr(self, path, attr, value): path = self.decode_path(path) attr = self.decode_path(attr) return self.fs.setxattr(path, attr, value) def delxattr(self, path, attr): path = self.decode_path(path) attr = self.decode_path(attr) return self.fs.delxattr(path, attr) def listxattrs(self, path): path = self.decode_path(path) return [self.encode_path(a) for a in self.fs.listxattrs(path)] def copy(self, src, dst, overwrite=False, chunk_size=16384): src = self.decode_path(src) dst = self.decode_path(dst) return self.fs.copy(src, dst, overwrite, chunk_size) def move(self, src, dst, overwrite=False, chunk_size=16384): src = self.decode_path(src) dst = self.decode_path(dst) return self.fs.move(src, dst, overwrite, chunk_size) def movedir(self, src, dst, overwrite=False, ignore_errors=False, chunk_size=16384): src = self.decode_path(src) dst = self.decode_path(dst) return self.fs.movedir(src, dst, overwrite, ignore_errors, chunk_size) def copydir(self, src, dst, overwrite=False, ignore_errors=False, chunk_size=16384): src = self.decode_path(src) dst = self.decode_path(dst) return self.fs.copydir(src, dst, overwrite, ignore_errors, chunk_size) class RPCFSServer(SimpleXMLRPCServer): """Server to expose an FS object via XML-RPC. This class takes as its first argument an FS instance, and as its second argument a (hostname,port) tuple on which to listen for XML-RPC requests. Example:: fs = OSFS('/var/srv/myfiles') s = RPCFSServer(fs,("",8080)) s.serve_forever() To cleanly shut down the server after calling serve_forever, set the attribute "serve_more_requests" to False. """ def __init__(self, fs, addr, requestHandler=None, logRequests=None): kwds = dict(allow_none=True) if requestHandler is not None: kwds['requestHandler'] = requestHandler if logRequests is not None: kwds['logRequests'] = logRequests self.serve_more_requests = True SimpleXMLRPCServer.__init__(self, addr, **kwds) self.register_instance(RPCFSInterface(fs)) def serve_forever(self): """Override serve_forever to allow graceful shutdown.""" while self.serve_more_requests: self.handle_request() fs-0.5.4/fs/expose/http.py0000664000175000017500000001252712512525115015344 0ustar willwill00000000000000__all__ = ["serve_fs"] import SimpleHTTPServer import SocketServer from fs.path import pathjoin, dirname from fs.errors import FSError from time import mktime from cStringIO import StringIO import cgi import urllib import posixpath import time import threading import socket def _datetime_to_epoch(d): return mktime(d.timetuple()) class FSHTTPRequestHandler(SimpleHTTPServer.SimpleHTTPRequestHandler): """A hacked together version of SimpleHTTPRequestHandler""" def __init__(self, fs, request, client_address, server): self._fs = fs SimpleHTTPServer.SimpleHTTPRequestHandler.__init__(self, request, client_address, server) def do_GET(self): """Serve a GET request.""" f = None try: f = self.send_head() if f: try: self.copyfile(f, self.wfile) except socket.error: pass finally: if f is not None: f.close() def send_head(self): """Common code for GET and HEAD commands. This sends the response code and MIME headers. Return value is either a file object (which has to be copied to the outputfile by the caller unless the command was HEAD, and must be closed by the caller under all circumstances), or None, in which case the caller has nothing further to do. """ path = self.translate_path(self.path) f = None if self._fs.isdir(path): if not self.path.endswith('/'): # redirect browser - doing basically what apache does self.send_response(301) self.send_header("Location", self.path + "/") self.end_headers() return None for index in ("index.html", "index.htm"): index = pathjoin(path, index) if self._fs.exists(index): path = index break else: return self.list_directory(path) ctype = self.guess_type(path) try: info = self._fs.getinfo(path) f = self._fs.open(path, 'rb') except FSError, e: self.send_error(404, str(e)) return None self.send_response(200) self.send_header("Content-type", ctype) self.send_header("Content-Length", str(info['size'])) if 'modified_time' in info: self.send_header("Last-Modified", self.date_time_string(_datetime_to_epoch(info['modified_time']))) self.end_headers() return f def list_directory(self, path): """Helper to produce a directory listing (absent index.html). Return value is either a file object, or None (indicating an error). In either case, the headers are sent, making the interface the same as for send_head(). """ try: dir_paths = self._fs.listdir(path, dirs_only=True) file_paths = self._fs.listdir(path, files_only=True) except FSError: self.send_error(404, "No permission to list directory") return None paths = [p+'/' for p in sorted(dir_paths, key=lambda p:p.lower())] + sorted(file_paths, key=lambda p:p.lower()) #list.sort(key=lambda a: a.lower()) f = StringIO() displaypath = cgi.escape(urllib.unquote(self.path)) f.write('') f.write("\nDirectory listing for %s\n" % displaypath) f.write("\n

Directory listing for %s

\n" % displaypath) f.write("
\n
    \n") parent = dirname(path) if path != parent: f.write('
  • ../
    • ' % urllib.quote(parent.rstrip('/') + '/')) for path in paths: f.write('
  • %s\n' % (urllib.quote(path), cgi.escape(path))) f.write("
\n
\n\n\n") length = f.tell() f.seek(0) self.send_response(200) self.send_header("Content-type", "text/html") self.send_header("Content-Length", str(length)) self.end_headers() return f def translate_path(self, path): # abandon query parameters path = path.split('?',1)[0] path = path.split('#',1)[0] path = posixpath.normpath(urllib.unquote(path)) return path def serve_fs(fs, address='', port=8000): """Serve an FS instance over http :param fs: an FS object :param address: IP address to serve on :param port: port number """ def Handler(request, client_address, server): return FSHTTPRequestHandler(fs, request, client_address, server) #class ThreadedTCPServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer): # pass httpd = SocketServer.TCPServer((address, port), Handler, bind_and_activate=False) #httpd = ThreadedTCPServer((address, port), Handler, bind_and_activate=False) httpd.allow_reuse_address = True httpd.server_bind() httpd.server_activate() server_thread = threading.Thread(target=httpd.serve_forever) server_thread.start() try: while True: time.sleep(0.1) except (KeyboardInterrupt, SystemExit): httpd.shutdown() if __name__ == "__main__": from fs.osfs import OSFS serve_fs(OSFS('~/'))fs-0.5.4/fs/expose/importhook.py0000664000175000017500000002003312512525115016547 0ustar willwill00000000000000""" fs.expose.importhook ==================== Expose an FS object to the python import machinery, via a PEP-302 loader. This module allows you to import python modules from an arbitrary FS object, by placing FS urls on sys.path and/or inserting objects into sys.meta_path. The main class in this module is FSImportHook, which is a PEP-302-compliant module finder and loader. If you place an instance of FSImportHook on sys.meta_path, you will be able to import modules from the exposed filesystem:: >>> from fs.memoryfs import MemoryFS >>> m = MemoryFS() >>> m.setcontents("helloworld.py","print 'hello world!'") >>> >>> import sys >>> from fs.expose.importhook import FSImportHook >>> sys.meta_path.append(FSImportHook(m)) >>> import helloworld hello world! It is also possible to install FSImportHook as an import path handler. This allows you to place filesystem URLs on sys.path and have them automagically opened for importing. This example would allow modules to be imported from an SFTP server:: >>> from fs.expose.importhook import FSImportHook >>> FSImportHook.install() >>> sys.path.append("sftp://some.remote.machine/mypath/") """ import os import sys import imp import marshal from fs.base import FS from fs.opener import fsopendir, OpenerError from fs.errors import * from fs.path import * from six import b class FSImportHook(object): """PEP-302-compliant module finder and loader for FS objects. FSImportHook is a module finder and loader that takes its data from an arbitrary FS object. The FS must have .py or .pyc files stored in the standard module structure. For easy use with sys.path, FSImportHook will also accept a filesystem URL, which is automatically opened using fs.opener. """ _VALID_MODULE_TYPES = set((imp.PY_SOURCE,imp.PY_COMPILED)) def __init__(self,fs_or_url): # If given a string, try to open it as an FS url. # Don't open things on the local filesystem though. if isinstance(fs_or_url,basestring): if ":/" not in fs_or_url: raise ImportError try: self.fs = fsopendir(fs_or_url) except OpenerError: raise ImportError except (CreateFailedError,ResourceNotFoundError,): raise ImportError self.path = fs_or_url # Otherwise, it must be an FS object of some sort. else: if not isinstance(fs_or_url,FS): raise ImportError self.fs = fs_or_url self.path = None @classmethod def install(cls): """Install this class into the import machinery. This classmethod installs the custom FSImportHook class into the import machinery of the running process, if it is not already installed. """ for imp in enumerate(sys.path_hooks): try: if issubclass(cls,imp): break except TypeError: pass else: sys.path_hooks.append(cls) sys.path_importer_cache.clear() @classmethod def uninstall(cls): """Uninstall this class from the import machinery. This classmethod uninstalls the custom FSImportHook class from the import machinery of the running process. """ to_rem = [] for i,imp in enumerate(sys.path_hooks): try: if issubclass(cls,imp): to_rem.append(imp) break except TypeError: pass for imp in to_rem: sys.path_hooks.remove(imp) sys.path_importer_cache.clear() def find_module(self,fullname,path=None): """Find the FS loader for the given module. This object is always its own loader, so this really just checks whether it's a valid module on the exposed filesystem. """ try: self._get_module_info(fullname) except ImportError: return None else: return self def _get_module_info(self,fullname): """Get basic information about the given module. If the specified module exists, this method returns a tuple giving its filepath, file type and whether it's a package. Otherwise, it raise ImportError. """ prefix = fullname.replace(".","/") # Is it a regular module? (path,type) = self._find_module_file(prefix) if path is not None: return (path,type,False) # Is it a package? prefix = pathjoin(prefix,"__init__") (path,type) = self._find_module_file(prefix) if path is not None: return (path,type,True) # No, it's nothing raise ImportError(fullname) def _find_module_file(self,prefix): """Find a module file from the given path prefix. This method iterates over the possible module suffixes, checking each in turn and returning the first match found. It returns a two-tuple (path,type) or (None,None) if there's no module. """ for (suffix,mode,type) in imp.get_suffixes(): if type in self._VALID_MODULE_TYPES: path = prefix + suffix if self.fs.isfile(path): return (path,type) return (None,None) def load_module(self,fullname): """Load the specified module. This method locates the file for the specified module, loads and executes it and returns the created module object. """ # Reuse an existing module if present. try: return sys.modules[fullname] except KeyError: pass # Try to create from source or bytecode. info = self._get_module_info(fullname) code = self.get_code(fullname,info) if code is None: raise ImportError(fullname) mod = imp.new_module(fullname) mod.__file__ = "" mod.__loader__ = self sys.modules[fullname] = mod try: exec code in mod.__dict__ mod.__file__ = self.get_filename(fullname,info) if self.is_package(fullname,info): if self.path is None: mod.__path__ = [] else: mod.__path__ = [self.path] return mod except Exception: sys.modules.pop(fullname,None) raise def is_package(self,fullname,info=None): """Check whether the specified module is a package.""" if info is None: info = self._get_module_info(fullname) (path,type,ispkg) = info return ispkg def get_code(self,fullname,info=None): """Get the bytecode for the specified module.""" if info is None: info = self._get_module_info(fullname) (path,type,ispkg) = info code = self.fs.getcontents(path, 'rb') if type == imp.PY_SOURCE: code = code.replace(b("\r\n"),b("\n")) return compile(code,path,"exec") elif type == imp.PY_COMPILED: if code[:4] != imp.get_magic(): return None return marshal.loads(code[8:]) else: return None return code def get_source(self,fullname,info=None): """Get the sourcecode for the specified module, if present.""" if info is None: info = self._get_module_info(fullname) (path,type,ispkg) = info if type != imp.PY_SOURCE: return None return self.fs.getcontents(path, 'rb').replace(b("\r\n"),b("\n")) def get_data(self,path): """Read the specified data file.""" try: return self.fs.getcontents(path, 'rb') except FSError, e: raise IOError(str(e)) def get_filename(self,fullname,info=None): """Get the __file__ attribute for the specified module.""" if info is None: info = self._get_module_info(fullname) (path,type,ispkg) = info return path fs-0.5.4/fs/expose/__init__.py0000664000175000017500000000000012512525115016103 0ustar willwill00000000000000fs-0.5.4/fs/httpfs.py0000664000175000017500000000474212512525115014372 0ustar willwill00000000000000""" fs.httpfs ========= """ from fs.base import FS from fs.path import normpath from fs.errors import ResourceNotFoundError, UnsupportedError from fs.filelike import FileWrapper from fs import iotools from urllib2 import urlopen, URLError from datetime import datetime class HTTPFS(FS): """Can barely be called a filesystem, because HTTP servers generally don't support typical filesystem functionality. This class exists to allow the :doc:`opener` system to read files over HTTP. If you do need filesystem like functionality over HTTP, see :mod:`fs.contrib.davfs`. """ _meta = {'read_only': True, 'network': True} def __init__(self, url): """ :param url: The base URL """ self.root_url = url def _make_url(self, path): path = normpath(path) url = '%s/%s' % (self.root_url.rstrip('/'), path.lstrip('/')) return url @iotools.filelike_to_stream def open(self, path, mode='r', buffering=-1, encoding=None, errors=None, newline=None, line_buffering=False, **kwargs): if '+' in mode or 'w' in mode or 'a' in mode: raise UnsupportedError('write') url = self._make_url(path) try: f = urlopen(url) except URLError, e: raise ResourceNotFoundError(path, details=e) except OSError, e: raise ResourceNotFoundError(path, details=e) return FileWrapper(f) def exists(self, path): return self.isfile(path) def isdir(self, path): return False def isfile(self, path): url = self._make_url(path) f = None try: try: f = urlopen(url) except (URLError, OSError): return False finally: if f is not None: f.close() return True def listdir(self, path="./", wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False): return [] def getinfo(self, path): url = self._make_url(path) info = urlopen(url).info().dict if 'content-length' in info: info['size'] = info['content-length'] if 'last-modified' in info: info['modified_time'] = datetime.strptime(info['last-modified'], "%a, %d %b %Y %H:%M:%S %Z") return info fs-0.5.4/fs/iotools.py0000664000175000017500000001654112573252057014563 0ustar willwill00000000000000from __future__ import unicode_literals from __future__ import print_function from fs import SEEK_SET, SEEK_CUR, SEEK_END import io from functools import wraps import six class RawWrapper(object): """Convert a Python 2 style file-like object in to a IO object""" def __init__(self, f, mode=None, name=None): self._f = f self.is_io = isinstance(f, io.IOBase) if mode is None and hasattr(f, 'mode'): mode = f.mode self.mode = mode self.name = name self.closed = False super(RawWrapper, self).__init__() def __repr__(self): return "".format(self._f) def close(self): self._f.close() self.closed = True def fileno(self): return self._f.fileno() def flush(self): return self._f.flush() def isatty(self): return self._f.isatty() def seek(self, offset, whence=SEEK_SET): return self._f.seek(offset, whence) def readable(self): if hasattr(self._f, 'readable'): return self._f.readable() return 'r' in self.mode def writable(self): if hasattr(self._f, 'writeable'): return self._fs.writeable() return 'w' in self.mode def seekable(self): if hasattr(self._f, 'seekable'): return self._f.seekable() try: self.seek(0, SEEK_CUR) except IOError: return False else: return True def tell(self): return self._f.tell() def truncate(self, size=None): return self._f.truncate(size) def write(self, data): if self.is_io: return self._f.write(data) self._f.write(data) return len(data) def read(self, n=-1): if n == -1: return self.readall() return self._f.read(n) def read1(self, n=-1): if self.is_io: return self._f.read1(n) return self.read(n) def readall(self): return self._f.read() def readinto(self, b): if self.is_io: return self._f.readinto(b) data = self._f.read(len(b)) bytes_read = len(data) b[:len(data)] = data return bytes_read def readline(self, limit=-1): return self._f.readline(limit) def readlines(self, hint=-1): return self._f.readlines(hint) def writelines(self, sequence): return self._f.writelines(sequence) def __enter__(self): return self def __exit__(self, *args, **kwargs): self.close() def __iter__(self): return iter(self._f) def filelike_to_stream(f): @wraps(f) def wrapper(self, path, mode='rt', buffering=-1, encoding=None, errors=None, newline=None, line_buffering=False, **kwargs): file_like = f(self, path, mode=mode, buffering=buffering, encoding=encoding, errors=errors, newline=newline, line_buffering=line_buffering, **kwargs) return make_stream(path, file_like, mode=mode, buffering=buffering, encoding=encoding, errors=errors, newline=newline, line_buffering=line_buffering) return wrapper def make_stream(name, f, mode='r', buffering=-1, encoding=None, errors=None, newline=None, line_buffering=False, **kwargs): """Take a Python 2.x binary file and returns an IO Stream""" r, w, a, binary = 'r' in mode, 'w' in mode, 'a' in mode, 'b' in mode if '+' in mode: r, w = True, True io_object = RawWrapper(f, mode=mode, name=name) if buffering >= 0: if r and w: io_object = io.BufferedRandom(io_object, buffering or io.DEFAULT_BUFFER_SIZE) elif r: io_object = io.BufferedReader(io_object, buffering or io.DEFAULT_BUFFER_SIZE) elif w: io_object = io.BufferedWriter(io_object, buffering or io.DEFAULT_BUFFER_SIZE) if not binary: io_object = io.TextIOWrapper(io_object, encoding=encoding or 'utf-8', errors=errors, newline=newline, line_buffering=line_buffering,) return io_object def decode_binary(data, encoding=None, errors=None, newline=None): """Decode bytes as though read from a text file""" return io.TextIOWrapper(io.BytesIO(data), encoding=encoding or 'utf-8', errors=errors, newline=newline).read() def make_bytes_io(data, encoding=None, errors=None): """Make a bytes IO object from either a string or an open file""" if hasattr(data, 'mode') and 'b' in data.mode: # It's already a binary file return data if not isinstance(data, basestring): # It's a file, but we don't know if its binary # TODO: Is there a better way than reading the entire file? data = data.read() or b'' if isinstance(data, six.text_type): # If its text, encoding in to bytes data = data.encode(encoding=encoding, errors=errors) return io.BytesIO(data) def copy_file_to_fs(f, fs, path, encoding=None, errors=None, progress_callback=None, chunk_size=64 * 1024): """Copy an open file to a path on an FS""" if progress_callback is None: progress_callback = lambda bytes_written: None read = f.read chunk = read(chunk_size) if isinstance(chunk, six.text_type): f = fs.open(path, 'wt', encoding=encoding, errors=errors) else: f = fs.open(path, 'wb') write = f.write bytes_written = 0 try: while chunk: write(chunk) bytes_written += len(chunk) progress_callback(bytes_written) chunk = read(chunk_size) finally: f.close() return bytes_written def line_iterator(f, size=None): """A not terribly efficient char by char line iterator""" read = f.read line = [] append = line.append c = 1 if size is None or size < 0: while c: c = read(1) if c: append(c) if c in (b'\n', b''): yield b''.join(line) del line[:] else: while c: c = read(1) if c: append(c) if c in (b'\n', b'') or len(line) >= size: yield b''.join(line) del line[:] if __name__ == "__main__": print("Reading a binary file") bin_file = open('tests/data/UTF-8-demo.txt', 'rb') with make_stream('UTF-8-demo.txt', bin_file, 'rb') as f: print(repr(f)) print(type(f.read(200))) print("Reading a text file") bin_file = open('tests/data/UTF-8-demo.txt', 'rb') with make_stream('UTF-8-demo.txt', bin_file, 'rt') as f: print(repr(f)) print(type(f.read(200))) print("Reading a buffered binary file") bin_file = open('tests/data/UTF-8-demo.txt', 'rb') with make_stream('UTF-8-demo.txt', bin_file, 'rb', buffering=0) as f: print(repr(f)) print(type(f.read(200))) fs-0.5.4/fs/mountfs.py0000664000175000017500000004415612512525115014560 0ustar willwill00000000000000""" fs.mountfs ========== Contains MountFS class which is a virtual filesystem which can have other filesystems linked as branched directories. For example, lets say we have two filesystems containing config files and resources respectively:: [config_fs] |-- config.cfg `-- defaults.cfg [resources_fs] |-- images | |-- logo.jpg | `-- photo.jpg `-- data.dat We can combine these filesystems in to a single filesystem with the following code:: from fs.mountfs import MountFS combined_fs = MountFS() combined_fs.mountdir('config', config_fs) combined_fs.mountdir('resources', resources_fs) This will create a single filesystem where paths under `config` map to `config_fs`, and paths under `resources` map to `resources_fs`:: [combined_fs] |-- config | |-- config.cfg | `-- defaults.cfg `-- resources |-- images | |-- logo.jpg | `-- photo.jpg `-- data.dat Now both filesystems can be accessed with the same path structure:: print combined_fs.getcontents('/config/defaults.cfg') read_jpg(combined_fs.open('/resources/images/logo.jpg') """ from fs.base import * from fs.errors import * from fs.path import * from fs import _thread_synchronize_default from fs import iotools class DirMount(object): def __init__(self, path, fs): self.path = path self.fs = fs def __str__(self): return "" % (self.path, self.fs) def __repr__(self): return "" % (self.path, self.fs) def __unicode__(self): return u"" % (self.path, self.fs) class FileMount(object): def __init__(self, path, open_callable, info_callable=None): self.open_callable = open_callable def no_info_callable(path): return {} self.info_callable = info_callable or no_info_callable class MountFS(FS): """A filesystem that delegates to other filesystems.""" _meta = { 'virtual': True, 'read_only' : False, 'unicode_paths' : True, 'case_insensitive_paths' : False, } DirMount = DirMount FileMount = FileMount def __init__(self, auto_close=True, thread_synchronize=_thread_synchronize_default): self.auto_close = auto_close super(MountFS, self).__init__(thread_synchronize=thread_synchronize) self.mount_tree = PathMap() def __str__(self): return "<%s [%s]>" % (self.__class__.__name__,self.mount_tree.items(),) __repr__ = __str__ def __unicode__(self): return u"<%s [%s]>" % (self.__class__.__name__,self.mount_tree.items(),) def _delegate(self, path): path = abspath(normpath(path)) object = None head_path = "/" tail_path = path for prefix in recursepath(path): try: object = self.mount_tree[prefix] except KeyError: pass else: head_path = prefix tail_path = path[len(head_path):] if type(object) is MountFS.DirMount: return object.fs, head_path, tail_path if type(object) is MountFS.FileMount: return self, "/", path try: self.mount_tree.iternames(path).next() except StopIteration: return None, None, None else: return self, "/", path @synchronize def close(self): # Explicitly closes children if requested if self.auto_close: for mount in self.mount_tree.itervalues(): mount.fs.close() # Free references (which may incidently call the close method of the child filesystems) self.mount_tree.clear() super(MountFS, self).close() def getsyspath(self, path, allow_none=False): fs, _mount_path, delegate_path = self._delegate(path) if fs is self or fs is None: if allow_none: return None else: raise NoSysPathError(path=path) return fs.getsyspath(delegate_path, allow_none=allow_none) def getpathurl(self, path, allow_none=False): fs, _mount_path, delegate_path = self._delegate(path) if fs is self or fs is None: if allow_none: return None else: raise NoPathURLError(path=path) return fs.getpathurl(delegate_path, allow_none=allow_none) @synchronize def desc(self, path): fs, _mount_path, delegate_path = self._delegate(path) if fs is self: if fs.isdir(path): return "Mount dir" else: return "Mounted file" return "Mounted dir, maps to path %s on %s" % (abspath(delegate_path) or '/', str(fs)) @synchronize def isdir(self, path): fs, _mount_path, delegate_path = self._delegate(path) if fs is None: path = normpath(path) if path in ("/", ""): return True return False if fs is self: obj = self.mount_tree.get(path, None) return not isinstance(obj, MountFS.FileMount) return fs.isdir(delegate_path) @synchronize def isfile(self, path): fs, _mount_path, delegate_path = self._delegate(path) if fs is None: return False if fs is self: obj = self.mount_tree.get(path, None) return isinstance(obj, MountFS.FileMount) return fs.isfile(delegate_path) @synchronize def exists(self, path): if path in ("/", ""): return True fs, _mount_path, delegate_path = self._delegate(path) if fs is None: return False if fs is self: return True return fs.exists(delegate_path) @synchronize def listdir(self, path="/", wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False): fs, _mount_path, delegate_path = self._delegate(path) if fs is None: if path in ("/", ""): return [] raise ResourceNotFoundError("path") elif fs is self: paths = self.mount_tree.names(path) return self._listdir_helper(path, paths, wildcard, full, absolute, dirs_only, files_only) else: paths = fs.listdir(delegate_path, wildcard=wildcard, full=False, absolute=False, dirs_only=dirs_only, files_only=files_only) for nm in self.mount_tree.names(path): if nm not in paths: if dirs_only: if self.isdir(pathjoin(path,nm)): paths.append(nm) elif files_only: if self.isfile(pathjoin(path,nm)): paths.append(nm) else: paths.append(nm) if full or absolute: if full: path = relpath(normpath(path)) else: path = abspath(normpath(path)) paths = [pathjoin(path, p) for p in paths] return paths @synchronize def ilistdir(self, path="/", wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False): fs, _mount_path, delegate_path = self._delegate(path) if fs is None: if path in ("/", ""): return raise ResourceNotFoundError(path) if fs is self: paths = self.mount_tree.names(path) for path in self._listdir_helper(path,paths,wildcard,full,absolute,dirs_only,files_only): yield path else: paths = fs.ilistdir(delegate_path, wildcard=wildcard, full=False, absolute=False, dirs_only=dirs_only) extra_paths = set(self.mount_tree.names(path)) if full: pathhead = relpath(normpath(path)) def mkpath(p): return pathjoin(pathhead,p) elif absolute: pathhead = abspath(normpath(path)) def mkpath(p): return pathjoin(pathhead,p) else: def mkpath(p): return p for p in paths: if p not in extra_paths: yield mkpath(p) for p in extra_paths: if dirs_only: if self.isdir(pathjoin(path,p)): yield mkpath(p) elif files_only: if self.isfile(pathjoin(path,p)): yield mkpath(p) else: yield mkpath(p) @synchronize def makedir(self, path, recursive=False, allow_recreate=False): fs, _mount_path, delegate_path = self._delegate(path) if fs is self or fs is None: raise UnsupportedError("make directory", msg="Can only makedir for mounted paths") if not delegate_path: if allow_recreate: return else: raise DestinationExistsError(path, msg="Can not create a directory that already exists (try allow_recreate=True): %(path)s") return fs.makedir(delegate_path, recursive=recursive, allow_recreate=allow_recreate) @synchronize def open(self, path, mode='r', buffering=-1, encoding=None, errors=None, newline=None, line_buffering=False, **kwargs): obj = self.mount_tree.get(path, None) if type(obj) is MountFS.FileMount: callable = obj.open_callable return callable(path, mode, **kwargs) fs, _mount_path, delegate_path = self._delegate(path) if fs is self or fs is None: raise ResourceNotFoundError(path) return fs.open(delegate_path, mode, **kwargs) @synchronize def setcontents(self, path, data=b'', encoding=None, errors=None, chunk_size=64*1024): obj = self.mount_tree.get(path, None) if type(obj) is MountFS.FileMount: return super(MountFS, self).setcontents(path, data, encoding=encoding, errors=errors, chunk_size=chunk_size) fs, _mount_path, delegate_path = self._delegate(path) if fs is self or fs is None: raise ParentDirectoryMissingError(path) return fs.setcontents(delegate_path, data, encoding=encoding, errors=errors, chunk_size=chunk_size) @synchronize def createfile(self, path, wipe=False): obj = self.mount_tree.get(path, None) if type(obj) is MountFS.FileMount: return super(MountFS, self).createfile(path, wipe=wipe) fs, _mount_path, delegate_path = self._delegate(path) if fs is self or fs is None: raise ParentDirectoryMissingError(path) return fs.createfile(delegate_path, wipe=wipe) @synchronize def remove(self, path): fs, _mount_path, delegate_path = self._delegate(path) if fs is self or fs is None: raise UnsupportedError("remove file", msg="Can only remove paths within a mounted dir") return fs.remove(delegate_path) @synchronize def removedir(self, path, recursive=False, force=False): path = normpath(path) if path in ('', '/'): raise RemoveRootError(path) fs, _mount_path, delegate_path = self._delegate(path) if fs is self or fs is None: raise ResourceInvalidError(path, msg="Can not removedir for an un-mounted path") return fs.removedir(delegate_path, recursive, force) @synchronize def rename(self, src, dst): fs1, _mount_path1, delegate_path1 = self._delegate(src) fs2, _mount_path2, delegate_path2 = self._delegate(dst) if fs1 is not fs2: raise OperationFailedError("rename resource", path=src) if fs1 is not self: return fs1.rename(delegate_path1, delegate_path2) object = self.mount_tree.get(src, None) _object2 = self.mount_tree.get(dst, None) if object is None: raise ResourceNotFoundError(src) raise UnsupportedError("rename resource", path=src) @synchronize def move(self,src,dst,**kwds): fs1, _mount_path1, delegate_path1 = self._delegate(src) fs2, _mount_path2, delegate_path2 = self._delegate(dst) if fs1 is fs2 and fs1 is not self: fs1.move(delegate_path1,delegate_path2,**kwds) else: super(MountFS,self).move(src,dst,**kwds) @synchronize def movedir(self,src,dst,**kwds): fs1, _mount_path1, delegate_path1 = self._delegate(src) fs2, _mount_path2, delegate_path2 = self._delegate(dst) if fs1 is fs2 and fs1 is not self: fs1.movedir(delegate_path1,delegate_path2,**kwds) else: super(MountFS,self).movedir(src,dst,**kwds) @synchronize def copy(self,src,dst,**kwds): fs1, _mount_path1, delegate_path1 = self._delegate(src) fs2, _mount_path2, delegate_path2 = self._delegate(dst) if fs1 is fs2 and fs1 is not self: fs1.copy(delegate_path1,delegate_path2,**kwds) else: super(MountFS,self).copy(src,dst,**kwds) @synchronize def copydir(self,src,dst,**kwds): fs1, _mount_path1, delegate_path1 = self._delegate(src) fs2, _mount_path2, delegate_path2 = self._delegate(dst) if fs1 is fs2 and fs1 is not self: fs1.copydir(delegate_path1,delegate_path2,**kwds) else: super(MountFS,self).copydir(src,dst,**kwds) @synchronize def mountdir(self, path, fs): """Mounts a host FS object on a given path. :param path: A path within the MountFS :param fs: A filesystem object to mount """ path = abspath(normpath(path)) self.mount_tree[path] = MountFS.DirMount(path, fs) mount = mountdir @synchronize def mountfile(self, path, open_callable=None, info_callable=None): """Mounts a single file path. :param path: A path within the MountFS :param open_callable: A callable that returns a file-like object, `open_callable` should have the same signature as :py:meth:`~fs.base.FS.open` :param info_callable: A callable that returns a dictionary with information regarding the file-like object, `info_callable` should have the same signagture as :py:meth:`~fs.base.FS.getinfo` """ self.mount_tree[path] = MountFS.FileMount(path, open_callable, info_callable) @synchronize def unmount(self, path): """Unmounts a path. :param path: Path to unmount :return: True if a path was unmounted, False if the path was already unmounted :rtype: bool """ try: del self.mount_tree[path] except KeyError: return False else: return True @synchronize def settimes(self, path, accessed_time=None, modified_time=None): path = normpath(path) fs, _mount_path, delegate_path = self._delegate(path) if fs is None: raise ResourceNotFoundError(path) if fs is self: raise UnsupportedError("settimes") fs.settimes(delegate_path, accessed_time, modified_time) @synchronize def getinfo(self, path): path = normpath(path) fs, _mount_path, delegate_path = self._delegate(path) if fs is None: if path in ("/", ""): return {} raise ResourceNotFoundError(path) if fs is self: if self.isfile(path): return self.mount_tree[path].info_callable(path) return {} return fs.getinfo(delegate_path) @synchronize def getsize(self, path): path = normpath(path) fs, _mount_path, delegate_path = self._delegate(path) if fs is None: raise ResourceNotFoundError(path) if fs is self: object = self.mount_tree.get(path, None) if object is None: raise ResourceNotFoundError(path) if not isinstance(object,MountFS.FileMount): raise ResourceInvalidError(path) size = object.info_callable(path).get("size", None) return size return fs.getinfo(delegate_path).get("size", None) @synchronize def getxattr(self,path,name,default=None): path = normpath(path) fs, _mount_path, delegate_path = self._delegate(path) if fs is None: if path in ("/", ""): return default raise ResourceNotFoundError(path) if fs is self: return default return fs.getxattr(delegate_path,name,default) @synchronize def setxattr(self,path,name,value): path = normpath(path) fs, _mount_path, delegate_path = self._delegate(path) if fs is None: raise ResourceNotFoundError(path) if fs is self: raise UnsupportedError("setxattr") return fs.setxattr(delegate_path,name,value) @synchronize def delxattr(self,path,name): path = normpath(path) fs, _mount_path, delegate_path = self._delegate(path) if fs is None: raise ResourceNotFoundError(path) if fs is self: return True return fs.delxattr(delegate_path, name) @synchronize def listxattrs(self,path): path = normpath(path) fs, _mount_path, delegate_path = self._delegate(path) if fs is None: raise ResourceNotFoundError(path) if fs is self: return [] return fs.listxattrs(delegate_path) fs-0.5.4/fs/xattrs.py0000664000175000017500000001653412512525115014411 0ustar willwill00000000000000""" fs.xattrs ========= Extended attribute support for FS This module defines a standard interface for FS subclasses that want to support extended file attributes, and a WrapFS subclass that can simulate extended attributes on top of an ordinary FS. FS instances offering extended attribute support must provide the following methods: * ``getxattr(path,name)`` Get the named attribute for the given path, or None if it does not exist * ``setxattr(path,name,value)`` Set the named attribute for the given path to the given value * ``delxattr(path,name)`` Delete the named attribute for the given path, raising KeyError if it does not exist * ``listxattrs(path)`` Iterate over all stored attribute names for the given path If extended attributes are required by FS-consuming code, it should use the function 'ensure_xattrs'. This will interrogate an FS object to determine if it has native xattr support, and return a wrapped version if it does not. """ import sys try: import cPickle as pickle except ImportError: import pickle from fs.path import * from fs.errors import * from fs.wrapfs import WrapFS from fs.base import synchronize def ensure_xattrs(fs): """Ensure that the given FS supports xattrs, simulating them if required. Given an FS object, this function returns an equivalent FS that has support for extended attributes. This may be the original object if they are supported natively, or a wrapper class is they must be simulated. :param fs: An FS object that must have xattrs """ try: # This attr doesn't have to exist, None should be returned by default fs.getxattr("/","testing-xattr") return fs except (AttributeError,UnsupportedError): return SimulateXAttr(fs) class SimulateXAttr(WrapFS): """FS wrapper class that simulates xattr support. The following methods are supplied for manipulating extended attributes: * listxattrs: list all extended attribute names for a path * getxattr: get an xattr of a path by name * setxattr: set an xattr of a path by name * delxattr: delete an xattr of a path by name For each file in the underlying FS, this class maintains a corresponding '.xattrs.FILENAME' file containing its extended attributes. Extended attributes of a directory are stored in the file '.xattrs' within the directory itself. """ def _get_attr_path(self, path, isdir=None): """Get the path of the file containing xattrs for the given path.""" if isdir is None: isdir = self.wrapped_fs.isdir(path) if isdir: attr_path = pathjoin(path, '.xattrs') else: dir_path, file_name = pathsplit(path) attr_path = pathjoin(dir_path, '.xattrs.'+file_name) return attr_path def _is_attr_path(self, path): """Check whether the given path references an xattrs file.""" _,name = pathsplit(path) if name.startswith(".xattrs"): return True return False def _get_attr_dict(self, path): """Retrieve the xattr dictionary for the given path.""" attr_path = self._get_attr_path(path) if self.wrapped_fs.exists(attr_path): try: return pickle.loads(self.wrapped_fs.getcontents(attr_path)) except EOFError: return {} else: return {} def _set_attr_dict(self, path, attrs): """Store the xattr dictionary for the given path.""" attr_path = self._get_attr_path(path) self.wrapped_fs.setcontents(attr_path, pickle.dumps(attrs)) @synchronize def setxattr(self, path, key, value): """Set an extended attribute on the given path.""" if not self.exists(path): raise ResourceNotFoundError(path) key = unicode(key) attrs = self._get_attr_dict(path) attrs[key] = str(value) self._set_attr_dict(path, attrs) @synchronize def getxattr(self, path, key, default=None): """Retrieve an extended attribute for the given path.""" if not self.exists(path): raise ResourceNotFoundError(path) attrs = self._get_attr_dict(path) return attrs.get(key, default) @synchronize def delxattr(self, path, key): if not self.exists(path): raise ResourceNotFoundError(path) attrs = self._get_attr_dict(path) try: del attrs[key] except KeyError: pass self._set_attr_dict(path, attrs) @synchronize def listxattrs(self,path): """List all the extended attribute keys set on the given path.""" if not self.exists(path): raise ResourceNotFoundError(path) return self._get_attr_dict(path).keys() def _encode(self,path): """Prevent requests for operations on .xattr files.""" if self._is_attr_path(path): raise PathError(path,msg="Paths cannot contain '.xattrs': %(path)s") return path def _decode(self,path): return path def listdir(self,path="",*args,**kwds): """Prevent .xattr from appearing in listings.""" entries = self.wrapped_fs.listdir(path,*args,**kwds) return [e for e in entries if not self._is_attr_path(e)] def ilistdir(self,path="",*args,**kwds): """Prevent .xattr from appearing in listings.""" for e in self.wrapped_fs.ilistdir(path,*args,**kwds): if not self._is_attr_path(e): yield e def remove(self,path): """Remove .xattr when removing a file.""" attr_file = self._get_attr_path(path,isdir=False) self.wrapped_fs.remove(path) try: self.wrapped_fs.remove(attr_file) except ResourceNotFoundError: pass def removedir(self,path,recursive=False,force=False): """Remove .xattr when removing a directory.""" try: self.wrapped_fs.removedir(path,recursive=recursive,force=force) except DirectoryNotEmptyError: # The xattr file could block the underlying removedir(). # Remove it, but be prepared to restore it on error. if self.listdir(path) != []: raise attr_file = self._get_attr_path(path,isdir=True) attr_file_contents = self.wrapped_fs.getcontents(attr_file) self.wrapped_fs.remove(attr_file) try: self.wrapped_fs.removedir(path,recursive=recursive) except FSError: self.wrapped_fs.setcontents(attr_file,attr_file_contents) raise def copy(self,src,dst,**kwds): """Ensure xattrs are copied when copying a file.""" self.wrapped_fs.copy(self._encode(src),self._encode(dst),**kwds) s_attr_file = self._get_attr_path(src) d_attr_file = self._get_attr_path(dst) try: self.wrapped_fs.copy(s_attr_file,d_attr_file,overwrite=True) except ResourceNotFoundError,e: pass def move(self,src,dst,**kwds): """Ensure xattrs are preserved when moving a file.""" self.wrapped_fs.move(self._encode(src),self._encode(dst),**kwds) s_attr_file = self._get_attr_path(src) d_attr_file = self._get_attr_path(dst) try: self.wrapped_fs.move(s_attr_file,d_attr_file,overwrite=True) except ResourceNotFoundError: pass fs-0.5.4/fs/remote.py0000664000175000017500000006475112512525115014363 0ustar willwill00000000000000""" fs.remote ========= Utilities for interfacing with remote filesystems This module provides reusable utility functions that can be used to construct FS subclasses interfacing with a remote filesystem. These include: * RemoteFileBuffer: a file-like object that locally buffers the contents of a remote file, writing them back on flush() or close(). * ConnectionManagerFS: a WrapFS subclass that tracks the connection state of a remote FS, and allows client code to wait for a connection to be re-established. * CacheFS: a WrapFS subclass that caches file and directory meta-data in memory, to speed access to a remote FS. """ from __future__ import with_statement import time import stat as statinfo from errno import EINVAL import fs.utils from fs.base import threading, FS from fs.wrapfs import WrapFS, wrap_fs_methods from fs.wrapfs.lazyfs import LazyFS from fs.path import * from fs.errors import * from fs.local_functools import wraps from fs.filelike import StringIO, SpooledTemporaryFile, FileWrapper from fs import SEEK_SET, SEEK_CUR, SEEK_END _SENTINAL = object() from six import PY3, b class RemoteFileBuffer(FileWrapper): """File-like object providing buffer for local file operations. Instances of this class manage a local tempfile buffer corresponding to the contents of a remote file. All reads and writes happen locally, with the content being copied to the remote file only on flush() or close(). Writes to the remote file are performed using the setcontents() method on the owning FS object. The intended use-case is for a remote filesystem (e.g. S3FS) to return instances of this class from its open() method, and to provide the file-uploading logic in its setcontents() method, as in the following pseudo-code:: def open(self,path,mode="r"): rf = self._get_remote_file(path) return RemoteFileBuffer(self,path,mode,rf) def setcontents(self,path,file): self._put_remote_file(path,file) The contents of the remote file are read into the buffer on-demand. """ max_size_in_memory = 1024 * 8 def __init__(self, fs, path, mode, rfile=None, write_on_flush=True): """RemoteFileBuffer constructor. The owning filesystem, path and mode must be provided. If the optional argument 'rfile' is provided, it must be a read()-able object or a string containing the initial file contents. """ wrapped_file = SpooledTemporaryFile(max_size=self.max_size_in_memory) self.fs = fs self.path = path self.write_on_flush = write_on_flush self._changed = False self._readlen = 0 # How many bytes already loaded from rfile self._rfile = None # Reference to remote file object self._eof = False # Reached end of rfile? if getattr(fs, "_lock", None) is not None: self._lock = fs._lock.__class__() else: self._lock = threading.RLock() if "r" in mode or "+" in mode or "a" in mode: if rfile is None: # File was just created, force to write anything self._changed = True self._eof = True if not hasattr(rfile, "read"): #rfile = StringIO(unicode(rfile)) rfile = StringIO(rfile) self._rfile = rfile else: # Do not use remote file object self._eof = True self._rfile = None self._changed = True if rfile is not None and hasattr(rfile,"close"): rfile.close() super(RemoteFileBuffer,self).__init__(wrapped_file,mode) # FIXME: What if mode with position on eof? if "a" in mode: # Not good enough... self.seek(0, SEEK_END) def __del__(self): # Don't try to close a partially-constructed file if "_lock" in self.__dict__: if not self.closed: try: self.close() except FSError: pass def _write(self,data,flushing=False): with self._lock: # Do we need to discard info from the buffer? toread = len(data) - (self._readlen - self.wrapped_file.tell()) if toread > 0: if not self._eof: self._fillbuffer(toread) else: self._readlen += toread self._changed = True self.wrapped_file.write(data) def _read_remote(self, length=None): """Read data from the remote file into the local buffer.""" chunklen = 1024 * 256 bytes_read = 0 while True: toread = chunklen if length is not None and length - bytes_read < chunklen: toread = length - bytes_read if not toread: break data = self._rfile.read(toread) datalen = len(data) if not datalen: self._eof = True break bytes_read += datalen self.wrapped_file.write(data) if datalen < toread: # We reached EOF, # no more reads needed self._eof = True break if self._eof and self._rfile is not None: self._rfile.close() self._readlen += bytes_read def _fillbuffer(self, length=None): """Fill the local buffer, leaving file position unchanged. This method is used for on-demand loading of data from the remote file into the buffer. It reads 'length' bytes from rfile and writes them into the buffer, seeking back to the original file position. """ curpos = self.wrapped_file.tell() if length == None: if not self._eof: # Read all data and we didn't reached EOF # Merge endpos - tell + bytes from rfile self.wrapped_file.seek(0, SEEK_END) self._read_remote() self._eof = True self.wrapped_file.seek(curpos) elif not self._eof: if curpos + length > self._readlen: # Read all data and we didn't reached EOF # Load endpos - tell() + len bytes from rfile toload = length - (self._readlen - curpos) self.wrapped_file.seek(0, SEEK_END) self._read_remote(toload) self.wrapped_file.seek(curpos) def _read(self, length=None): if length is not None and length < 0: length = None with self._lock: self._fillbuffer(length) data = self.wrapped_file.read(length if length != None else -1) if not data: data = None return data def _seek(self,offset,whence=SEEK_SET): with self._lock: if not self._eof: # Count absolute position of seeking if whence == SEEK_SET: abspos = offset elif whence == SEEK_CUR: abspos = offset + self.wrapped_file.tell() elif whence == SEEK_END: abspos = None else: raise IOError(EINVAL, 'Invalid whence') if abspos != None: toread = abspos - self._readlen if toread > 0: self.wrapped_file.seek(self._readlen) self._fillbuffer(toread) else: self.wrapped_file.seek(self._readlen) self._fillbuffer() self.wrapped_file.seek(offset, whence) def _truncate(self,size): with self._lock: if not self._eof and self._readlen < size: # Read the rest of file self._fillbuffer(size - self._readlen) # Lock rfile self._eof = True elif self._readlen >= size: # Crop rfile metadata self._readlen = size if size != None else 0 # Lock rfile self._eof = True self.wrapped_file.truncate(size) self._changed = True self.flush() if self._rfile is not None: self._rfile.close() def flush(self): with self._lock: self.wrapped_file.flush() if self.write_on_flush: self._setcontents() def _setcontents(self): if not self._changed: # Nothing changed, no need to write data back return # If not all data loaded, load until eof if not self._eof: self._fillbuffer() if "w" in self.mode or "a" in self.mode or "+" in self.mode: pos = self.wrapped_file.tell() self.wrapped_file.seek(0) self.fs.setcontents(self.path, self.wrapped_file) self.wrapped_file.seek(pos) def close(self): with self._lock: if not self.closed: self._setcontents() if self._rfile is not None: self._rfile.close() super(RemoteFileBuffer,self).close() class ConnectionManagerFS(LazyFS): """FS wrapper providing simple connection management of a remote FS. The ConnectionManagerFS class is designed to wrap a remote FS object and provide some convenience methods for dealing with its remote connection state. The boolean attribute 'connected' indicates whether the remote filesystem has an active connection, and is initially True. If any of the remote filesystem methods raises a RemoteConnectionError, 'connected' will switch to False and remain so until a successful remote method call. Application code can use the method 'wait_for_connection' to block until the connection is re-established. Currently this reconnection is checked by a simple polling loop; eventually more sophisticated operating-system integration may be added. Since some remote FS classes can raise RemoteConnectionError during initialization, this class makes use of lazy initialization. The remote FS can be specified as an FS instance, an FS subclass, or a (class,args) or (class,args,kwds) tuple. For example:: >>> fs = ConnectionManagerFS(MyRemoteFS("http://www.example.com/")) Traceback (most recent call last): ... RemoteConnectionError: couldn't connect to "http://www.example.com/" >>> fs = ConnectionManagerFS((MyRemoteFS,["http://www.example.com/"])) >>> fs.connected False >>> """ poll_interval = 1 def __init__(self,wrapped_fs,poll_interval=None,connected=True): super(ConnectionManagerFS,self).__init__(wrapped_fs) if poll_interval is not None: self.poll_interval = poll_interval self._connection_cond = threading.Condition() self._poll_thread = None self._poll_sleeper = threading.Event() self.connected = connected def setcontents(self, path, data=b'', encoding=None, errors=None, chunk_size=64*1024): return self.wrapped_fs.setcontents(path, data, encoding=encoding, errors=errors, chunk_size=chunk_size) def __getstate__(self): state = super(ConnectionManagerFS,self).__getstate__() del state["_connection_cond"] del state["_poll_sleeper"] state["_poll_thread"] = None return state def __setstate__(self,state): super(ConnectionManagerFS,self).__setstate__(state) self._connection_cond = threading.Condition() self._poll_sleeper = threading.Event() def wait_for_connection(self,timeout=None,force_wait=False): self._connection_cond.acquire() try: if force_wait: self.connected = False if not self.connected: if not self._poll_thread: target = self._poll_connection self._poll_thread = threading.Thread(target=target) self._poll_thread.daemon = True self._poll_thread.start() self._connection_cond.wait(timeout) finally: self._connection_cond.release() def _poll_connection(self): while not self.connected and not self.closed: try: self.wrapped_fs.getinfo("/") except RemoteConnectionError: self._poll_sleeper.wait(self.poll_interval) self._poll_sleeper.clear() except FSError: break else: break self._connection_cond.acquire() try: if not self.closed: self.connected = True self._poll_thread = None self._connection_cond.notifyAll() finally: self._connection_cond.release() def close(self): if not self.closed: try: super(ConnectionManagerFS,self).close() except (RemoteConnectionError,): pass if self._poll_thread: self.connected = True self._poll_sleeper.set() self._poll_thread.join() self._poll_thread = None def _ConnectionManagerFS_method_wrapper(func): """Method wrapper for ConnectionManagerFS. This method wrapper keeps an eye out for RemoteConnectionErrors and adjusts self.connected accordingly. """ @wraps(func) def wrapper(self,*args,**kwds): try: result = func(self,*args,**kwds) except RemoteConnectionError: self.connected = False raise except FSError: self.connected = True raise else: self.connected = True return result return wrapper wrap_fs_methods(_ConnectionManagerFS_method_wrapper,ConnectionManagerFS) class CachedInfo(object): """Info objects stored in cache for CacheFS.""" __slots__ = ("timestamp","info","has_full_info","has_full_children") def __init__(self,info={},has_full_info=True,has_full_children=False): self.timestamp = time.time() self.info = info self.has_full_info = has_full_info self.has_full_children = has_full_children def clone(self): new_ci = self.__class__() new_ci.update_from(self) return new_ci def update_from(self,other): self.timestamp = other.timestamp self.info = other.info self.has_full_info = other.has_full_info self.has_full_children = other.has_full_children @classmethod def new_file_stub(cls): info = {"info" : 0700 | statinfo.S_IFREG} return cls(info,has_full_info=False) @classmethod def new_dir_stub(cls): info = {"info" : 0700 | statinfo.S_IFDIR} return cls(info,has_full_info=False) class CacheFSMixin(FS): """Simple FS mixin to cache meta-data of a remote filesystems. This FS mixin implements a simplistic cache that can help speed up access to a remote filesystem. File and directory meta-data is cached but the actual file contents are not. If you want to add caching to an existing FS object, use the CacheFS class instead; it's an easy-to-use wrapper rather than a mixin. This mixin class is provided for FS implementors who want to use caching internally in their own classes. FYI, the implementation of CacheFS is this: class CacheFS(CacheFSMixin,WrapFS): pass """ def __init__(self,*args,**kwds): """CacheFSMixin constructor. The optional keyword argument 'cache_timeout' specifies the cache timeout in seconds. The default timeout is 1 second. To prevent cache entries from ever timing out, set it to None. The optional keyword argument 'max_cache_size' specifies the maximum number of entries to keep in the cache. To allow the cache to grow without bound, set it to None. The default is 1000. """ self.cache_timeout = kwds.pop("cache_timeout",1) self.max_cache_size = kwds.pop("max_cache_size",1000) self.__cache = PathMap() self.__cache_size = 0 self.__cache_lock = threading.RLock() super(CacheFSMixin,self).__init__(*args,**kwds) def clear_cache(self,path=""): with self.__cache_lock: self.__cache.clear(path) try: scc = super(CacheFSMixin,self).clear_cache except AttributeError: pass else: scc() def __getstate__(self): state = super(CacheFSMixin,self).__getstate__() state.pop("_CacheFSMixin__cache",None) state.pop("_CacheFSMixin__cache_size",None) state.pop("_CacheFSMixin__cache_lock",None) return state def __setstate__(self,state): super(CacheFSMixin,self).__setstate__(state) self.__cache = PathMap() self.__cache_size = 0 self.__cache_lock = threading.RLock() def __get_cached_info(self,path,default=_SENTINAL): try: info = self.__cache[path] if self.cache_timeout is not None: now = time.time() if info.timestamp < (now - self.cache_timeout): with self.__cache_lock: self.__expire_from_cache(path) raise KeyError return info except KeyError: if default is not _SENTINAL: return default raise def __set_cached_info(self,path,new_ci,old_ci=None): was_room = True with self.__cache_lock: # Free up some room in the cache if self.max_cache_size is not None and old_ci is None: while self.__cache_size >= self.max_cache_size: try: to_del = iter(self.__cache).next() except StopIteration: break else: was_room = False self.__expire_from_cache(to_del) # Atomically add to the cache. # If there's a race, newest information wins ci = self.__cache.setdefault(path,new_ci) if ci is new_ci: self.__cache_size += 1 else: if old_ci is None or ci is old_ci: if ci.timestamp < new_ci.timestamp: ci.update_from(new_ci) return was_room def __expire_from_cache(self,path): del self.__cache[path] self.__cache_size -= 1 for ancestor in recursepath(path): try: self.__cache[ancestor].has_full_children = False except KeyError: pass def open(self, path, mode='r', buffering=-1, encoding=None, errors=None, newline=None, line_buffering=False, **kwargs): # Try to validate the entry using the cached info try: ci = self.__get_cached_info(path) except KeyError: if path in ("", "/"): raise ResourceInvalidError(path) try: ppath = dirname(path) pci = self.__get_cached_info(ppath) except KeyError: pass else: if not fs.utils.isdir(super(CacheFSMixin, self), ppath, pci.info): raise ResourceInvalidError(path) if pci.has_full_children: raise ResourceNotFoundError(path) else: if not fs.utils.isfile(super(CacheFSMixin, self), path, ci.info): raise ResourceInvalidError(path) f = super(CacheFSMixin, self).open(path, mode=mode, buffering=buffering, encoding=encoding, errors=errors, newline=newline, line_buffering=line_buffering, **kwargs) if "w" in mode or "a" in mode or "+" in mode: with self.__cache_lock: self.__cache.clear(path) f = self._CacheInvalidatingFile(self, path, f, mode) return f class _CacheInvalidatingFile(FileWrapper): def __init__(self, owner, path, wrapped_file, mode=None): self.path = path sup = super(CacheFSMixin._CacheInvalidatingFile, self) sup.__init__(wrapped_file, mode) self.owner = owner def _write(self, string, flushing=False): with self.owner._CacheFSMixin__cache_lock: self.owner._CacheFSMixin__cache.clear(self.path) sup = super(CacheFSMixin._CacheInvalidatingFile, self) return sup._write(string, flushing=flushing) def _truncate(self, size): with self.owner._CacheFSMixin__cache_lock: self.owner._CacheFSMixin__cache.clear(self.path) sup = super(CacheFSMixin._CacheInvalidatingFile, self) return sup._truncate(size) def exists(self, path): try: self.getinfo(path) except ResourceNotFoundError: return False else: return True def isdir(self, path): try: self.__cache.iternames(path).next() return True except StopIteration: pass except RuntimeError: pass try: info = self.getinfo(path) except ResourceNotFoundError: return False else: return fs.utils.isdir(super(CacheFSMixin, self), path, info) def isfile(self, path): try: self.__cache.iternames(path).next() return False except StopIteration: pass except RuntimeError: pass try: info = self.getinfo(path) except ResourceNotFoundError: return False else: return fs.utils.isfile(super(CacheFSMixin, self), path, info) def getinfo(self, path): try: ci = self.__get_cached_info(path) if not ci.has_full_info: raise KeyError info = ci.info except KeyError: info = super(CacheFSMixin, self).getinfo(path) self.__set_cached_info(path, CachedInfo(info)) return info def listdir(self,path="",*args,**kwds): return list(nm for (nm, _info) in self.listdirinfo(path,*args,**kwds)) def ilistdir(self,path="",*args,**kwds): for (nm, _info) in self.ilistdirinfo(path,*args,**kwds): yield nm def listdirinfo(self,path="",*args,**kwds): items = super(CacheFSMixin,self).listdirinfo(path,*args,**kwds) with self.__cache_lock: names = set() for (nm,info) in items: names.add(basename(nm)) cpath = pathjoin(path,basename(nm)) ci = CachedInfo(info) self.__set_cached_info(cpath,ci) to_del = [] for nm in self.__cache.names(path): if nm not in names: to_del.append(nm) for nm in to_del: self.__cache.clear(pathjoin(path,nm)) #try: # pci = self.__cache[path] #except KeyError: # pci = CachedInfo.new_dir_stub() # self.__cache[path] = pci #pci.has_full_children = True return items def ilistdirinfo(self,path="",*args,**kwds): items = super(CacheFSMixin,self).ilistdirinfo(path,*args,**kwds) for (nm,info) in items: cpath = pathjoin(path,basename(nm)) ci = CachedInfo(info) self.__set_cached_info(cpath,ci) yield (nm,info) def getsize(self,path): return self.getinfo(path)["size"] def setcontents(self, path, data=b'', encoding=None, errors=None, chunk_size=64*1024): supsc = super(CacheFSMixin, self).setcontents res = supsc(path, data, encoding=None, errors=None, chunk_size=chunk_size) with self.__cache_lock: self.__cache.clear(path) self.__cache[path] = CachedInfo.new_file_stub() return res def createfile(self, path, wipe=False): super(CacheFSMixin,self).createfile(path, wipe=wipe) with self.__cache_lock: self.__cache.clear(path) self.__cache[path] = CachedInfo.new_file_stub() def makedir(self,path,*args,**kwds): super(CacheFSMixin,self).makedir(path,*args,**kwds) with self.__cache_lock: self.__cache.clear(path) self.__cache[path] = CachedInfo.new_dir_stub() def remove(self,path): super(CacheFSMixin,self).remove(path) with self.__cache_lock: self.__cache.clear(path) def removedir(self,path,**kwds): super(CacheFSMixin,self).removedir(path,**kwds) with self.__cache_lock: self.__cache.clear(path) def rename(self,src,dst): super(CacheFSMixin,self).rename(src,dst) with self.__cache_lock: for (subpath,ci) in self.__cache.items(src): self.__cache[pathjoin(dst,subpath)] = ci.clone() self.__cache.clear(src) def copy(self,src,dst,**kwds): super(CacheFSMixin,self).copy(src,dst,**kwds) with self.__cache_lock: for (subpath,ci) in self.__cache.items(src): self.__cache[pathjoin(dst,subpath)] = ci.clone() def copydir(self,src,dst,**kwds): super(CacheFSMixin,self).copydir(src,dst,**kwds) with self.__cache_lock: for (subpath,ci) in self.__cache.items(src): self.__cache[pathjoin(dst,subpath)] = ci.clone() def move(self,src,dst,**kwds): super(CacheFSMixin,self).move(src,dst,**kwds) with self.__cache_lock: for (subpath,ci) in self.__cache.items(src): self.__cache[pathjoin(dst,subpath)] = ci.clone() self.__cache.clear(src) def movedir(self,src,dst,**kwds): super(CacheFSMixin,self).movedir(src,dst,**kwds) with self.__cache_lock: for (subpath,ci) in self.__cache.items(src): self.__cache[pathjoin(dst,subpath)] = ci.clone() self.__cache.clear(src) def settimes(self,path,*args,**kwds): super(CacheFSMixin,self).settimes(path,*args,**kwds) with self.__cache_lock: self.__cache.pop(path,None) class CacheFS(CacheFSMixin,WrapFS): """Simple FS wrapper to cache meta-data of a remote filesystems. This FS mixin implements a simplistic cache that can help speed up access to a remote filesystem. File and directory meta-data is cached but the actual file contents are not. """ pass fs-0.5.4/fs/commands/0000755000000000000000000000000012621617365014310 5ustar rootroot00000000000000fs-0.5.4/fs/commands/fsls.py0000664000175000017500000001500612512525115015625 0ustar willwill00000000000000#!/usr/bin/env python from fs.errors import FSError from fs.opener import opener from fs.path import pathsplit, abspath, isdotfile, iswildcard from fs.commands.runner import Command from collections import defaultdict import sys class FSls(Command): usage = """fsls [OPTIONS]... [PATH] List contents of [PATH]""" def get_optparse(self): optparse = super(FSls, self).get_optparse() optparse.add_option('-u', '--full', dest='fullpath', action="store_true", default=False, help="output full path", metavar="FULL") optparse.add_option('-s', '--syspath', dest='syspath', action="store_true", default=False, help="output system path (if one exists)", metavar="SYSPATH") optparse.add_option('-r', '--url', dest='url', action="store_true", default=False, help="output URL in place of path (if one exists)", metavar="URL") optparse.add_option('-d', '--dirsonly', dest='dirsonly', action="store_true", default=False, help="list directories only", metavar="DIRSONLY") optparse.add_option('-f', '--filesonly', dest='filesonly', action="store_true", default=False, help="list files only", metavar="FILESONLY") optparse.add_option('-l', '--long', dest='long', action="store_true", default=False, help="use a long listing format", metavar="LONG") optparse.add_option('-a', '--all', dest='all', action='store_true', default=False, help="do not hide dot files") return optparse def do_run(self, options, args): output = self.output if not args: args = [u'.'] dir_paths = [] file_paths = [] fs_used = set() for fs_url in args: fs, path = self.open_fs(fs_url) fs_used.add(fs) path = path or '.' wildcard = None if iswildcard(path): path, wildcard = pathsplit(path) if path != '.' and fs.isfile(path): if not options.dirsonly: file_paths.append(path) else: if not options.filesonly: dir_paths += fs.listdir(path, wildcard=wildcard, full=options.fullpath or options.url, dirs_only=True) if not options.dirsonly: file_paths += fs.listdir(path, wildcard=wildcard, full=options.fullpath or options.url, files_only=True) for fs in fs_used: try: fs.close() except FSError: pass if options.syspath: # Path without a syspath, just won't be displayed dir_paths = filter(None, [fs.getsyspath(path, allow_none=True) for path in dir_paths]) file_paths = filter(None, [fs.getsyspath(path, allow_none=True) for path in file_paths]) if options.url: # Path without a syspath, just won't be displayed dir_paths = filter(None, [fs.getpathurl(path, allow_none=True) for path in dir_paths]) file_paths = filter(None, [fs.getpathurl(path, allow_none=True) for path in file_paths]) dirs = frozenset(dir_paths) paths = sorted(dir_paths + file_paths, key=lambda p: p.lower()) if not options.all: paths = [path for path in paths if not isdotfile(path)] if not paths: return def columnize(paths, num_columns): col_height = (len(paths) + num_columns - 1) / num_columns columns = [[] for _ in xrange(num_columns)] col_no = 0 col_pos = 0 for path in paths: columns[col_no].append(path) col_pos += 1 if col_pos >= col_height: col_no += 1 col_pos = 0 padded_columns = [] wrap_filename = self.wrap_filename wrap_dirname = self.wrap_dirname def wrap(path): if path in dirs: return wrap_dirname(path.ljust(max_width)) else: return wrap_filename(path.ljust(max_width)) for column in columns: if column: max_width = max([len(path) for path in column]) else: max_width = 1 max_width = min(max_width, terminal_width) padded_columns.append([wrap(path) for path in column]) return padded_columns def condense_columns(columns): max_column_height = max([len(col) for col in columns]) lines = [[] for _ in xrange(max_column_height)] for column in columns: for line, path in zip(lines, column): line.append(path) return '\n'.join(u' '.join(line) for line in lines) if options.long: for path in paths: if path in dirs: output((self.wrap_dirname(path), '\n')) else: output((self.wrap_filename(path), '\n')) else: terminal_width = self.terminal_width path_widths = [len(path) for path in paths] smallest_paths = min(path_widths) num_paths = len(paths) num_cols = min(terminal_width // (smallest_paths + 2), num_paths) while num_cols: col_height = (num_paths + num_cols - 1) // num_cols line_width = 0 for col_no in xrange(num_cols): try: col_width = max(path_widths[col_no * col_height: (col_no + 1) * col_height]) except ValueError: continue line_width += col_width if line_width > terminal_width: break line_width += 2 else: if line_width - 1 <= terminal_width: break num_cols -= 1 num_cols = max(1, num_cols) columns = columnize(paths, num_cols) output((condense_columns(columns), '\n')) def run(): return FSls().run() if __name__ == "__main__": sys.exit(run()) fs-0.5.4/fs/commands/fstree.py0000664000175000017500000000627512512525115016156 0ustar willwill00000000000000#!/usr/bin/env python import sys from fs.opener import opener from fs.commands.runner import Command from fs.utils import print_fs class FSTree(Command): usage = """fstree [OPTION]... [PATH] Recursively display the contents of PATH in an ascii tree""" def get_optparse(self): optparse = super(FSTree, self).get_optparse() optparse.add_option('-l', '--level', dest='depth', type="int", default=5, help="Descend only LEVEL directories deep (-1 for infinite)", metavar="LEVEL") optparse.add_option('-g', '--gui', dest='gui', action='store_true', default=False, help="browse the tree with a gui") optparse.add_option('-a', '--all', dest='all', action='store_true', default=False, help="do not hide dot files") optparse.add_option('--dirsfirst', dest='dirsfirst', action='store_true', default=False, help="List directories before files") optparse.add_option('-P', dest="pattern", default=None, help="Only list files that match the given pattern") optparse.add_option('-d', dest="dirsonly", default=False, action='store_true', help="List directories only") return optparse def do_run(self, options, args): if not args: args = ['.'] for fs, path, is_dir in self.get_resources(args, single=True): if not is_dir: self.error(u"'%s' is not a dir\n" % path) return 1 fs.cache_hint(True) if options.gui: from fs.browsewin import browse if path: fs = fs.opendir(path) browse(fs, hide_dotfiles=not options.all) else: if options.depth < 0: max_levels = None else: max_levels = options.depth self.output(self.wrap_dirname(args[0] + '\n')) dircount, filecount = print_fs(fs, path or '', file_out=self.output_file, max_levels=max_levels, terminal_colors=self.terminal_colors, hide_dotfiles=not options.all, dirs_first=options.dirsfirst, files_wildcard=options.pattern, dirs_only=options.dirsonly) self.output('\n') def pluralize(one, many, count): if count == 1: return '%i %s' % (count, one) else: return '%i %s' % (count, many) self.output("%s, %s\n" % (pluralize('directory', 'directories', dircount), pluralize('file', 'files', filecount))) def run(): return FSTree().run() if __name__ == "__main__": sys.exit(run()) fs-0.5.4/fs/commands/fscat.py0000664000175000017500000000130712512525115015755 0ustar willwill00000000000000#!/usr/bin/env python from fs.commands.runner import Command import sys class FSCat(Command): usage = """fscat [OPTION]... [FILE]... Concetanate FILE(s)""" version = "1.0" def do_run(self, options, args): count = 0 for fs, path, is_dir in self.get_resources(args): if is_dir: self.error('%s is a directory\n' % path) return 1 self.output(fs.getcontents(path)) count += 1 if self.is_terminal() and count: self.output('\n') def run(): return FSCat().run() if __name__ == "__main__": sys.exit(run()) fs-0.5.4/fs/commands/fsrm.py0000664000175000017500000000354112512525115015626 0ustar willwill00000000000000#!/usr/bin/env python from fs.errors import ResourceNotFoundError from fs.opener import opener from fs.commands.runner import Command import sys class FSrm(Command): usage = """fsrm [OPTION]... [PATH] Remove a file or directory at PATH""" def get_optparse(self): optparse = super(FSrm, self).get_optparse() optparse.add_option('-f', '--force', dest='force', action='store_true', default=False, help='ignore non-existent files, never prompt') optparse.add_option('-i', '--interactive', dest='interactive', action='store_true', default=False, help='prompt before removing') optparse.add_option('-r', '--recursive', dest='recursive', action='store_true', default=False, help='remove directories and their contents recursively') return optparse def do_run(self, options, args): interactive = options.interactive verbose = options.verbose for fs, path, is_dir in self.get_resources(args): if interactive: if is_dir: msg = "remove directory '%s'?" % path else: msg = "remove file '%s'?" % path if not self.ask(msg) in 'yY': continue try: if is_dir: fs.removedir(path, force=options.recursive) else: fs.remove(path) except ResourceNotFoundError: if not options.force: raise else: if verbose: self.output("removed '%s'\n" % path) def run(): return FSrm().run() if __name__ == "__main__": sys.exit(run()) fs-0.5.4/fs/commands/fscp.py0000664000175000017500000002257012512525115015615 0ustar willwill00000000000000#!/usr/bin/env python from fs.utils import copyfile, copyfile_non_atomic from fs.path import pathjoin, iswildcard from fs.commands.runner import Command import sys import Queue as queue import time import threading class FileOpThread(threading.Thread): def __init__(self, action, name, dest_fs, queue, on_done, on_error): self.action = action self.dest_fs = dest_fs self.queue = queue self.on_done = on_done self.on_error = on_error self.finish_event = threading.Event() super(FileOpThread, self).__init__() def run(self): while not self.finish_event.isSet(): try: path_type, fs, path, dest_path = self.queue.get(timeout=0.1) except queue.Empty: continue try: if path_type == FScp.DIR: self.dest_fs.makedir(path, recursive=True, allow_recreate=True) else: self.action(fs, path, self.dest_fs, dest_path, overwrite=True) except Exception, e: self.on_error(e) self.queue.task_done() break else: self.queue.task_done() self.on_done(path_type, fs, path, self.dest_fs, dest_path) class FScp(Command): DIR, FILE = 0, 1 usage = """fscp [OPTION]... [SOURCE]... [DESTINATION] Copy SOURCE to DESTINATION""" def get_action(self): if self.options.threads > 1: return copyfile_non_atomic else: return copyfile def get_verb(self): return 'copying...' def get_optparse(self): optparse = super(FScp, self).get_optparse() optparse.add_option('-p', '--progress', dest='progress', action="store_true", default=False, help="show progress", metavar="PROGRESS") optparse.add_option('-t', '--threads', dest='threads', action="store", default=1, help="number of threads to use", type="int", metavar="THREAD_COUNT") return optparse def do_run(self, options, args): self.options = options if len(args) < 2: self.error("at least two filesystems required\n") return 1 srcs = args[:-1] dst = args[-1] dst_fs, dst_path = self.open_fs(dst, writeable=True, create_dir=True) if dst_path is not None and dst_fs.isfile(dst_path): self.error('Destination must be a directory\n') return 1 if dst_path: dst_fs = dst_fs.makeopendir(dst_path) dst_path = None copy_fs_paths = [] progress = options.progress if progress: sys.stdout.write(self.progress_bar(len(srcs), 0, 'scanning...')) sys.stdout.flush() self.root_dirs = [] for i, fs_url in enumerate(srcs): src_fs, src_path = self.open_fs(fs_url) if src_path is None: src_path = '/' if iswildcard(src_path): for file_path in src_fs.listdir(wildcard=src_path, full=True): copy_fs_paths.append((self.FILE, src_fs, file_path, file_path)) else: if src_fs.isdir(src_path): self.root_dirs.append((src_fs, src_path)) src_sub_fs = src_fs.opendir(src_path) for dir_path, file_paths in src_sub_fs.walk(): if dir_path not in ('', '/'): copy_fs_paths.append((self.DIR, src_sub_fs, dir_path, dir_path)) sub_fs = src_sub_fs.opendir(dir_path) for file_path in file_paths: copy_fs_paths.append((self.FILE, sub_fs, file_path, pathjoin(dir_path, file_path))) else: if src_fs.exists(src_path): copy_fs_paths.append((self.FILE, src_fs, src_path, src_path)) else: self.error('%s is not a file or directory\n' % src_path) return 1 if progress: sys.stdout.write(self.progress_bar(len(srcs), i + 1, 'scanning...')) sys.stdout.flush() if progress: sys.stdout.write(self.progress_bar(len(copy_fs_paths), 0, self.get_verb())) sys.stdout.flush() if self.options.threads > 1: copy_fs_dirs = [r for r in copy_fs_paths if r[0] == self.DIR] copy_fs_paths = [r for r in copy_fs_paths if r[0] == self.FILE] for path_type, fs, path, dest_path in copy_fs_dirs: dst_fs.makedir(path, allow_recreate=True, recursive=True) self.lock = threading.RLock() self.total_files = len(copy_fs_paths) self.done_files = 0 file_queue = queue.Queue() threads = [FileOpThread(self.get_action(), 'T%i' % i, dst_fs, file_queue, self.on_done, self.on_error) for i in xrange(options.threads)] for thread in threads: thread.start() self.action_errors = [] complete = False try: enqueue = file_queue.put for resource in copy_fs_paths: enqueue(resource) while not file_queue.empty(): time.sleep(0) if self.any_error(): raise SystemExit # Can't use queue.join here, or KeyboardInterrupt will not be # caught until the queue is finished #file_queue.join() except KeyboardInterrupt: options.progress = False self.output("\nCancelling...\n") except SystemExit: options.progress = False finally: sys.stdout.flush() for thread in threads: thread.finish_event.set() for thread in threads: thread.join() complete = True if not self.any_error(): self.post_actions() dst_fs.close() if self.action_errors: for error in self.action_errors: self.error(self.wrap_error(unicode(error)) + '\n') sys.stdout.flush() else: if complete and options.progress: sys.stdout.write(self.progress_bar(self.total_files, self.done_files, '')) sys.stdout.write('\n') sys.stdout.flush() def post_actions(self): pass def on_done(self, path_type, src_fs, src_path, dst_fs, dst_path): self.lock.acquire() try: if self.options.verbose: if path_type == self.DIR: print "mkdir %s" % dst_fs.desc(dst_path) else: print "%s -> %s" % (src_fs.desc(src_path), dst_fs.desc(dst_path)) elif self.options.progress: self.done_files += 1 sys.stdout.write(self.progress_bar(self.total_files, self.done_files, self.get_verb())) sys.stdout.flush() finally: self.lock.release() def on_error(self, e): self.lock.acquire() try: self.action_errors.append(e) finally: self.lock.release() def any_error(self): self.lock.acquire() try: return bool(self.action_errors) finally: self.lock.release() def progress_bar(self, total, remaining, msg=''): bar_width = 20 throbber = '|/-\\' throb = throbber[remaining % len(throbber)] done = float(remaining) / total done_steps = int(done * bar_width) bar_steps = ('#' * done_steps).ljust(bar_width) msg = '%s %i%%' % (msg, int(done * 100.0)) msg = msg.ljust(20) if total == remaining: throb = '' bar = '\r%s[%s] %s\r' % (throb, bar_steps, msg.lstrip()) return bar def run(): return FScp().run() if __name__ == "__main__": sys.exit(run()) fs-0.5.4/fs/commands/fsmount.py0000664000175000017500000001037012512525115016350 0ustar willwill00000000000000#!/usr/bin/env python from fs.commands.runner import Command import sys import platform import os import os.path platform = platform.system() class FSMount(Command): if platform == "Windows": usage = """fsmount [OPTIONS]... [FS] [DRIVE LETTER] or fsmount -u [DRIVER LETTER] Mounts a filesystem on a drive letter""" else: usage = """fsmount [OPTIONS]... [FS] [SYSTEM PATH] or fsmount -u [SYSTEM PATH] Mounts a file system on a system path""" version = "1.0" def get_optparse(self): optparse = super(FSMount, self).get_optparse() optparse.add_option('-f', '--foreground', dest='foreground', action="store_true", default=False, help="run the mount process in the foreground", metavar="FOREGROUND") optparse.add_option('-u', '--unmount', dest='unmount', action="store_true", default=False, help="unmount path", metavar="UNMOUNT") optparse.add_option('-n', '--nocache', dest='nocache', action="store_true", default=False, help="do not cache network filesystems", metavar="NOCACHE") return optparse def do_run(self, options, args): windows = platform == "Windows" if options.unmount: if windows: try: mount_path = args[0][:1] except IndexError: self.error('Driver letter required\n') return 1 from fs.expose import dokan mount_path = mount_path[:1].upper() self.output('unmounting %s:...\n' % mount_path, True) dokan.unmount(mount_path) return else: try: mount_path = args[0] except IndexError: self.error(self.usage + '\n') return 1 from fs.expose import fuse self.output('unmounting %s...\n' % mount_path, True) fuse.unmount(mount_path) return try: fs_url = args[0] except IndexError: self.error(self.usage + '\n') return 1 try: mount_path = args[1] except IndexError: if windows: mount_path = mount_path[:1].upper() self.error(self.usage + '\n') else: self.error(self.usage + '\n') return 1 fs, path = self.open_fs(fs_url, create_dir=True) if path: if not fs.isdir(path): self.error('%s is not a directory on %s' % (fs_url, fs)) return 1 fs = fs.opendir(path) path = '/' if not options.nocache: fs.cache_hint(True) if windows: from fs.expose import dokan if len(mount_path) > 1: self.error('Driver letter should be one character') return 1 self.output("Mounting %s on %s:\n" % (fs, mount_path), True) flags = dokan.DOKAN_OPTION_REMOVABLE if options.debug: flags |= dokan.DOKAN_OPTION_DEBUG | dokan.DOKAN_OPTION_STDERR mp = dokan.mount(fs, mount_path, numthreads=5, foreground=options.foreground, flags=flags, volname=str(fs)) else: if not os.path.exists(mount_path): try: os.makedirs(mount_path) except: pass from fs.expose import fuse self.output("Mounting %s on %s\n" % (fs, mount_path), True) if options.foreground: fuse_process = fuse.mount(fs, mount_path, foreground=True) else: if not os.fork(): mp = fuse.mount(fs, mount_path, foreground=True) else: fs.close = lambda: None def run(): return FSMount().run() if __name__ == "__main__": sys.exit(run()) fs-0.5.4/fs/commands/fsmkdir.py0000664000175000017500000000071112512525115016312 0ustar willwill00000000000000#!/usr/bin/env python from fs.commands.runner import Command import sys class FSMkdir(Command): usage = """fsmkdir [PATH] Make a directory""" version = "1.0" def do_run(self, options, args): for fs_url in args: self.open_fs(fs_url, create_dir=True) def run(): return FSMkdir().run() if __name__ == "__main__": sys.exit(run()) fs-0.5.4/fs/commands/fsmv.py0000664000175000017500000000135612512525115015634 0ustar willwill00000000000000#!/usr/bin/env python from fs.utils import movefile, movefile_non_atomic, contains_files from fs.commands import fscp import sys class FSmv(fscp.FScp): usage = """fsmv [OPTION]... [SOURCE] [DESTINATION] Move files from SOURCE to DESTINATION""" def get_verb(self): return 'moving...' def get_action(self): if self.options.threads > 1: return movefile_non_atomic else: return movefile def post_actions(self): for fs, dirpath in self.root_dirs: if not contains_files(fs, dirpath): fs.removedir(dirpath, force=True) def run(): return FSmv().run() if __name__ == "__main__": sys.exit(run()) fs-0.5.4/fs/commands/fsinfo.py0000664000175000017500000000645312512525115016150 0ustar willwill00000000000000#!/usr/bin/env python from fs.commands.runner import Command import sys from datetime import datetime class FSInfo(Command): usage = """fsinfo [OPTION]... [PATH] Display information regarding an FS resource""" def get_optparse(self): optparse = super(FSInfo, self).get_optparse() optparse.add_option('-k', '--key', dest='keys', action='append', default=[], help='display KEYS only') optparse.add_option('-s', '--simple', dest='simple', action='store_true', default=False, help='info displayed in simple format (no table)') optparse.add_option('-o', '--omit', dest='omit', action='store_true', default=False, help='omit path name from output') optparse.add_option('-d', '--dirsonly', dest='dirsonly', action="store_true", default=False, help="list directories only", metavar="DIRSONLY") optparse.add_option('-f', '--filesonly', dest='filesonly', action="store_true", default=False, help="list files only", metavar="FILESONLY") return optparse def do_run(self, options, args): def wrap_value(val): if val.rstrip() == '\0': return self.wrap_error('... missing ...') return val def make_printable(text): if not isinstance(text, basestring): try: text = str(text) except: try: text = unicode(text) except: text = repr(text) return text keys = options.keys or None for fs, path, is_dir in self.get_resources(args, files_only=options.filesonly, dirs_only=options.dirsonly): if not options.omit: if options.simple: file_line = u'%s\n' % self.wrap_filename(path) else: file_line = u'[%s] %s\n' % (self.wrap_filename(path), self.wrap_faded(fs.desc(path))) self.output(file_line) info = fs.getinfo(path) for k, v in info.items(): if k.startswith('_'): del info[k] elif not isinstance(v, (basestring, int, long, float, bool, datetime)): del info[k] if keys: table = [(k, make_printable(info.get(k, '\0'))) for k in keys] else: keys = sorted(info.keys()) table = [(k, make_printable(info[k])) for k in sorted(info.keys())] if options.simple: for row in table: self.output(row[-1] + '\n') else: self.output_table(table, {0:self.wrap_table_header, 1:wrap_value}) def run(): return FSInfo().run() if __name__ == "__main__": sys.exit(run()) fs-0.5.4/fs/commands/fsserve.py0000664000175000017500000000663512512525115016343 0ustar willwill00000000000000#!/usr/bin/env python import sys from fs.opener import opener from fs.commands.runner import Command from fs.utils import print_fs import errno class FSServe(Command): usage = """fsserve [OPTION]... [PATH] Serves the contents of PATH with one of a number of methods""" def get_optparse(self): optparse = super(FSServe, self).get_optparse() optparse.add_option('-t', '--type', dest='type', type="string", default="http", help="Server type to create (http, rpc, sftp)", metavar="TYPE") optparse.add_option('-a', '--addr', dest='addr', type="string", default="127.0.0.1", help="Server address", metavar="ADDR") optparse.add_option('-p', '--port', dest='port', type="int", help="Port number", metavar="") return optparse def do_run(self, options, args): try: fs_url = args[0] except IndexError: fs_url = './' fs, path = self.open_fs(fs_url) if fs.isdir(path): fs = fs.opendir(path) path = '/' self.output("Opened %s\n" % fs, verbose=True) port = options.port try: if options.type == 'http': from fs.expose.http import serve_fs if port is None: port = 80 self.output("Starting http server on %s:%i\n" % (options.addr, port), verbose=True) serve_fs(fs, options.addr, port) elif options.type == 'rpc': from fs.expose.xmlrpc import RPCFSServer if port is None: port = 80 s = RPCFSServer(fs, (options.addr, port)) self.output("Starting rpc server on %s:%i\n" % (options.addr, port), verbose=True) s.serve_forever() elif options.type == 'ftp': from fs.expose.ftp import serve_fs if port is None: port = 21 self.output("Starting ftp server on %s:%i\n" % (options.addr, port), verbose=True) serve_fs(fs, options.addr, port) elif options.type == 'sftp': from fs.expose.sftp import BaseSFTPServer import logging log = logging.getLogger('paramiko') if options.debug: log.setLevel(logging.DEBUG) elif options.verbose: log.setLevel(logging.INFO) ch = logging.StreamHandler() ch.setLevel(logging.DEBUG) log.addHandler(ch) if port is None: port = 22 server = BaseSFTPServer((options.addr, port), fs) try: self.output("Starting sftp server on %s:%i\n" % (options.addr, port), verbose=True) server.serve_forever() except Exception, e: pass finally: server.server_close() else: self.error("Server type '%s' not recognised\n" % options.type) except IOError, e: if e.errno == errno.EACCES: self.error('Permission denied\n') return 1 else: self.error(str(e) + '\n') return 1 def run(): return FSServe().run() if __name__ == "__main__": sys.exit(run()) fs-0.5.4/fs/commands/runner.py0000664000175000017500000003042012512525115016164 0ustar willwill00000000000000import warnings warnings.filterwarnings("ignore") from fs.opener import opener, OpenerError, Opener from fs.errors import FSError from fs.path import splitext, pathsplit, isdotfile, iswildcard import re import sys import platform import six from optparse import OptionParser from collections import defaultdict if platform.system() == 'Windows': def getTerminalSize(): try: ## {{{ http://code.activestate.com/recipes/440694/ (r3) from ctypes import windll, create_string_buffer # stdin handle is -10 # stdout handle is -11 # stderr handle is -12 h = windll.kernel32.GetStdHandle(-12) csbi = create_string_buffer(22) res = windll.kernel32.GetConsoleScreenBufferInfo(h, csbi) if res: import struct (bufx, bufy, curx, cury, wattr, left, top, right, bottom, maxx, maxy) = struct.unpack("hhhhHhhhhhh", csbi.raw) sizex = right - left + 1 sizey = bottom - top + 1 else: sizex, sizey = 80, 25 # can't determine actual size - return default values return sizex, sizey except: return 80, 25 else: def getTerminalSize(): def ioctl_GWINSZ(fd): try: import fcntl, termios, struct, os cr = struct.unpack('hh', fcntl.ioctl(fd, termios.TIOCGWINSZ, '1234')) except: return None return cr cr = ioctl_GWINSZ(0) or ioctl_GWINSZ(1) or ioctl_GWINSZ(2) if not cr: import os try: fd = os.open(os.ctermid(), os.O_RDONLY) cr = ioctl_GWINSZ(fd) os.close(fd) except: pass if cr: return int(cr[1]), int(cr[0]) try: h, w = os.popen("stty size", "r").read().split() return int(w), int(h) except: pass return 80, 25 def _unicode(text): if not isinstance(text, unicode): return text.decode('ascii', 'replace') return text class Command(object): usage = '' version = '' def __init__(self, usage='', version=''): if six.PY3: self.output_file = sys.stdout.buffer self.error_file = sys.stderr.buffer else: self.output_file = sys.stdout self.error_file = sys.stderr self.encoding = getattr(self.output_file, 'encoding', 'utf-8') or 'utf-8' self.verbosity_level = 0 self.terminal_colors = not sys.platform.startswith('win') and self.is_terminal() if self.is_terminal(): w, _h = getTerminalSize() self.terminal_width = w else: self.terminal_width = 80 self.name = self.__class__.__name__.lower() def is_terminal(self): try: return self.output_file.isatty() except AttributeError: return False def wrap_dirname(self, dirname): if not self.terminal_colors: return dirname return '\x1b[1;34m%s\x1b[0m' % dirname def wrap_error(self, msg): if not self.terminal_colors: return msg return '\x1b[31m%s\x1b[0m' % msg def wrap_filename(self, fname): fname = _unicode(fname) if not self.terminal_colors: return fname if '://' in fname: return fname # if '.' in fname: # name, ext = splitext(fname) # fname = u'%s\x1b[36m%s\x1b[0m' % (name, ext) if isdotfile(fname): fname = '\x1b[33m%s\x1b[0m' % fname return fname def wrap_faded(self, text): text = _unicode(text) if not self.terminal_colors: return text return u'\x1b[2m%s\x1b[0m' % text def wrap_link(self, text): if not self.terminal_colors: return text return u'\x1b[1;33m%s\x1b[0m' % text def wrap_strong(self, text): if not self.terminal_colors: return text return u'\x1b[1m%s\x1b[0m' % text def wrap_table_header(self, name): if not self.terminal_colors: return name return '\x1b[1;32m%s\x1b[0m' % name def highlight_fsurls(self, text): if not self.terminal_colors: return text re_fs = r'(\S*?://\S*)' def repl(matchobj): fs_url = matchobj.group(0) return self.wrap_link(fs_url) return re.sub(re_fs, repl, text) def open_fs(self, fs_url, writeable=False, create_dir=False): fs, path = opener.parse(fs_url, writeable=writeable, create_dir=create_dir) fs.cache_hint(True) return fs, path def expand_wildcard(self, fs, path): if path is None: return [], [] pathname, resourcename = pathsplit(path) if iswildcard(resourcename): dir_paths = fs.listdir(pathname, wildcard=resourcename, absolute=True, dirs_only=True) file_paths = fs.listdir(pathname, wildcard=resourcename, absolute=True, files_only=True) return dir_paths, file_paths else: if fs.isdir(path): #file_paths = fs.listdir(path, # absolute=True) return [path], [] return [], [path] def get_resources(self, fs_urls, dirs_only=False, files_only=False, single=False): fs_paths = [self.open_fs(fs_url) for fs_url in fs_urls] resources = [] for fs, path in fs_paths: if path and iswildcard(path): if not files_only: dir_paths = fs.listdir(wildcard=path, dirs_only=True) for path in dir_paths: resources.append([fs, path, True]) if not dirs_only: file_paths = fs.listdir(wildcard=path, files_only=True) for path in file_paths: resources.append([fs, path, False]) else: path = path or '/' is_dir = fs.isdir(path) resource = [fs, path, is_dir] if not files_only and not dirs_only: resources.append(resource) elif files_only and not is_dir: resources.append(resource) elif dirs_only and is_dir: resources.append(resource) if single: break return resources def ask(self, msg): return raw_input('%s: %s ' % (self.name, msg)) def text_encode(self, text): if not isinstance(text, unicode): text = text.decode('ascii', 'replace') text = text.encode(self.encoding, 'replace') return text def output(self, msgs, verbose=False): if verbose and not self.options.verbose: return if isinstance(msgs, basestring): msgs = (msgs,) for msg in msgs: self.output_file.write(self.text_encode(msg)) def output_table(self, table, col_process=None, verbose=False): if verbose and not self.verbose: return if col_process is None: col_process = {} max_row_widths = defaultdict(int) for row in table: for col_no, col in enumerate(row): max_row_widths[col_no] = max(max_row_widths[col_no], len(col)) lines = [] for row in table: out_col = [] for col_no, col in enumerate(row): td = col.ljust(max_row_widths[col_no]) if col_no in col_process: td = col_process[col_no](td) out_col.append(td) lines.append(self.text_encode('%s\n' % ' '.join(out_col).rstrip())) for l in lines: self.output_file.write(l) #self.output(''.join(lines)) def error(self, *msgs): for msg in msgs: self.error_file.write("{}: {}".format(self.name, msg).encode(self.encoding)) def get_optparse(self): optparse = OptionParser(usage=self.usage, version=self.version) optparse.add_option('--debug', dest='debug', action="store_true", default=False, help="Show debug information", metavar="DEBUG") optparse.add_option('-v', '--verbose', dest='verbose', action="store_true", default=False, help="make output verbose", metavar="VERBOSE") optparse.add_option('--listopeners', dest='listopeners', action="store_true", default=False, help="list all FS openers", metavar="LISTOPENERS") optparse.add_option('--fs', dest='fs', action='append', type="string", help="import an FS opener e.g --fs foo.bar.MyOpener", metavar="OPENER") return optparse def list_openers(self): opener_table = [] for fs_opener in opener.openers.itervalues(): names = fs_opener.names desc = getattr(fs_opener, 'desc', '') opener_table.append((names, desc)) opener_table.sort(key=lambda r: r[0]) def wrap_line(text): lines = text.split('\n') for line in lines: words = [] line_len = 0 for word in line.split(): if word == '*': word = ' *' if line_len + len(word) > self.terminal_width: self.output((self.highlight_fsurls(' '.join(words)), '\n')) del words[:] line_len = 0 words.append(word) line_len += len(word) + 1 if words: self.output(self.highlight_fsurls(' '.join(words))) self.output('\n') for names, desc in opener_table: self.output(('-' * self.terminal_width, '\n')) proto = ', '.join([n + '://' for n in names]) self.output((self.wrap_dirname('[%s]' % proto), '\n\n')) if not desc.strip(): desc = "No information available" wrap_line(desc) self.output('\n') def run(self): parser = self.get_optparse() options, args = parser.parse_args() self.options = options if options.listopeners: self.list_openers() return 0 ilocals = {} if options.fs: for import_opener in options.fs: module_name, opener_class = import_opener.rsplit('.', 1) try: opener_module = __import__(module_name, globals(), ilocals, [opener_class], -1) except ImportError: self.error("Unable to import opener %s\n" % import_opener) return 0 new_opener = getattr(opener_module, opener_class) try: if not issubclass(new_opener, Opener): self.error('%s is not an fs.opener.Opener\n' % import_opener) return 0 except TypeError: self.error('%s is not an opener class\n' % import_opener) return 0 if options.verbose: self.output('Imported opener %s\n' % import_opener) opener.add(new_opener) if not six.PY3: args = [unicode(arg, sys.getfilesystemencoding()) for arg in args] self.verbose = options.verbose try: return self.do_run(options, args) or 0 except FSError, e: self.error(self.wrap_error(unicode(e)) + '\n') if options.debug: raise return 1 except KeyboardInterrupt: if self.is_terminal(): self.output("\n") return 0 except SystemExit: return 0 except Exception, e: self.error(self.wrap_error('Error - %s\n' % unicode(e))) if options.debug: raise return 1 if __name__ == "__main__": command = Command() sys.exit(command.run())fs-0.5.4/fs/commands/__init__.py0000664000175000017500000000000012512525115016401 0ustar willwill00000000000000fs-0.5.4/fs/local_functools.py0000664000175000017500000000057712512525115016252 0ustar willwill00000000000000""" A version of functools.wraps for Python versions that don't support it. Note that this module can't be named "functools" because it would shadow the stdlib module that it tries to emulate. Absolute imports would fix this problem but are only availabe from Python 2.5. """ try: from functools import wraps as wraps except ImportError: wraps = lambda f: lambda f: f fs-0.5.4/fs/compatibility.py0000664000175000017500000000270712512525115015732 0ustar willwill00000000000000""" Some functions for Python3 compatibility. Not for general usage, the functionality in this file is exposed elsewhere """ import six from six import PY3 def copy_file_to_fs(data, dst_fs, dst_path, chunk_size=64 * 1024, progress_callback=None, finished_callback=None): """Copy data from a string or a file-like object to a given fs/path""" if progress_callback is None: progress_callback = lambda bytes_written: None bytes_written = 0 f = None try: progress_callback(bytes_written) if hasattr(data, "read"): read = data.read chunk = read(chunk_size) if isinstance(chunk, six.text_type): f = dst_fs.open(dst_path, 'w') else: f = dst_fs.open(dst_path, 'wb') write = f.write while chunk: write(chunk) bytes_written += len(chunk) progress_callback(bytes_written) chunk = read(chunk_size) else: if isinstance(data, six.text_type): f = dst_fs.open(dst_path, 'w') else: f = dst_fs.open(dst_path, 'wb') f.write(data) bytes_written += len(data) progress_callback(bytes_written) if hasattr(f, 'flush'): f.flush() if finished_callback is not None: finished_callback() finally: if f is not None: f.close() fs-0.5.4/fs/remotefs.py0000664000175000017500000001336712512525115014711 0ustar willwill00000000000000# Work in Progress - Do not use from __future__ import with_statement from fs.base import FS from fs.expose.serve import packetstream from collections import defaultdict import threading from threading import Lock, RLock from json import dumps import Queue as queue import socket from six import b class PacketHandler(threading.Thread): def __init__(self, transport, prelude_callback=None): super(PacketHandler, self).__init__() self.transport = transport self.encoder = packetstream.JSONFileEncoder(transport) self.decoder = packetstream.JSONDecoder(prelude_callback=None) self.queues = defaultdict(queue.Queue) self._encoder_lock = threading.Lock() self._queues_lock = threading.Lock() self._call_id_lock = threading.Lock() self.call_id = 0 def run(self): decoder = self.decoder read = self.transport.read on_packet = self.on_packet while True: data = read(1024*16) if not data: print "No data" break print "data", repr(data) for header, payload in decoder.feed(data): print repr(header) print repr(payload) on_packet(header, payload) def _new_call_id(self): with self._call_id_lock: self.call_id += 1 return self.call_id def get_thread_queue(self, queue_id=None): if queue_id is None: queue_id = threading.current_thread().ident with self._queues_lock: return self.queues[queue_id] def send_packet(self, header, payload=''): call_id = self._new_call_id() queue_id = threading.current_thread().ident client_ref = "%i:%i" % (queue_id, call_id) header['client_ref'] = client_ref with self._encoder_lock: self.encoder.write(header, payload) return call_id def get_packet(self, call_id): if call_id is not None: queue_id = threading.current_thread().ident client_ref = "%i:%i" % (queue_id, call_id) else: client_ref = None queue = self.get_thread_queue() while True: header, payload = queue.get() print repr(header) print repr(payload) if client_ref is not None and header.get('client_ref') != client_ref: continue break return header, payload def on_packet(self, header, payload): client_ref = header.get('client_ref', '') queue_id, call_id = client_ref.split(':', 1) queue_id = int(queue_id) #queue_id = header.get('queue_id', '') queue = self.get_thread_queue(queue_id) queue.put((header, payload)) class _SocketFile(object): def __init__(self, socket): self.socket = socket def read(self, size): try: return self.socket.recv(size) except: return b('') def write(self, data): self.socket.sendall(data) def close(self): self.socket.shutdown(socket.SHUT_RDWR) self.socket.close() class _RemoteFile(object): def __init__(self, path, connection): self.path = path self.connection = connection class RemoteFS(FS): _meta = { 'thead_safe' : True, 'network' : True, 'virtual' : False, 'read_only' : False, 'unicode_paths' : True, } def __init__(self, addr='', port=3000, username=None, password=None, resource=None, transport=None): self.addr = addr self.port = port self.username = None self.password = None self.resource = None self.transport = transport if self.transport is None: self.transport = self._open_connection() self.packet_handler = PacketHandler(self.transport) self.packet_handler.start() self._remote_call('auth', username=username, password=password, resource=resource) def _open_connection(self): sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.connect((self.addr, self.port)) socket_file = _SocketFile(sock) socket_file.write(b('pyfs/0.1\n')) return socket_file def _make_call(self, method_name, *args, **kwargs): call = dict(type='rpc', method=method_name, args=args, kwargs=kwargs) return call def _remote_call(self, method_name, *args, **kwargs): call = self._make_call(method_name, *args, **kwargs) call_id = self.packet_handler.send_packet(call) header, payload = self.packet_handler.get_packet(call_id) return header, payload def ping(self, msg): call_id = self.packet_handler.send_packet({'type':'rpc', 'method':'ping'}, msg) header, payload = self.packet_handler.get_packet(call_id) print "PING" print header print payload def close(self): self.transport.close() self.packet_handler.join() def open(self, path, mode="r", **kwargs): pass def exists(self, path): remote = self._remote_call('exists', path) return remote.get('response') if __name__ == "__main__": rfs = RemoteFS() rfs.close() fs-0.5.4/fs/appdirfs.py0000664000175000017500000001007512512525115014666 0ustar willwill00000000000000""" fs.appdirfs =========== A collection of filesystems that map to application specific locations. These classes abstract away the different requirements for user data across platforms, which vary in their conventions. They are all subclasses of :class:`fs.osfs.OSFS`, all that differs from `OSFS` is the constructor which detects the appropriate location given the name of the application, author name and other parameters. Uses `appdirs` (https://github.com/ActiveState/appdirs), written by Trent Mick and Sridhar Ratnakumar """ from fs.osfs import OSFS from fs.appdirs import AppDirs __all__ = ['UserDataFS', 'SiteDataFS', 'UserCacheFS', 'UserLogFS'] class UserDataFS(OSFS): """A filesystem for per-user application data.""" def __init__(self, appname, appauthor=None, version=None, roaming=False, create=True): """ :param appname: the name of the application :param appauthor: the name of the author (used on Windows) :param version: optional version string, if a unique location per version of the application is required :param roaming: if True, use a *roaming* profile on Windows, see http://technet.microsoft.com/en-us/library/cc766489(WS.10).aspx :param create: if True (the default) the directory will be created if it does not exist """ app_dirs = AppDirs(appname, appauthor, version, roaming) super(UserDataFS, self).__init__(app_dirs.user_data_dir, create=create) class SiteDataFS(OSFS): """A filesystem for application site data.""" def __init__(self, appname, appauthor=None, version=None, roaming=False, create=True): """ :param appname: the name of the application :param appauthor: the name of the author (not used on linux) :param version: optional version string, if a unique location per version of the application is required :param roaming: if True, use a *roaming* profile on Windows, see http://technet.microsoft.com/en-us/library/cc766489(WS.10).aspx :param create: if True (the default) the directory will be created if it does not exist """ app_dirs = AppDirs(appname, appauthor, version, roaming) super(SiteDataFS, self).__init__(app_dirs.site_data_dir, create=create) class UserCacheFS(OSFS): """A filesystem for per-user application cache data.""" def __init__(self, appname, appauthor=None, version=None, roaming=False, create=True): """ :param appname: the name of the application :param appauthor: the name of the author (not used on linux) :param version: optional version string, if a unique location per version of the application is required :param roaming: if True, use a *roaming* profile on Windows, see http://technet.microsoft.com/en-us/library/cc766489(WS.10).aspx :param create: if True (the default) the directory will be created if it does not exist """ app_dirs = AppDirs(appname, appauthor, version, roaming) super(UserCacheFS, self).__init__(app_dirs.user_cache_dir, create=create) class UserLogFS(OSFS): """A filesystem for per-user application log data.""" def __init__(self, appname, appauthor=None, version=None, roaming=False, create=True): """ :param appname: the name of the application :param appauthor: the name of the author (not used on linux) :param version: optional version string, if a unique location per version of the application is required :param roaming: if True, use a *roaming* profile on Windows, see http://technet.microsoft.com/en-us/library/cc766489(WS.10).aspx :param create: if True (the default) the directory will be created if it does not exist """ app_dirs = AppDirs(appname, appauthor, version, roaming) super(UserLogFS, self).__init__(app_dirs.user_log_dir, create=create) if __name__ == "__main__": udfs = UserDataFS('exampleapp', appauthor='pyfs') print udfs udfs2 = UserDataFS('exampleapp2', appauthor='pyfs', create=False) print udfs2 fs-0.5.4/fs/appdirs.py0000664000175000017500000003271612512525115014526 0ustar willwill00000000000000#!/usr/bin/env python # Copyright (c) 2005-2010 ActiveState Software Inc. """Utilities for determining application-specific dirs. See for details and usage. """ # Dev Notes: # - MSDN on where to store app data files: # http://support.microsoft.com/default.aspx?scid=kb;en-us;310294#XSLTH3194121123120121120120 # - Mac OS X: http://developer.apple.com/documentation/MacOSX/Conceptual/BPFileSystem/index.html # - XDG spec for Un*x: http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html __version_info__ = (1, 2, 0) __version__ = '.'.join(map(str, __version_info__)) import sys import os PY3 = sys.version_info[0] == 3 if PY3: unicode = str class AppDirsError(Exception): pass def user_data_dir(appname, appauthor=None, version=None, roaming=False): r"""Return full path to the user-specific data dir for this application. "appname" is the name of application. "appauthor" (only required and used on Windows) is the name of the appauthor or distributing body for this application. Typically it is the owning company name. "version" is an optional version path element to append to the path. You might want to use this if you want multiple versions of your app to be able to run independently. If used, this would typically be ".". "roaming" (boolean, default False) can be set True to use the Windows roaming appdata directory. That means that for users on a Windows network setup for roaming profiles, this user data will be sync'd on login. See for a discussion of issues. Typical user data directories are: Mac OS X: ~/Library/Application Support/ Unix: ~/.config/ # or in $XDG_CONFIG_HOME if defined Win XP (not roaming): C:\Documents and Settings\\Application Data\\ Win XP (roaming): C:\Documents and Settings\\Local Settings\Application Data\\ Win 7 (not roaming): C:\Users\\AppData\Local\\ Win 7 (roaming): C:\Users\\AppData\Roaming\\ For Unix, we follow the XDG spec and support $XDG_CONFIG_HOME. We don't use $XDG_DATA_HOME as that data dir is mostly used at the time of installation, instead of the application adding data during runtime. Also, in practice, Linux apps tend to store their data in "~/.config/" instead of "~/.local/share/". """ if sys.platform.startswith("win"): if appauthor is None: raise AppDirsError("must specify 'appauthor' on Windows") const = roaming and "CSIDL_APPDATA" or "CSIDL_LOCAL_APPDATA" path = os.path.join(_get_win_folder(const), appauthor, appname) elif sys.platform == 'darwin': path = os.path.join( os.path.expanduser('~/Library/Application Support/'), appname) else: path = os.path.join( os.getenv('XDG_CONFIG_HOME', os.path.expanduser("~/.config")), appname.lower()) if version: path = os.path.join(path, version) return path def site_data_dir(appname, appauthor=None, version=None): """Return full path to the user-shared data dir for this application. "appname" is the name of application. "appauthor" (only required and used on Windows) is the name of the appauthor or distributing body for this application. Typically it is the owning company name. "version" is an optional version path element to append to the path. You might want to use this if you want multiple versions of your app to be able to run independently. If used, this would typically be ".". Typical user data directories are: Mac OS X: /Library/Application Support/ Unix: /etc/xdg/ Win XP: C:\Documents and Settings\All Users\Application Data\\ Vista: (Fail! "C:\ProgramData" is a hidden *system* directory on Vista.) Win 7: C:\ProgramData\\ # Hidden, but writeable on Win 7. For Unix, this is using the $XDG_CONFIG_DIRS[0] default. WARNING: Do not use this on Windows. See the Vista-Fail note above for why. """ if sys.platform.startswith("win"): if appauthor is None: raise AppDirsError("must specify 'appauthor' on Windows") path = os.path.join(_get_win_folder("CSIDL_COMMON_APPDATA"), appauthor, appname) elif sys.platform == 'darwin': path = os.path.join( os.path.expanduser('/Library/Application Support'), appname) else: # XDG default for $XDG_CONFIG_DIRS[0]. Perhaps should actually # *use* that envvar, if defined. path = "/etc/xdg/"+appname.lower() if version: path = os.path.join(path, version) return path def user_cache_dir(appname, appauthor=None, version=None, opinion=True): r"""Return full path to the user-specific cache dir for this application. "appname" is the name of application. "appauthor" (only required and used on Windows) is the name of the appauthor or distributing body for this application. Typically it is the owning company name. "version" is an optional version path element to append to the path. You might want to use this if you want multiple versions of your app to be able to run independently. If used, this would typically be ".". "opinion" (boolean) can be False to disable the appending of "Cache" to the base app data dir for Windows. See discussion below. Typical user cache directories are: Mac OS X: ~/Library/Caches/ Unix: ~/.cache/ (XDG default) Win XP: C:\Documents and Settings\\Local Settings\Application Data\\\Cache Vista: C:\Users\\AppData\Local\\\Cache On Windows the only suggestion in the MSDN docs is that local settings go in the `CSIDL_LOCAL_APPDATA` directory. This is identical to the non-roaming app data dir (the default returned by `user_data_dir` above). Apps typically put cache data somewhere *under* the given dir here. Some examples: ...\Mozilla\Firefox\Profiles\\Cache ...\Acme\SuperApp\Cache\1.0 OPINION: This function appends "Cache" to the `CSIDL_LOCAL_APPDATA` value. This can be disabled with the `opinion=False` option. """ if sys.platform.startswith("win"): if appauthor is None: raise AppDirsError("must specify 'appauthor' on Windows") path = os.path.join(_get_win_folder("CSIDL_LOCAL_APPDATA"), appauthor, appname) if opinion: path = os.path.join(path, "Cache") elif sys.platform == 'darwin': path = os.path.join( os.path.expanduser('~/Library/Caches'), appname) else: path = os.path.join( os.getenv('XDG_CACHE_HOME', os.path.expanduser('~/.cache')), appname.lower()) if version: path = os.path.join(path, version) return path def user_log_dir(appname, appauthor=None, version=None, opinion=True): r"""Return full path to the user-specific log dir for this application. "appname" is the name of application. "appauthor" (only required and used on Windows) is the name of the appauthor or distributing body for this application. Typically it is the owning company name. "version" is an optional version path element to append to the path. You might want to use this if you want multiple versions of your app to be able to run independently. If used, this would typically be ".". "opinion" (boolean) can be False to disable the appending of "Logs" to the base app data dir for Windows, and "log" to the base cache dir for Unix. See discussion below. Typical user cache directories are: Mac OS X: ~/Library/Logs/ Unix: ~/.cache//log # or under $XDG_CACHE_HOME if defined Win XP: C:\Documents and Settings\\Local Settings\Application Data\\\Logs Vista: C:\Users\\AppData\Local\\\Logs On Windows the only suggestion in the MSDN docs is that local settings go in the `CSIDL_LOCAL_APPDATA` directory. (Note: I'm interested in examples of what some windows apps use for a logs dir.) OPINION: This function appends "Logs" to the `CSIDL_LOCAL_APPDATA` value for Windows and appends "log" to the user cache dir for Unix. This can be disabled with the `opinion=False` option. """ if sys.platform == "darwin": path = os.path.join( os.path.expanduser('~/Library/Logs'), appname) elif sys.platform == "win32": path = user_data_dir(appname, appauthor, version); version=False if opinion: path = os.path.join(path, "Logs") else: path = user_cache_dir(appname, appauthor, version); version=False if opinion: path = os.path.join(path, "log") if version: path = os.path.join(path, version) return path class AppDirs(object): """Convenience wrapper for getting application dirs.""" def __init__(self, appname, appauthor, version=None, roaming=False): self.appname = appname self.appauthor = appauthor self.version = version self.roaming = roaming @property def user_data_dir(self): return user_data_dir(self.appname, self.appauthor, version=self.version, roaming=self.roaming) @property def site_data_dir(self): return site_data_dir(self.appname, self.appauthor, version=self.version) @property def user_cache_dir(self): return user_cache_dir(self.appname, self.appauthor, version=self.version) @property def user_log_dir(self): return user_log_dir(self.appname, self.appauthor, version=self.version) #---- internal support stuff def _get_win_folder_from_registry(csidl_name): """This is a fallback technique at best. I'm not sure if using the registry for this guarantees us the correct answer for all CSIDL_* names. """ import _winreg shell_folder_name = { "CSIDL_APPDATA": "AppData", "CSIDL_COMMON_APPDATA": "Common AppData", "CSIDL_LOCAL_APPDATA": "Local AppData", }[csidl_name] key = _winreg.OpenKey(_winreg.HKEY_CURRENT_USER, r"Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders") dir, type = _winreg.QueryValueEx(key, shell_folder_name) return dir def _get_win_folder_with_pywin32(csidl_name): from win32com.shell import shellcon, shell dir = shell.SHGetFolderPath(0, getattr(shellcon, csidl_name), 0, 0) # Try to make this a unicode path because SHGetFolderPath does # not return unicode strings when there is unicode data in the # path. try: dir = unicode(dir) # Downgrade to short path name if have highbit chars. See # . has_high_char = False for c in dir: if ord(c) > 255: has_high_char = True break if has_high_char: try: import win32api dir = win32api.GetShortPathName(dir) except ImportError: pass except UnicodeError: pass return dir def _get_win_folder_with_ctypes(csidl_name): import ctypes csidl_const = { "CSIDL_APPDATA": 26, "CSIDL_COMMON_APPDATA": 35, "CSIDL_LOCAL_APPDATA": 28, }[csidl_name] buf = ctypes.create_unicode_buffer(1024) ctypes.windll.shell32.SHGetFolderPathW(None, csidl_const, None, 0, buf) # Downgrade to short path name if have highbit chars. See # . has_high_char = False for c in buf: if ord(c) > 255: has_high_char = True break if has_high_char: buf2 = ctypes.create_unicode_buffer(1024) if ctypes.windll.kernel32.GetShortPathNameW(buf.value, buf2, 1024): buf = buf2 return buf.value if sys.platform == "win32": try: import win32com.shell _get_win_folder = _get_win_folder_with_pywin32 except ImportError: try: _get_win_folder = _get_win_folder_with_ctypes except ImportError: _get_win_folder = _get_win_folder_from_registry #---- self test code if __name__ == "__main__": appname = "MyApp" appauthor = "MyCompany" props = ("user_data_dir", "site_data_dir", "user_cache_dir", "user_log_dir") print("-- app dirs (without optional 'version')") dirs = AppDirs(appname, appauthor, version="1.0") for prop in props: print("%s: %s" % (prop, getattr(dirs, prop))) print("\n-- app dirs (with optional 'version')") dirs = AppDirs(appname, appauthor) for prop in props: print("%s: %s" % (prop, getattr(dirs, prop))) fs-0.5.4/fs/base.py0000664000175000017500000014772512567364306014022 0ustar willwill00000000000000#!/usr/bin/env python """ fs.base ======= This module defines the most basic filesystem abstraction, the FS class. Instances of FS represent a filesystem containing files and directories that can be queried and manipulated. To implement a new kind of filesystem, start by sublcassing the base FS class. For more information regarding implementing a working PyFilesystem interface, see :ref:`implementers`. """ from __future__ import with_statement __all__ = ['DummyLock', 'silence_fserrors', 'NullFile', 'synchronize', 'FS', 'flags_to_mode', 'NoDefaultMeta'] import os import os.path import shutil import fnmatch import datetime import time import errno try: import threading except ImportError: import dummy_threading as threading from fs.path import * from fs.errors import * from fs.local_functools import wraps import six from six import b class DummyLock(object): """A dummy lock object that doesn't do anything. This is used as a placeholder when locking is disabled. We can't directly use the Lock class from the dummy_threading module, since it attempts to sanity-check the sequence of acquire/release calls in a way that breaks when real threading is available. """ def acquire(self, blocking=1): """Acquiring a DummyLock always succeeds.""" return 1 def release(self): """Releasing a DummyLock always succeeds.""" pass def __enter__(self): return self def __exit__(self, exc_type, exc_value, traceback): pass def silence_fserrors(f, *args, **kwargs): """Perform a function call and return ``None`` if an :class:`fs.errors.FSError` is thrown :param f: Function to call :param args: Parameters to f :param kwargs: Keyword parameters to f """ try: return f(*args, **kwargs) except FSError: return None class NoDefaultMeta(object): """A singleton used to signify that there is no default for getmeta""" pass class NullFile(object): """A NullFile is a file object that has no functionality. Null files are returned by the :meth:`fs.base.FS.safeopen` method in FS objects when the file doesn't exist. This can simplify code by negating the need to check if a file exists, or handling exceptions. """ def __init__(self): self.closed = False def __iter__(self): return self def __enter__(self): return self def __exit__(self, exc_type, exc_value, traceback): self.closed = True def flush(self): pass def next(self): raise StopIteration def readline(self, *args, **kwargs): return b("") def close(self): self.closed = True def read(self, size=None): return b("") def seek(self, *args, **kwargs): pass def tell(self): return 0 def truncate(self, *args, **kwargs): return 0 def write(self, data): pass def writelines(self, *args, **kwargs): pass def synchronize(func): """Decorator to synchronize a method on self._lock.""" @wraps(func) def acquire_lock(self, *args, **kwargs): self._lock.acquire() try: return func(self, *args, **kwargs) finally: self._lock.release() return acquire_lock class FS(object): """The base class for Filesystem abstraction objects. An instance of a class derived from FS is an abstraction on some kind of filesystem, such as the OS filesystem or a zip file. """ _meta = {} def __init__(self, thread_synchronize=True): """The base class for Filesystem objects. :param thread_synconize: If True, a lock object will be created for the object, otherwise a dummy lock will be used. :type thread_synchronize: bool """ self.closed = False super(FS, self).__init__() self.thread_synchronize = thread_synchronize if thread_synchronize: self._lock = threading.RLock() else: self._lock = DummyLock() def __del__(self): if not getattr(self, 'closed', True): try: self.close() except: pass def __enter__(self): return self def __exit__(self, type, value, traceback): self.close() def cachehint(self, enabled): """Recommends the use of caching. Implementations are free to use or ignore this value. :param enabled: If True the implementation is permitted to aggressively cache directory structure / file information. Caching such information can speed up many operations, particularly for network based filesystems. The downside of caching is that changes made to directories or files outside of this interface may not be picked up immediately. """ pass # Deprecating cache_hint in favour of no underscore version, for consistency cache_hint = cachehint def close(self): """Close the filesystem. This will perform any shutdown related operations required. This method will be called automatically when the filesystem object is garbage collected, but it is good practice to call it explicitly so that any attached resourced are freed when they are no longer required. """ self.closed = True def __getstate__(self): # Locks can't be pickled, so instead we just indicate the # type of lock that should be there. None == no lock, # True == a proper lock, False == a dummy lock. state = self.__dict__.copy() lock = state.get("_lock", None) if lock is not None: if isinstance(lock, threading._RLock): state["_lock"] = True else: state["_lock"] = False return state def __setstate__(self, state): self.__dict__.update(state) lock = state.get("_lock") if lock is not None: if lock: self._lock = threading.RLock() else: self._lock = DummyLock() def getmeta(self, meta_name, default=NoDefaultMeta): """Retrieve a meta value associated with an FS object. Meta values are a way for an FS implementation to report potentially useful information associated with the file system. A meta key is a lower case string with no spaces. Meta keys may also be grouped in namespaces in a dotted notation, e.g. 'atomic.namespaces'. FS implementations aren't obliged to return any meta values, but the following are common: * *read_only* True if the file system cannot be modified * *thread_safe* True if the implementation is thread safe * *network* True if the file system requires network access * *unicode_paths* True if the file system supports unicode paths * *case_insensitive_paths* True if the file system ignores the case of paths * *atomic.makedir* True if making a directory is an atomic operation * *atomic.rename* True if rename is an atomic operation, (and not implemented as a copy followed by a delete) * *atomic.setcontents* True if the implementation supports setting the contents of a file as an atomic operation (without opening a file) * *free_space* The free space (in bytes) available on the file system * *total_space* The total space (in bytes) available on the file system * *virtual* True if the filesystem defers to other filesystems * *invalid_path_chars* A string containing characters that may not be used in paths FS implementations may expose non-generic meta data through a self-named namespace. e.g. ``"somefs.some_meta"`` Since no meta value is guaranteed to exist, it is advisable to always supply a default value to ``getmeta``. :param meta_name: The name of the meta value to retrieve :param default: An option default to return, if the meta value isn't present :raises `fs.errors.NoMetaError`: If specified meta value is not present, and there is no default """ if meta_name not in self._meta: if default is not NoDefaultMeta: return default raise NoMetaError(meta_name=meta_name) return self._meta[meta_name] def hasmeta(self, meta_name): """Check that a meta value is supported :param meta_name: The name of a meta value to check :rtype: bool """ try: self.getmeta(meta_name) except NoMetaError: return False return True def validatepath(self, path): """Validate an fs path, throws an :class:`~fs.errors.InvalidPathError` exception if validation fails. A path is invalid if it fails to map to a path on the underlaying filesystem. The default implementation checks for the presence of any of the characters in the meta value 'invalid_path_chars', but implementations may have other requirements for paths. :param path: an fs path to validatepath :raises `fs.errors.InvalidPathError`: if `path` does not map on to a valid path on this filesystem """ invalid_chars = self.getmeta('invalid_path_chars', default=None) if invalid_chars: re_invalid_chars = getattr(self, '_re_invalid_chars', None) if re_invalid_chars is None: self._re_invalid_chars = re_invalid_chars = re.compile('|'.join(re.escape(c) for c in invalid_chars), re.UNICODE) if re_invalid_chars.search(path): raise InvalidCharsInPathError(path) def isvalidpath(self, path): """Check if a path is valid on this filesystem :param path: an fs path """ try: self.validatepath(path) except InvalidPathError: return False else: return True def getsyspath(self, path, allow_none=False): """Returns the system path (a path recognized by the OS) if one is present. If the path does not map to a system path (and `allow_none` is False) then a NoSysPathError exception is thrown. Otherwise, the system path will be returned as a unicode string. :param path: a path within the filesystem :param allow_none: if True, this method will return None when there is no system path, rather than raising NoSysPathError :type allow_none: bool :raises `fs.errors.NoSysPathError`: if the path does not map on to a system path, and allow_none is set to False (default) :rtype: unicode """ if not allow_none: raise NoSysPathError(path=path) return None def hassyspath(self, path): """Check if the path maps to a system path (a path recognized by the OS). :param path: path to check :returns: True if `path` maps to a system path :rtype: bool """ return self.getsyspath(path, allow_none=True) is not None def getpathurl(self, path, allow_none=False): """Returns a url that corresponds to the given path, if one exists. If the path does not have an equivalent URL form (and allow_none is False) then a :class:`~fs.errors.NoPathURLError` exception is thrown. Otherwise the URL will be returns as an unicode string. :param path: a path within the filesystem :param allow_none: if true, this method can return None if there is no URL form of the given path :type allow_none: bool :raises `fs.errors.NoPathURLError`: If no URL form exists, and allow_none is False (the default) :rtype: unicode """ if not allow_none: raise NoPathURLError(path=path) return None def haspathurl(self, path): """Check if the path has an equivalent URL form :param path: path to check :returns: True if `path` has a URL form :rtype: bool """ return self.getpathurl(path, allow_none=True) is not None def open(self, path, mode='r', buffering=-1, encoding=None, errors=None, newline=None, line_buffering=False, **kwargs): """Open a the given path as a file-like object. :param path: a path to file that should be opened :type path: string :param mode: mode of file to open, identical to the mode string used in 'file' and 'open' builtins :type mode: string :param kwargs: additional (optional) keyword parameters that may be required to open the file :type kwargs: dict :rtype: a file-like object :raises `fs.errors.ParentDirectoryMissingError`: if an intermediate directory is missing :raises `fs.errors.ResourceInvalidError`: if an intermediate directory is an file :raises `fs.errors.ResourceNotFoundError`: if the path is not found """ raise UnsupportedError("open file") def safeopen(self, path, mode='r', buffering=-1, encoding=None, errors=None, newline=None, line_buffering=False, **kwargs): """Like :py:meth:`~fs.base.FS.open`, but returns a :py:class:`~fs.base.NullFile` if the file could not be opened. A ``NullFile`` is a dummy file which has all the methods of a file-like object, but contains no data. :param path: a path to file that should be opened :type path: string :param mode: mode of file to open, identical to the mode string used in 'file' and 'open' builtins :type mode: string :param kwargs: additional (optional) keyword parameters that may be required to open the file :type kwargs: dict :rtype: a file-like object """ try: f = self.open(path, mode=mode, buffering=buffering, encoding=encoding, errors=errors, newline=newline, line_buffering=line_buffering, **kwargs) except ResourceNotFoundError: return NullFile() return f def exists(self, path): """Check if a path references a valid resource. :param path: A path in the filesystem :type path: string :rtype: bool """ return self.isfile(path) or self.isdir(path) def isdir(self, path): """Check if a path references a directory. :param path: a path in the filesystem :type path: string :rtype: bool """ raise UnsupportedError("check for directory") def isfile(self, path): """Check if a path references a file. :param path: a path in the filesystem :type path: string :rtype: bool """ raise UnsupportedError("check for file") def __iter__(self): """ Iterates over paths returned by :py:meth:`~fs.base.listdir` method with default params. """ for f in self.listdir(): yield f def listdir(self, path="./", wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False): """Lists the the files and directories under a given path. The directory contents are returned as a list of unicode paths. :param path: root of the path to list :type path: string :param wildcard: Only returns paths that match this wildcard :type wildcard: string containing a wildcard, or a callable that accepts a path and returns a boolean :param full: returns full paths (relative to the root) :type full: bool :param absolute: returns absolute paths (paths beginning with /) :type absolute: bool :param dirs_only: if True, only return directories :type dirs_only: bool :param files_only: if True, only return files :type files_only: bool :rtype: iterable of paths :raises `fs.errors.ParentDirectoryMissingError`: if an intermediate directory is missing :raises `fs.errors.ResourceInvalidError`: if the path exists, but is not a directory :raises `fs.errors.ResourceNotFoundError`: if the path is not found """ raise UnsupportedError("list directory") def listdirinfo(self, path="./", wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False): """Retrieves a list of paths and path info under a given path. This method behaves like listdir() but instead of just returning the name of each item in the directory, it returns a tuple of the name and the info dict as returned by getinfo. This method may be more efficient than calling :py:meth:`~fs.base.FS.getinfo` on each individual item returned by :py:meth:`~fs.base.FS.listdir`, particularly for network based filesystems. :param path: root of the path to list :param wildcard: filter paths that match this wildcard :param dirs_only: only retrieve directories :type dirs_only: bool :param files_only: only retrieve files :type files_only: bool :raises `fs.errors.ResourceNotFoundError`: If the path is not found :raises `fs.errors.ResourceInvalidError`: If the path exists, but is not a directory """ path = normpath(path) def getinfo(p): try: if full or absolute: return self.getinfo(p) else: return self.getinfo(pathjoin(path, p)) except FSError: return {} return [(p, getinfo(p)) for p in self.listdir(path, wildcard=wildcard, full=full, absolute=absolute, dirs_only=dirs_only, files_only=files_only)] def _listdir_helper(self, path, entries, wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False): """A helper method called by listdir method that applies filtering. Given the path to a directory and a list of the names of entries within that directory, this method applies the semantics of the listdir() keyword arguments. An appropriately modified and filtered list of directory entries is returned. """ path = normpath(path) if dirs_only and files_only: raise ValueError("dirs_only and files_only can not both be True") if wildcard is not None: if not callable(wildcard): wildcard_re = re.compile(fnmatch.translate(wildcard)) wildcard = lambda fn: bool(wildcard_re.match(fn)) entries = [p for p in entries if wildcard(p)] if dirs_only: isdir = self.isdir entries = [p for p in entries if isdir(pathcombine(path, p))] elif files_only: isfile = self.isfile entries = [p for p in entries if isfile(pathcombine(path, p))] if full: entries = [pathcombine(path, p) for p in entries] elif absolute: path = abspath(path) entries = [(pathcombine(path, p)) for p in entries] return entries def ilistdir(self, path="./", wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False): """Generator yielding the files and directories under a given path. This method behaves identically to :py:meth:`fs.base.FS.listdir` but returns an generator instead of a list. Depending on the filesystem this may be more efficient than calling :py:meth:`fs.base.FS.listdir` and iterating over the resulting list. """ return iter(self.listdir(path, wildcard=wildcard, full=full, absolute=absolute, dirs_only=dirs_only, files_only=files_only)) def ilistdirinfo(self, path="./", wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False): """Generator yielding paths and path info under a given path. This method behaves identically to :py:meth:`~fs.base.listdirinfo` but returns an generator instead of a list. Depending on the filesystem this may be more efficient than calling :py:meth:`~fs.base.listdirinfo` and iterating over the resulting list. """ return iter(self.listdirinfo(path, wildcard, full, absolute, dirs_only, files_only)) def makedir(self, path, recursive=False, allow_recreate=False): """Make a directory on the filesystem. :param path: path of directory :type path: string :param recursive: if True, any intermediate directories will also be created :type recursive: bool :param allow_recreate: if True, re-creating a directory wont be an error :type allow_create: bool :raises `fs.errors.DestinationExistsError`: if the path is already a directory, and allow_recreate is False :raises `fs.errors.ParentDirectoryMissingError`: if a containing directory is missing and recursive is False :raises `fs.errors.ResourceInvalidError`: if a path is an existing file :raises `fs.errors.ResourceNotFoundError`: if the path is not found """ raise UnsupportedError("make directory") def remove(self, path): """Remove a file from the filesystem. :param path: Path of the resource to remove :type path: string :raises `fs.errors.ParentDirectoryMissingError`: if an intermediate directory is missing :raises `fs.errors.ResourceInvalidError`: if the path is a directory :raises `fs.errors.ResourceNotFoundError`: if the path does not exist """ raise UnsupportedError("remove resource") def removedir(self, path, recursive=False, force=False): """Remove a directory from the filesystem :param path: path of the directory to remove :type path: string :param recursive: if True, empty parent directories will be removed :type recursive: bool :param force: if True, any directory contents will be removed :type force: bool :raises `fs.errors.DirectoryNotEmptyError`: if the directory is not empty and force is False :raises `fs.errors.ParentDirectoryMissingError`: if an intermediate directory is missing :raises `fs.errors.ResourceInvalidError`: if the path is not a directory :raises `fs.errors.ResourceNotFoundError`: if the path does not exist """ raise UnsupportedError("remove directory") def rename(self, src, dst): """Renames a file or directory :param src: path to rename :type src: string :param dst: new name :type dst: string :raises ParentDirectoryMissingError: if a containing directory is missing :raises ResourceInvalidError: if the path or a parent path is not a directory or src is a parent of dst or one of src or dst is a dir and the other don't :raises ResourceNotFoundError: if the src path does not exist """ raise UnsupportedError("rename resource") @convert_os_errors def settimes(self, path, accessed_time=None, modified_time=None): """Set the accessed time and modified time of a file :param path: path to a file :type path: string :param accessed_time: the datetime the file was accessed (defaults to current time) :type accessed_time: datetime :param modified_time: the datetime the file was modified (defaults to current time) :type modified_time: datetime """ with self._lock: sys_path = self.getsyspath(path, allow_none=True) if sys_path is not None: now = datetime.datetime.now() if accessed_time is None: accessed_time = now if modified_time is None: modified_time = now accessed_time = int(time.mktime(accessed_time.timetuple())) modified_time = int(time.mktime(modified_time.timetuple())) os.utime(sys_path, (accessed_time, modified_time)) return True else: raise UnsupportedError("settimes") def getinfo(self, path): """Returns information for a path as a dictionary. The exact content of this dictionary will vary depending on the implementation, but will likely include a few common values. The following values will be found in info dictionaries for most implementations: * "size" - Number of bytes used to store the file or directory * "created_time" - A datetime object containing the time the resource was created * "accessed_time" - A datetime object containing the time the resource was last accessed * "modified_time" - A datetime object containing the time the resource was modified :param path: a path to retrieve information for :type path: string :rtype: dict :raises `fs.errors.ParentDirectoryMissingError`: if an intermediate directory is missing :raises `fs.errors.ResourceInvalidError`: if the path is not a directory :raises `fs.errors.ResourceNotFoundError`: if the path does not exist """ raise UnsupportedError("get resource info") def getinfokeys(self, path, *keys): """Get specified keys from info dict, as returned from `getinfo`. The returned dictionary may not contain all the keys that were asked for, if they aren't available. This method allows a filesystem to potentially provide a faster way of retrieving these info values if you are only interested in a subset of them. :param path: a path to retrieve information for :param keys: the info keys you would like to retrieve :rtype: dict """ info = self.getinfo(path) return dict((k, info[k]) for k in keys if k in info) def desc(self, path): """Returns short descriptive text regarding a path. Intended mainly as a debugging aid. :param path: A path to describe :rtype: str """ #if not self.exists(path): # return '' try: sys_path = self.getsyspath(path) except NoSysPathError: return "No description available" return sys_path def getcontents(self, path, mode='rb', encoding=None, errors=None, newline=None): """Returns the contents of a file as a string. :param path: A path of file to read :param mode: Mode to open file with (should be 'rb' for binary or 't' for text) :param encoding: Encoding to use when reading contents in text mode :param errors: Unicode errors parameter if text mode is use :param newline: Newlines parameter for text mode decoding :rtype: str :returns: file contents """ if 'r' not in mode: raise ValueError("mode must contain 'r' to be readable") f = None try: f = self.open(path, mode=mode, encoding=encoding, errors=errors, newline=newline) contents = f.read() return contents finally: if f is not None: f.close() def _setcontents(self, path, data, encoding=None, errors=None, chunk_size=1024 * 64, progress_callback=None, finished_callback=None): """Does the work of setcontents. Factored out, so that `setcontents_async` can use it""" if progress_callback is None: progress_callback = lambda bytes_written: None if finished_callback is None: finished_callback = lambda: None if not data: progress_callback(0) self.createfile(path, wipe=True) finished_callback() return 0 bytes_written = 0 progress_callback(0) if hasattr(data, 'read'): read = data.read chunk = read(chunk_size) if isinstance(chunk, six.text_type): f = self.open(path, 'wt', encoding=encoding, errors=errors) else: f = self.open(path, 'wb') write = f.write try: while chunk: write(chunk) bytes_written += len(chunk) progress_callback(bytes_written) chunk = read(chunk_size) finally: f.close() else: if isinstance(data, six.text_type): with self.open(path, 'wt', encoding=encoding, errors=errors) as f: f.write(data) bytes_written += len(data) else: with self.open(path, 'wb') as f: f.write(data) bytes_written += len(data) progress_callback(bytes_written) finished_callback() return bytes_written def setcontents(self, path, data=b'', encoding=None, errors=None, chunk_size=1024 * 64): """A convenience method to create a new file from a string or file-like object :param path: a path of the file to create :param data: a string or bytes object containing the contents for the new file :param encoding: if `data` is a file open in text mode, or a text string, then use this `encoding` to write to the destination file :param errors: if `data` is a file open in text mode or a text string, then use `errors` when opening the destination file :param chunk_size: Number of bytes to read in a chunk, if the implementation has to resort to a read / copy loop """ return self._setcontents(path, data, encoding=encoding, errors=errors, chunk_size=1024 * 64) def setcontents_async(self, path, data, encoding=None, errors=None, chunk_size=1024 * 64, progress_callback=None, finished_callback=None, error_callback=None): """Create a new file from a string or file-like object asynchronously This method returns a ``threading.Event`` object. Call the ``wait`` method on the event object to block until all data has been written, or simply ignore it. :param path: a path of the file to create :param data: a string or a file-like object containing the contents for the new file :param encoding: if `data` is a file open in text mode, or a text string, then use this `encoding` to write to the destination file :param errors: if `data` is a file open in text mode or a text string, then use `errors` when opening the destination file :param chunk_size: Number of bytes to read and write in a chunk :param progress_callback: A function that is called periodically with the number of bytes written. :param finished_callback: A function that is called when all data has been written :param error_callback: A function that is called with an exception object if any error occurs during the copy process. :returns: An event object that is set when the copy is complete, call the `wait` method of this object to block until the data is written """ finished_event = threading.Event() def do_setcontents(): try: self._setcontents(path, data, encoding=encoding, errors=errors, chunk_size=1024 * 64, progress_callback=progress_callback, finished_callback=finished_callback) except Exception, e: if error_callback is not None: error_callback(e) finally: finished_event.set() threading.Thread(target=do_setcontents).start() return finished_event def createfile(self, path, wipe=False): """Creates an empty file if it doesn't exist :param path: path to the file to create :param wipe: if True, the contents of the file will be erased """ with self._lock: if not wipe and self.isfile(path): return f = None try: f = self.open(path, 'wb') finally: if f is not None: f.close() def opendir(self, path): """Opens a directory and returns a FS object representing its contents. :param path: path to directory to open :type path: string :return: the opened dir :rtype: an FS object """ from fs.wrapfs.subfs import SubFS if not self.exists(path): raise ResourceNotFoundError(path) if not self.isdir(path): raise ResourceInvalidError("path should reference a directory") return SubFS(self, path) def walk(self, path="/", wildcard=None, dir_wildcard=None, search="breadth", ignore_errors=False): """Walks a directory tree and yields the root path and contents. Yields a tuple of the path of each directory and a list of its file contents. :param path: root path to start walking :type path: string :param wildcard: if given, only return files that match this wildcard :type wildcard: a string containing a wildcard (e.g. `*.txt`) or a callable that takes the file path and returns a boolean :param dir_wildcard: if given, only walk directories that match the wildcard :type dir_wildcard: a string containing a wildcard (e.g. `*.txt`) or a callable that takes the directory name and returns a boolean :param search: a string identifying the method used to walk the directories. There are two such methods: * ``"breadth"`` yields paths in the top directories first * ``"depth"`` yields the deepest paths first :param ignore_errors: ignore any errors reading the directory :type ignore_errors: bool :rtype: iterator of (current_path, paths) """ path = normpath(path) if not self.exists(path): raise ResourceNotFoundError(path) def listdir(path, *args, **kwargs): if ignore_errors: try: return self.listdir(path, *args, **kwargs) except: return [] else: return self.listdir(path, *args, **kwargs) if wildcard is None: wildcard = lambda f: True elif not callable(wildcard): wildcard_re = re.compile(fnmatch.translate(wildcard)) wildcard = lambda fn: bool(wildcard_re.match(fn)) if dir_wildcard is None: dir_wildcard = lambda f: True elif not callable(dir_wildcard): dir_wildcard_re = re.compile(fnmatch.translate(dir_wildcard)) dir_wildcard = lambda fn: bool(dir_wildcard_re.match(fn)) if search == "breadth": dirs = [path] dirs_append = dirs.append dirs_pop = dirs.pop isdir = self.isdir while dirs: current_path = dirs_pop() paths = [] paths_append = paths.append try: for filename in listdir(current_path, dirs_only=True): path = pathcombine(current_path, filename) if dir_wildcard(path): dirs_append(path) for filename in listdir(current_path, files_only=True): path = pathcombine(current_path, filename) if wildcard(filename): paths_append(filename) except ResourceNotFoundError: # Could happen if another thread / process deletes something whilst we are walking pass yield (current_path, paths) elif search == "depth": def recurse(recurse_path): try: for path in listdir(recurse_path, wildcard=dir_wildcard, full=True, dirs_only=True): for p in recurse(path): yield p except ResourceNotFoundError: # Could happen if another thread / process deletes something whilst we are walking pass yield (recurse_path, listdir(recurse_path, wildcard=wildcard, files_only=True)) for p in recurse(path): yield p else: raise ValueError("Search should be 'breadth' or 'depth'") def walkfiles(self, path="/", wildcard=None, dir_wildcard=None, search="breadth", ignore_errors=False): """Like the 'walk' method, but just yields file paths. :param path: root path to start walking :type path: string :param wildcard: if given, only return files that match this wildcard :type wildcard: A string containing a wildcard (e.g. `*.txt`) or a callable that takes the file path and returns a boolean :param dir_wildcard: if given, only walk directories that match the wildcard :type dir_wildcard: A string containing a wildcard (e.g. `*.txt`) or a callable that takes the directory name and returns a boolean :param search: a string identifying the method used to walk the directories. There are two such methods: * ``"breadth"`` yields paths in the top directories first * ``"depth"`` yields the deepest paths first :param ignore_errors: ignore any errors reading the directory :type ignore_errors: bool :rtype: iterator of file paths """ for path, files in self.walk(normpath(path), wildcard=wildcard, dir_wildcard=dir_wildcard, search=search, ignore_errors=ignore_errors): for f in files: yield pathcombine(path, f) def walkdirs(self, path="/", wildcard=None, search="breadth", ignore_errors=False): """Like the 'walk' method but yields directories. :param path: root path to start walking :type path: string :param wildcard: if given, only return directories that match this wildcard :type wildcard: A string containing a wildcard (e.g. `*.txt`) or a callable that takes the directory name and returns a boolean :param search: a string identifying the method used to walk the directories. There are two such methods: * ``"breadth"`` yields paths in the top directories first * ``"depth"`` yields the deepest paths first :param ignore_errors: ignore any errors reading the directory :type ignore_errors: bool :rtype: iterator of dir paths """ for p, _files in self.walk(path, dir_wildcard=wildcard, search=search, ignore_errors=ignore_errors): yield p def getsize(self, path): """Returns the size (in bytes) of a resource. :param path: a path to the resource :type path: string :returns: the size of the file :rtype: integer """ info = self.getinfo(path) size = info.get('size', None) if size is None: raise OperationFailedError("get size of resource", path) return size def copy(self, src, dst, overwrite=False, chunk_size=1024 * 64): """Copies a file from src to dst. :param src: the source path :type src: string :param dst: the destination path :type dst: string :param overwrite: if True, then an existing file at the destination may be overwritten; If False then DestinationExistsError will be raised. :type overwrite: bool :param chunk_size: size of chunks to use if a simple copy is required (defaults to 64K). :type chunk_size: bool """ with self._lock: if not self.isfile(src): if self.isdir(src): raise ResourceInvalidError(src, msg="Source is not a file: %(path)s") raise ResourceNotFoundError(src) if not overwrite and self.exists(dst): raise DestinationExistsError(dst) src_syspath = self.getsyspath(src, allow_none=True) dst_syspath = self.getsyspath(dst, allow_none=True) if src_syspath is not None and dst_syspath is not None: self._shutil_copyfile(src_syspath, dst_syspath) else: src_file = None try: src_file = self.open(src, "rb") self.setcontents(dst, src_file, chunk_size=chunk_size) except ResourceNotFoundError: if self.exists(src) and not self.exists(dirname(dst)): raise ParentDirectoryMissingError(dst) finally: if src_file is not None: src_file.close() @classmethod @convert_os_errors def _shutil_copyfile(cls, src_syspath, dst_syspath): try: shutil.copyfile(src_syspath, dst_syspath) except IOError, e: # shutil reports ENOENT when a parent directory is missing if getattr(e, "errno", None) == errno.ENOENT: if not os.path.exists(dirname(dst_syspath)): raise ParentDirectoryMissingError(dst_syspath) raise @classmethod @convert_os_errors def _shutil_movefile(cls, src_syspath, dst_syspath): shutil.move(src_syspath, dst_syspath) def move(self, src, dst, overwrite=False, chunk_size=16384): """moves a file from one location to another. :param src: source path :type src: string :param dst: destination path :type dst: string :param overwrite: When True the destination will be overwritten (if it exists), otherwise a DestinationExistsError will be thrown :type overwrite: bool :param chunk_size: Size of chunks to use when copying, if a simple copy is required :type chunk_size: integer :raise `fs.errors.DestinationExistsError`: if destination exists and `overwrite` is False """ with self._lock: src_syspath = self.getsyspath(src, allow_none=True) dst_syspath = self.getsyspath(dst, allow_none=True) # Try to do an os-level rename if possible. # Otherwise, fall back to copy-and-remove. if src_syspath is not None and dst_syspath is not None: if not os.path.isfile(src_syspath): if os.path.isdir(src_syspath): raise ResourceInvalidError(src, msg="Source is not a file: %(path)s") raise ResourceNotFoundError(src) if not overwrite and os.path.exists(dst_syspath): raise DestinationExistsError(dst) try: os.rename(src_syspath, dst_syspath) return except OSError: pass self.copy(src, dst, overwrite=overwrite, chunk_size=chunk_size) self.remove(src) def movedir(self, src, dst, overwrite=False, ignore_errors=False, chunk_size=16384): """moves a directory from one location to another. :param src: source directory path :type src: string :param dst: destination directory path :type dst: string :param overwrite: if True then any existing files in the destination directory will be overwritten :type overwrite: bool :param ignore_errors: if True then this method will ignore FSError exceptions when moving files :type ignore_errors: bool :param chunk_size: size of chunks to use when copying, if a simple copy is required :type chunk_size: integer :raise `fs.errors.DestinationExistsError`: if destination exists and `overwrite` is False """ with self._lock: if not self.isdir(src): if self.isfile(src): raise ResourceInvalidError(src, msg="Source is not a directory: %(path)s") raise ResourceNotFoundError(src) if not overwrite and self.exists(dst): raise DestinationExistsError(dst) src_syspath = self.getsyspath(src, allow_none=True) dst_syspath = self.getsyspath(dst, allow_none=True) if src_syspath is not None and dst_syspath is not None: try: os.rename(src_syspath, dst_syspath) return except OSError: pass def movefile_noerrors(src, dst, **kwargs): try: return self.move(src, dst, **kwargs) except FSError: return if ignore_errors: movefile = movefile_noerrors else: movefile = self.move src = abspath(src) dst = abspath(dst) if dst: self.makedir(dst, allow_recreate=overwrite) for dirname, filenames in self.walk(src, search="depth"): dst_dirname = relpath(frombase(src, abspath(dirname))) dst_dirpath = pathjoin(dst, dst_dirname) self.makedir(dst_dirpath, allow_recreate=True, recursive=True) for filename in filenames: src_filename = pathjoin(dirname, filename) dst_filename = pathjoin(dst_dirpath, filename) movefile(src_filename, dst_filename, overwrite=overwrite, chunk_size=chunk_size) self.removedir(dirname) def copydir(self, src, dst, overwrite=False, ignore_errors=False, chunk_size=16384): """copies a directory from one location to another. :param src: source directory path :type src: string :param dst: destination directory path :type dst: string :param overwrite: if True then any existing files in the destination directory will be overwritten :type overwrite: bool :param ignore_errors: if True, exceptions when copying will be ignored :type ignore_errors: bool :param chunk_size: size of chunks to use when copying, if a simple copy is required (defaults to 16K) """ with self._lock: if not self.isdir(src): raise ResourceInvalidError(src, msg="Source is not a directory: %(path)s") def copyfile_noerrors(src, dst, **kwargs): try: return self.copy(src, dst, **kwargs) except FSError: return if ignore_errors: copyfile = copyfile_noerrors else: copyfile = self.copy src = abspath(src) dst = abspath(dst) if not overwrite and self.exists(dst): raise DestinationExistsError(dst) if dst: self.makedir(dst, allow_recreate=True) for dirname, filenames in self.walk(src): dst_dirname = relpath(frombase(src, abspath(dirname))) dst_dirpath = pathjoin(dst, dst_dirname) self.makedir(dst_dirpath, allow_recreate=True, recursive=True) for filename in filenames: src_filename = pathjoin(dirname, filename) dst_filename = pathjoin(dst_dirpath, filename) copyfile(src_filename, dst_filename, overwrite=overwrite, chunk_size=chunk_size) def isdirempty(self, path): """Check if a directory is empty (contains no files or sub-directories) :param path: a directory path :rtype: bool """ with self._lock: path = normpath(path) iter_dir = iter(self.ilistdir(path)) try: next(iter_dir) except StopIteration: return True return False def makeopendir(self, path, recursive=False): """makes a directory (if it doesn't exist) and returns an FS object for the newly created directory. :param path: path to the new directory :param recursive: if True any intermediate directories will be created :return: the opened dir :rtype: an FS object """ with self._lock: self.makedir(path, allow_recreate=True, recursive=recursive) dir_fs = self.opendir(path) return dir_fs def printtree(self, max_levels=5): """Prints a tree structure of the FS object to the console :param max_levels: The maximum sub-directories to display, defaults to 5. Set to None for no limit """ from fs.utils import print_fs print_fs(self, max_levels=max_levels) tree = printtree def browse(self, hide_dotfiles=False): """Displays the FS tree in a graphical window (requires wxPython) :param hide_dotfiles: If True, files and folders that begin with a dot will be hidden """ from fs.browsewin import browse browse(self, hide_dotfiles) def getmmap(self, path, read_only=False, copy=False): """Returns a mmap object for this path. See http://docs.python.org/library/mmap.html for more details on the mmap module. :param path: A path on this filesystem :param read_only: If True, the mmap may not be modified :param copy: If False then changes wont be written back to the file :raises `fs.errors.NoMMapError`: Only paths that have a syspath can be opened as a mmap """ syspath = self.getsyspath(path, allow_none=True) if syspath is None: raise NoMMapError(path) try: import mmap except ImportError: raise NoMMapError(msg="mmap not supported") if read_only: f = open(syspath, 'rb') access = mmap.ACCESS_READ else: if copy: f = open(syspath, 'rb') access = mmap.ACCESS_COPY else: f = open(syspath, 'r+b') access = mmap.ACCESS_WRITE m = mmap.mmap(f.fileno(), 0, access=access) return m def flags_to_mode(flags, binary=True): """Convert an os.O_* flag bitmask into an FS mode string.""" if flags & os.O_WRONLY: if flags & os.O_TRUNC: mode = "w" elif flags & os.O_APPEND: mode = "a" else: mode = "r+" elif flags & os.O_RDWR: if flags & os.O_TRUNC: mode = "w+" elif flags & os.O_APPEND: mode = "a+" else: mode = "r+" else: mode = "r" if flags & os.O_EXCL: mode += "x" if binary: mode += 'b' else: mode += 't' return mode fs-0.5.4/fs/filelike.py0000664000175000017500000007123512512525115014647 0ustar willwill00000000000000""" fs.filelike =========== This module takes care of the groundwork for implementing and manipulating objects that provide a rich file-like interface, including reading, writing, seeking and iteration. The main class is FileLikeBase, which implements the entire file-like interface on top of primitive _read(), _write(), _seek(), _tell() and _truncate() methods. Subclasses may implement any or all of these methods to obtain the related higher-level file behaviors. Other useful classes include: * StringIO: a version of the builtin StringIO class, patched to more closely preserve the semantics of a standard file. * FileWrapper: a generic base class for wrappers around a filelike object (think e.g. compression or decryption). * SpooledTemporaryFile: a version of the builtin SpooledTemporaryFile class, patched to more closely preserve the semantics of a standard file. * LimitBytesFile: a filelike wrapper that limits the total bytes read from a file; useful for turning a socket into a file without reading past end-of-data. """ # Copyright (C) 2006-2009, Ryan Kelly # All rights reserved; available under the terms of the MIT License. import tempfile as _tempfile import fs class NotReadableError(IOError): pass class NotWritableError(IOError): pass class NotSeekableError(IOError): pass class NotTruncatableError(IOError): pass import six from six import PY3, b if PY3: from six import BytesIO as _StringIO else: try: from cStringIO import StringIO as _StringIO except ImportError: from StringIO import StringIO as _StringIO class FileLikeBase(object): """Base class for implementing file-like objects. This class takes a lot of the legwork out of writing file-like objects with a rich interface. It implements the higher-level file-like methods on top of five primitive methods: _read, _write, _seek, _tell and _truncate. See their docstrings for precise details on how these methods behave. Subclasses then need only implement some subset of these methods for rich file-like interface compatibility. They may of course override other methods as desired. The class is missing the following attributes and methods, which don't really make sense for anything but real files: * fileno() * isatty() * encoding * mode * name * newlines Unlike standard file objects, all read methods share the same buffer and so can be freely mixed (e.g. read(), readline(), next(), ...). This class understands and will accept the following mode strings, with any additional characters being ignored: * r - open the file for reading only. * r+ - open the file for reading and writing. * r- - open the file for streamed reading; do not allow seek/tell. * w - open the file for writing only; create the file if it doesn't exist; truncate it to zero length. * w+ - open the file for reading and writing; create the file if it doesn't exist; truncate it to zero length. * w- - open the file for streamed writing; do not allow seek/tell. * a - open the file for writing only; create the file if it doesn't exist; place pointer at end of file. * a+ - open the file for reading and writing; create the file if it doesn't exist; place pointer at end of file. These are mostly standard except for the "-" indicator, which has been added for efficiency purposes in cases where seeking can be expensive to simulate (e.g. compressed files). Note that any file opened for both reading and writing must also support seeking. """ def __init__(self,bufsize=1024*64): """FileLikeBase Constructor. The optional argument 'bufsize' specifies the number of bytes to read at a time when looking for a newline character. Setting this to a larger number when lines are long should improve efficiency. """ super(FileLikeBase, self).__init__() # File-like attributes self.closed = False self.softspace = 0 # Our own attributes self._bufsize = bufsize # buffer size for chunked reading self._rbuffer = None # data that's been read but not returned self._wbuffer = None # data that's been given but not written self._sbuffer = None # data between real & apparent file pos self._soffset = 0 # internal offset of file pointer # # The following five methods are the ones that subclasses are expected # to implement. Carefully check their docstrings. # def _read(self,sizehint=-1): """Read approximately bytes from the file-like object. This method is to be implemented by subclasses that wish to be readable. It should read approximately bytes from the file and return them as a string. If is missing or less than or equal to zero, try to read all the remaining contents. The method need not guarantee any particular number of bytes - it may return more bytes than requested, or fewer. If needed the size hint may be completely ignored. It may even return an empty string if no data is yet available. Because of this, the method must return None to signify that EOF has been reached. The higher-level methods will never indicate EOF until None has been read from _read(). Once EOF is reached, it should be safe to call _read() again, immediately returning None. """ raise NotReadableError("Object not readable") def _write(self,string,flushing=False): """Write the given string to the file-like object. This method must be implemented by subclasses wishing to be writable. It must attempt to write as much of the given data as possible to the file, but need not guarantee that it is all written. It may return None to indicate that all data was written, or return as a string any data that could not be written. If the keyword argument 'flushing' is true, it indicates that the internal write buffers are being flushed, and *all* the given data is expected to be written to the file. If unwritten data is returned when 'flushing' is true, an IOError will be raised. """ raise NotWritableError("Object not writable") def _seek(self,offset,whence): """Set the file's internal position pointer, approximately. This method should set the file's position to approximately 'offset' bytes relative to the position specified by 'whence'. If it is not possible to position the pointer exactly at the given offset, it should be positioned at a convenient *smaller* offset and the file data between the real and apparent position should be returned. At minimum, this method must implement the ability to seek to the start of the file, i.e. offset=0 and whence=0. If more complex seeks are difficult to implement then it may raise NotImplementedError to have them simulated (inefficiently) by the higher-level machinery of this class. """ raise NotSeekableError("Object not seekable") def _tell(self): """Get the location of the file's internal position pointer. This method must be implemented by subclasses that wish to be seekable, and must return the position of the file's internal pointer. Due to buffering, the position seen by users of this class (the "apparent position") may be different to the position returned by this method (the "actual position"). """ raise NotSeekableError("Object not seekable") def _truncate(self,size): """Truncate the file's size to . This method must be implemented by subclasses that wish to be truncatable. It must truncate the file to exactly the given size or fail with an IOError. Note that will never be None; if it was not specified by the user then it is calculated as the file's apparent position (which may be different to its actual position due to buffering). """ raise NotTruncatableError("Object not truncatable") # # The following methods provide the public API of the filelike object. # Subclasses shouldn't need to mess with these (except perhaps for # close() and flush()) # def _check_mode(self,mode,mstr=None): """Check whether the file may be accessed in the given mode. 'mode' must be one of "r" or "w", and this function returns False if the file-like object has a 'mode' attribute, and it does not permit access in that mode. If there is no 'mode' attribute, it defaults to "r+". If seek support is not required, use "r-" or "w-" as the mode string. To check a mode string other than self.mode, pass it in as the second argument. """ if mstr is None: try: mstr = self.mode except AttributeError: mstr = "r+" if "+" in mstr: return True if "-" in mstr and "-" not in mode: return False if "r" in mode: if "r" not in mstr: return False if "w" in mode: if "w" not in mstr and "a" not in mstr: return False return True def _assert_mode(self,mode,mstr=None): """Check whether the file may be accessed in the given mode. This method is equivalent to _check_assert(), but raises IOError instead of returning False. """ if mstr is None: try: mstr = self.mode except AttributeError: mstr = "r+" if "+" in mstr: return True if "-" in mstr and "-" not in mode: raise NotSeekableError("File does not support seeking.") if "r" in mode: if "r" not in mstr: raise NotReadableError("File not opened for reading") if "w" in mode: if "w" not in mstr and "a" not in mstr: raise NotWritableError("File not opened for writing") return True def flush(self): """Flush internal write buffer, if necessary.""" if self.closed: raise IOError("File has been closed") if self._check_mode("w-") and self._wbuffer is not None: buffered = b("") if self._sbuffer: buffered = buffered + self._sbuffer self._sbuffer = None buffered = buffered + self._wbuffer self._wbuffer = None leftover = self._write(buffered,flushing=True) if leftover and not isinstance(leftover, int): raise IOError("Could not flush write buffer.") def close(self): """Flush write buffers and close the file. The file may not be accessed further once it is closed. """ # Errors in subclass constructors can cause this to be called without # having called FileLikeBase.__init__(). Since we need the attrs it # initializes in cleanup, ensure we call it here. if not hasattr(self,"closed"): FileLikeBase.__init__(self) if not self.closed: self.flush() self.closed = True def __del__(self): self.close() def __enter__(self): return self def __exit__(self,exc_type,exc_val,exc_tb): self.close() return False def next(self): """next() method complying with the iterator protocol. File-like objects are their own iterators, with each call to next() returning subsequent lines from the file. """ ln = self.readline() if ln == b(""): raise StopIteration() return ln def __iter__(self): return self def truncate(self,size=None): """Truncate the file to the given size. If is not specified or is None, the current file position is used. Note that this method may fail at runtime if the underlying filelike object is not truncatable. """ if "-" in getattr(self,"mode",""): raise NotTruncatableError("File is not seekable, can't truncate.") if self._wbuffer: self.flush() if size is None: size = self.tell() self._truncate(size) def seek(self,offset,whence=0): """Move the internal file pointer to the given location.""" if whence > 2 or whence < 0: raise ValueError("Invalid value for 'whence': " + str(whence)) if "-" in getattr(self,"mode",""): raise NotSeekableError("File is not seekable.") # Ensure that there's nothing left in the write buffer if self._wbuffer: self.flush() # Adjust for any data left in the read buffer if whence == 1 and self._rbuffer: offset = offset - len(self._rbuffer) self._rbuffer = None # Adjust for any discrepancy in actual vs apparent seek position if whence == 1: if self._sbuffer: offset = offset + len(self._sbuffer) if self._soffset: offset = offset + self._soffset self._sbuffer = None self._soffset = 0 # Shortcut the special case of staying put. # As per posix, this has already cases the buffers to be flushed. if offset == 0 and whence == 1: return # Catch any failed attempts to read while simulating seek try: # Try to do a whence-wise seek if it is implemented. sbuf = None try: sbuf = self._seek(offset,whence) except NotImplementedError: # Try to simulate using an absolute seek. try: if whence == 1: offset = self._tell() + offset elif whence == 2: if hasattr(self,"size"): offset = self.size + offset else: self._do_read_rest() offset = self.tell() + offset else: # absolute seek already failed, don't try again raise NotImplementedError sbuf = self._seek(offset,0) except NotImplementedError: # Simulate by reseting to start self._seek(0,0) self._soffset = offset finally: self._sbuffer = sbuf except NotReadableError: raise NotSeekableError("File not readable, can't simulate seek") def tell(self): """Determine current position of internal file pointer.""" # Need to adjust for unread/unwritten data in buffers pos = self._tell() if self._rbuffer: pos = pos - len(self._rbuffer) if self._wbuffer: pos = pos + len(self._wbuffer) if self._sbuffer: pos = pos + len(self._sbuffer) if self._soffset: pos = pos + self._soffset return pos def read(self,size=-1): """Read at most 'size' bytes from the file. Bytes are returned as a string. If 'size' is negative, zero or missing, the remainder of the file is read. If EOF is encountered immediately, the empty string is returned. """ if self.closed: raise IOError("File has been closed") self._assert_mode("r-") return self._do_read(size) def _do_read(self,size): """Private method to read from the file. This method behaves the same as self.read(), but skips some permission and sanity checks. It is intended for use in simulating seek(), where we may want to read (and discard) information from a file not opened in read mode. Note that this may still fail if the file object actually can't be read from - it just won't check whether the mode string gives permission. """ # If we were previously writing, ensure position is correct if self._wbuffer is not None: self.seek(0,1) # Discard any data that should have been seeked over if self._sbuffer: s = len(self._sbuffer) self._sbuffer = None self.read(s) elif self._soffset: s = self._soffset self._soffset = 0 while s > self._bufsize: self._do_read(self._bufsize) s -= self._bufsize self._do_read(s) # Should the entire file be read? if size < 0: if self._rbuffer: data = [self._rbuffer] else: data = [] self._rbuffer = b("") newData = self._read() while newData is not None: data.append(newData) newData = self._read() output = b("").join(data) # Otherwise, we need to return a specific amount of data else: if self._rbuffer: newData = self._rbuffer data = [newData] else: newData = b("") data = [] sizeSoFar = len(newData) while sizeSoFar < size: newData = self._read(size-sizeSoFar) if not newData: break data.append(newData) sizeSoFar += len(newData) data = b("").join(data) if sizeSoFar > size: # read too many bytes, store in the buffer self._rbuffer = data[size:] data = data[:size] else: self._rbuffer = b("") output = data return output def _do_read_rest(self): """Private method to read the file through to EOF.""" data = self._do_read(self._bufsize) while data != b(""): data = self._do_read(self._bufsize) def readline(self,size=-1): """Read a line from the file, or at most bytes.""" bits = [] indx = -1 sizeSoFar = 0 while indx == -1: nextBit = self.read(self._bufsize) bits.append(nextBit) sizeSoFar += len(nextBit) if not nextBit: break if size > 0 and sizeSoFar >= size: break indx = nextBit.find(b("\n")) # If not found, return whole string up to length # Any leftovers are pushed onto front of buffer if indx == -1: data = b("").join(bits) if size > 0 and sizeSoFar > size: extra = data[size:] data = data[:size] self._rbuffer = extra + self._rbuffer return data # If found, push leftovers onto front of buffer # Add one to preserve the newline in the return value indx += 1 extra = bits[-1][indx:] bits[-1] = bits[-1][:indx] self._rbuffer = extra + self._rbuffer return b("").join(bits) def readlines(self,sizehint=-1): """Return a list of all lines in the file.""" return [ln for ln in self] def xreadlines(self): """Iterator over lines in the file - equivalent to iter(self).""" return iter(self) def write(self,string): """Write the given string to the file.""" if self.closed: raise IOError("File has been closed") self._assert_mode("w-") # If we were previously reading, ensure position is correct if self._rbuffer is not None: self.seek(0, 1) # If we're actually behind the apparent position, we must also # write the data in the gap. if self._sbuffer: string = self._sbuffer + string self._sbuffer = None elif self._soffset: s = self._soffset self._soffset = 0 try: string = self._do_read(s) + string except NotReadableError: raise NotSeekableError("File not readable, could not complete simulation of seek") self.seek(0, 0) if self._wbuffer: string = self._wbuffer + string leftover = self._write(string) if leftover is None or isinstance(leftover, int): self._wbuffer = b("") return len(string) - (leftover or 0) else: self._wbuffer = leftover return len(string) - len(leftover) def writelines(self,seq): """Write a sequence of lines to the file.""" for ln in seq: self.write(ln) class FileWrapper(FileLikeBase): """Base class for objects that wrap a file-like object. This class provides basic functionality for implementing file-like objects that wrap another file-like object to alter its functionality in some way. It takes care of house-keeping duties such as flushing and closing the wrapped file. Access to the wrapped file is given by the attribute wrapped_file. By convention, the subclass's constructor should accept this as its first argument and pass it to its superclass's constructor in the same position. This class provides a basic implementation of _read() and _write() which just calls read() and write() on the wrapped object. Subclasses will probably want to override these. """ _append_requires_overwrite = False def __init__(self,wrapped_file,mode=None): """FileWrapper constructor. 'wrapped_file' must be a file-like object, which is to be wrapped in another file-like object to provide additional functionality. If given, 'mode' must be the access mode string under which the wrapped file is to be accessed. If not given or None, it is looked up on the wrapped file if possible. Otherwise, it is not set on the object. """ # This is used for working around flush/close inefficiencies self.__closing = False super(FileWrapper,self).__init__() self.wrapped_file = wrapped_file if mode is None: self.mode = getattr(wrapped_file,"mode","r+") else: self.mode = mode self._validate_mode() # Copy useful attributes of wrapped_file if hasattr(wrapped_file,"name"): self.name = wrapped_file.name # Respect append-mode setting if "a" in self.mode: if self._check_mode("r"): self.wrapped_file.seek(0) self.seek(0,2) def _validate_mode(self): """Check that various file-mode conditions are satisfied.""" # If append mode requires overwriting the underlying file, # if must not be opened in append mode. if self._append_requires_overwrite: if self._check_mode("w"): if "a" in getattr(self.wrapped_file,"mode",""): raise ValueError("Underlying file can't be in append mode") def __del__(self): # Errors in subclass constructors could result in this being called # without invoking FileWrapper.__init__. Establish some simple # invariants to prevent errors in this case. if not hasattr(self,"wrapped_file"): self.wrapped_file = None if not hasattr(self,"_FileWrapper__closing"): self.__closing = False # Close the wrapper and the underlying file independently, so the # latter is still closed on cleanup even if the former errors out. try: if FileWrapper is not None: super(FileWrapper,self).close() finally: if hasattr(getattr(self,"wrapped_file",None),"close"): self.wrapped_file.close() def close(self): """Close the object for reading/writing.""" # The superclass implementation of this will call flush(), # which calls flush() on our wrapped object. But we then call # close() on it, which will call its flush() again! To avoid # this inefficiency, our flush() will not flush the wrapped # file when we're closing. if not self.closed: self.__closing = True super(FileWrapper,self).close() if hasattr(self.wrapped_file,"close"): self.wrapped_file.close() def flush(self): """Flush the write buffers of the file.""" super(FileWrapper,self).flush() if not self.__closing and hasattr(self.wrapped_file,"flush"): self.wrapped_file.flush() def _read(self,sizehint=-1): data = self.wrapped_file.read(sizehint) if data == b(""): return None return data def _write(self,string,flushing=False): self.wrapped_file.write(string) def _seek(self,offset,whence): self.wrapped_file.seek(offset,whence) def _tell(self): return self.wrapped_file.tell() def _truncate(self,size): return self.wrapped_file.truncate(size) class StringIO(FileWrapper): """StringIO wrapper that more closely matches standard file behavior. This is a simple compatibility wrapper around the native StringIO class which fixes some corner-cases of its behavior. Specifically: * adding __enter__ and __exit__ methods * having truncate(size) zero-fill when growing the file """ def __init__(self,data=None,mode=None): wrapped_file = _StringIO() if data is not None: wrapped_file.write(data) wrapped_file.seek(0) super(StringIO,self).__init__(wrapped_file,mode) def getvalue(self): return self.wrapped_file.getvalue() def _truncate(self,size): pos = self.wrapped_file.tell() self.wrapped_file.truncate(size) curlen = len(self.wrapped_file.getvalue()) if size > curlen: self.wrapped_file.seek(curlen) try: self.wrapped_file.write(b("\x00")*(size-curlen)) finally: self.wrapped_file.seek(pos) class SpooledTemporaryFile(FileWrapper): """SpooledTemporaryFile wrapper with some compatibility fixes. This is a simple compatibility wrapper around the native class of the same name, fixing some corner-cases of its behavior. Specifically: * have truncate() accept a size argument * roll to disk is seeking past the max in-memory size * use improved StringIO class from this module """ def __init__(self,max_size=0,mode="w+b",bufsize=-1,*args,**kwds): try: stf_args = (max_size,mode,bufsize) + args wrapped_file = _tempfile.SpooledTemporaryFile(*stf_args,**kwds) wrapped_file._file = StringIO() #wrapped_file._file = six.BytesIO() self.__is_spooled = True except AttributeError: ntf_args = (mode,bufsize) + args wrapped_file = _tempfile.NamedTemporaryFile(*ntf_args,**kwds) self.__is_spooled = False super(SpooledTemporaryFile,self).__init__(wrapped_file) def _seek(self,offset,whence): if self.__is_spooled: max_size = self.wrapped_file._max_size if whence == fs.SEEK_SET: if offset > max_size: self.wrapped_file.rollover() elif whence == fs.SEEK_CUR: if offset + self.wrapped_file.tell() > max_size: self.wrapped_file.rollover() else: if offset > 0: self.wrapped_file.rollover() self.wrapped_file.seek(offset,whence) def _truncate(self,size): if self.__is_spooled: self.wrapped_file._file.truncate(size) else: self.wrapped_file.truncate(size) def fileno(self): return self.wrapped_file.fileno() class LimitBytesFile(FileWrapper): """Filelike wrapper to limit bytes read from a stream.""" def __init__(self,size,fileobj,*args,**kwds): self.size = size self.__remaining = size super(LimitBytesFile,self).__init__(fileobj,*args,**kwds) def _read(self,sizehint=-1): if self.__remaining <= 0: return None if sizehint is None or sizehint < 0 or sizehint > self.__remaining: sizehint = self.__remaining data = super(LimitBytesFile,self)._read(sizehint) if data is not None: self.__remaining -= len(data) return data fs-0.5.4/fs/ftpfs.py0000664000175000017500000013453612512525115014211 0ustar willwill00000000000000""" fs.ftpfs ======== FTPFS is a filesystem for accessing an FTP server (uses ftplib in standard library) """ __all__ = ['FTPFS'] import sys import fs from fs.base import * from fs.errors import * from fs.path import pathsplit, abspath, dirname, recursepath, normpath, pathjoin, isbase from fs import iotools from ftplib import FTP, error_perm, error_temp, error_proto, error_reply try: from ftplib import _GLOBAL_DEFAULT_TIMEOUT except ImportError: _GLOBAL_DEFAULT_TIMEOUT = object() import threading import datetime import calendar from socket import error as socket_error from fs.local_functools import wraps import six from six import PY3, b if PY3: from six import BytesIO as StringIO else: try: from cStringIO import StringIO except ImportError: from StringIO import StringIO import time # ----------------------------------------------- # Taken from http://www.clapper.org/software/python/grizzled/ # ----------------------------------------------- class Enum(object): def __init__(self, *names): self._names_map = dict((name, i) for i, name in enumerate(names)) def __getattr__(self, name): return self._names_map[name] MONTHS = ('jan', 'feb', 'mar', 'apr', 'may', 'jun', 'jul', 'aug', 'sep', 'oct', 'nov', 'dec') MTIME_TYPE = Enum('UNKNOWN', 'LOCAL', 'REMOTE_MINUTE', 'REMOTE_DAY') """ ``MTIME_TYPE`` identifies how a modification time ought to be interpreted (assuming the caller cares). - ``LOCAL``: Time is local to the client, granular to (at least) the minute - ``REMOTE_MINUTE``: Time is local to the server and granular to the minute - ``REMOTE_DAY``: Time is local to the server and granular to the day. - ``UNKNOWN``: Time's locale is unknown. """ ID_TYPE = Enum('UNKNOWN', 'FULL') """ ``ID_TYPE`` identifies how a file's identifier should be interpreted. - ``FULL``: The ID is known to be complete. - ``UNKNOWN``: The ID is not set or its type is unknown. """ # --------------------------------------------------------------------------- # Globals # --------------------------------------------------------------------------- now = time.time() current_year = time.localtime().tm_year # --------------------------------------------------------------------------- # Classes # --------------------------------------------------------------------------- class FTPListData(object): """ The `FTPListDataParser` class's ``parse_line()`` method returns an instance of this class, capturing the parsed data. :IVariables: name : str The name of the file, if parsable try_cwd : bool ``True`` if the entry might be a directory (i.e., the caller might want to try an FTP ``CWD`` command), ``False`` if it cannot possibly be a directory. try_retr : bool ``True`` if the entry might be a retrievable file (i.e., the caller might want to try an FTP ``RETR`` command), ``False`` if it cannot possibly be a file. size : long The file's size, in bytes mtime : long The file's modification time, as a value that can be passed to ``time.localtime()``. mtime_type : `MTIME_TYPE` How to interpret the modification time. See `MTIME_TYPE`. id : str A unique identifier for the file. The unique identifier is unique on the *server*. On a Unix system, this identifier might be the device number and the file's inode; on other system's, it might be something else. It's also possible for this field to be ``None``. id_type : `ID_TYPE` How to interpret the identifier. See `ID_TYPE`. """ def __init__(self, raw_line): self.raw_line = raw_line self.name = None self.try_cwd = False self.try_retr = False self.size = 0 self.mtime_type = MTIME_TYPE.UNKNOWN self.mtime = 0 self.id_type = ID_TYPE.UNKNOWN self.id = None class FTPListDataParser(object): """ An ``FTPListDataParser`` object can be used to parse one or more lines that were retrieved by an FTP ``LIST`` command that was sent to a remote server. """ def __init__(self): pass def parse_line(self, ftp_list_line): """ Parse a line from an FTP ``LIST`` command. :Parameters: ftp_list_line : str The line of output :rtype: `FTPListData` :return: An `FTPListData` object describing the parsed line, or ``None`` if the line could not be parsed. Note that it's possible for this method to return a partially-filled `FTPListData` object (e.g., one without a name). """ buf = ftp_list_line if len(buf) < 2: # an empty name in EPLF, with no info, could be 2 chars return None c = buf[0] if c == '+': return self._parse_EPLF(buf) elif c in 'bcdlps-': return self._parse_unix_style(buf) i = buf.find(';') if i > 0: return self._parse_multinet(buf, i) if c in '0123456789': return self._parse_msdos(buf) return None # UNIX ls does not show the year for dates in the last six months. # So we have to guess the year. # # Apparently NetWare uses ``twelve months'' instead of ``six months''; ugh. # Some versions of ls also fail to show the year for future dates. def _guess_time(self, month, mday, hour=0, minute=0): year = None t = None for year in range(current_year - 1, current_year + 100): t = self._get_mtime(year, month, mday, hour, minute) if (now - t) < (350 * 86400): return t return 0 def _get_mtime(self, year, month, mday, hour=0, minute=0, second=0): return time.mktime((year, month, mday, hour, minute, second, 0, 0, -1)) def _get_month(self, buf): if len(buf) == 3: for i in range(0, 12): if buf.lower().startswith(MONTHS[i]): return i+1 return -1 def _parse_EPLF(self, buf): result = FTPListData(buf) # see http://cr.yp.to/ftp/list/eplf.html #"+i8388621.29609,m824255902,/,\tdev" #"+i8388621.44468,m839956783,r,s10376,\tRFCEPLF" i = 1 for j in range(1, len(buf)): if buf[j] == '\t': result.name = buf[j+1:] break if buf[j] == ',': c = buf[i] if c == '/': result.try_cwd = True elif c == 'r': result.try_retr = True elif c == 's': result.size = long(buf[i+1:j]) elif c == 'm': result.mtime_type = MTIME_TYPE.LOCAL result.mtime = long(buf[i+1:j]) elif c == 'i': result.id_type = ID_TYPE.FULL result.id = buf[i+1:j-i-1] i = j + 1 return result def _parse_unix_style(self, buf): # UNIX-style listing, without inum and without blocks: # "-rw-r--r-- 1 root other 531 Jan 29 03:26 README" # "dr-xr-xr-x 2 root other 512 Apr 8 1994 etc" # "dr-xr-xr-x 2 root 512 Apr 8 1994 etc" # "lrwxrwxrwx 1 root other 7 Jan 25 00:17 bin -> usr/bin" # # Also produced by Microsoft's FTP servers for Windows: # "---------- 1 owner group 1803128 Jul 10 10:18 ls-lR.Z" # "d--------- 1 owner group 0 May 9 19:45 Softlib" # # Also WFTPD for MSDOS: # "-rwxrwxrwx 1 noone nogroup 322 Aug 19 1996 message.ftp" # # Also NetWare: # "d [R----F--] supervisor 512 Jan 16 18:53 login" # "- [R----F--] rhesus 214059 Oct 20 15:27 cx.exe" # # Also NetPresenz for the Mac: # "-------r-- 326 1391972 1392298 Nov 22 1995 MegaPhone.sit" # "drwxrwxr-x folder 2 May 10 1996 network" result = FTPListData(buf) buflen = len(buf) c = buf[0] if c == 'd': result.try_cwd = True if c == '-': result.try_retr = True if c == 'l': result.try_retr = True result.try_cwd = True state = 1 i = 0 tokens = buf.split() for j in range(1, buflen): if (buf[j] == ' ') and (buf[j - 1] != ' '): if state == 1: # skipping perm state = 2 elif state == 2: # skipping nlink state = 3 if ((j - i) == 6) and (buf[i] == 'f'): # NetPresenz state = 4 elif state == 3: # skipping UID/GID state = 4 elif state == 4: # getting tentative size try: size = long(buf[i:j]) except ValueError: pass state = 5 elif state == 5: # searching for month, else getting tentative size month = self._get_month(buf[i:j]) if month >= 0: state = 6 else: size = long(buf[i:j]) elif state == 6: # have size and month mday = long(buf[i:j]) state = 7 elif state == 7: # have size, month, mday if (j - i == 4) and (buf[i+1] == ':'): hour = long(buf[i]) minute = long(buf[i+2:i+4]) result.mtime_type = MTIME_TYPE.REMOTE_MINUTE result.mtime = self._guess_time(month, mday, hour, minute) elif (j - i == 5) and (buf[i+2] == ':'): hour = long(buf[i:i+2]) minute = long(buf[i+3:i+5]) result.mtime_type = MTIME_TYPE.REMOTE_MINUTE result.mtime = self._guess_time(month, mday, hour, minute) elif j - i >= 4: year = long(buf[i:j]) result.mtime_type = MTIME_TYPE.REMOTE_DAY result.mtime = self._get_mtime(year, month, mday) else: break result.name = buf[j+1:] state = 8 elif state == 8: # twiddling thumbs pass i = j + 1 while (i < buflen) and (buf[i] == ' '): i += 1 #if state != 8: #return None result.size = size if c == 'l': i = 0 while (i + 3) < len(result.name): if result.name[i:i+4] == ' -> ': result.target = result.name[i+4:] result.name = result.name[:i] break i += 1 # eliminate extra NetWare spaces if (buf[1] == ' ') or (buf[1] == '['): namelen = len(result.name) if namelen > 3: result.name = result.name.strip() return result def _parse_multinet(self, buf, i): # MultiNet (some spaces removed from examples) # "00README.TXT;1 2 30-DEC-1996 17:44 [SYSTEM] (RWED,RWED,RE,RE)" # "CORE.DIR;1 1 8-SEP-1996 16:09 [SYSTEM] (RWE,RWE,RE,RE)" # and non-MultiNet VMS: #"CII-MANUAL.TEX;1 213/216 29-JAN-1996 03:33:12 [ANONYMOU,ANONYMOUS] (RWED,RWED,,)" result = FTPListData(buf) result.name = buf[:i] buflen = len(buf) if i > 4: if buf[i-4:i] == '.DIR': result.name = result.name[0:-4] result.try_cwd = True if not result.try_cwd: result.try_retr = True try: i = buf.index(' ', i) i = _skip(buf, i, ' ') i = buf.index(' ', i) i = _skip(buf, i, ' ') j = i j = buf.index('-', j) mday = long(buf[i:j]) j = _skip(buf, j, '-') i = j j = buf.index('-', j) month = self._get_month(buf[i:j]) if month < 0: raise IndexError j = _skip(buf, j, '-') i = j j = buf.index(' ', j) year = long(buf[i:j]) j = _skip(buf, j, ' ') i = j j = buf.index(':', j) hour = long(buf[i:j]) j = _skip(buf, j, ':') i = j while (buf[j] != ':') and (buf[j] != ' '): j += 1 if j == buflen: raise IndexError # abort, abort! minute = long(buf[i:j]) result.mtime_type = MTIME_TYPE.REMOTE_MINUTE result.mtime = self._get_mtime(year, month, mday, hour, minute) except IndexError: pass return result def _parse_msdos(self, buf): # MSDOS format # 04-27-00 09:09PM licensed # 07-18-00 10:16AM pub # 04-14-00 03:47PM 589 readme.htm buflen = len(buf) i = 0 j = 0 try: result = FTPListData(buf) j = buf.index('-', j) month = long(buf[i:j]) j = _skip(buf, j, '-') i = j j = buf.index('-', j) mday = long(buf[i:j]) j = _skip(buf, j, '-') i = j j = buf.index(' ', j) year = long(buf[i:j]) if year < 50: year += 2000 if year < 1000: year += 1900 j = _skip(buf, j, ' ') i = j j = buf.index(':', j) hour = long(buf[i:j]) j = _skip(buf, j, ':') i = j while not (buf[j] in 'AP'): j += 1 if j == buflen: raise IndexError minute = long(buf[i:j]) if buf[j] == 'A': j += 1 if j == buflen: raise IndexError if buf[j] == 'P': hour = (hour + 12) % 24 j += 1 if j == buflen: raise IndexError if buf[j] == 'M': j += 1 if j == buflen: raise IndexError j = _skip(buf, j, ' ') if buf[j] == '<': result.try_cwd = True j = buf.index(' ', j) else: i = j j = buf.index(' ', j) result.size = long(buf[i:j]) result.try_retr = True j = _skip(buf, j, ' ') result.name = buf[j:] result.mtime_type = MTIME_TYPE.REMOTE_MINUTE result.mtime = self._get_mtime(year, month, mday, hour, minute) except IndexError: pass return result class FTPMlstDataParser(object): """ An ``FTPMlstDataParser`` object can be used to parse one or more lines that were retrieved by an FTP ``MLST`` or ``MLSD`` command that was sent to a remote server. """ def __init__(self): pass def parse_line(self, ftp_list_line): """ Parse a line from an FTP ``MLST`` or ``MLSD`` command. :Parameters: ftp_list_line : str The line of output :rtype: `FTPListData` :return: An `FTPListData` object describing the parsed line, or ``None`` if the line could not be parsed. Note that it's possible for this method to return a partially-filled `FTPListData` object (e.g., one without a mtime). """ result = FTPListData(ftp_list_line) # pull out the name parts = ftp_list_line.partition(' ') result.name = parts[2] # parse the facts if parts[0][-1] == ';': for fact in parts[0][:-1].split(';'): parts = fact.partition('=') factname = parts[0].lower() factvalue = parts[2] if factname == 'unique': if factvalue == "0g0" or factvalue == "0g1": # Matrix FTP server sometimes returns bogus "unique" facts result.id_type = ID_TYPE.UNKNOWN else: result.id_type = ID_TYPE.FULL result.id = factvalue elif factname == 'modify': result.mtime_type = MTIME_TYPE.LOCAL result.mtime = calendar.timegm((int(factvalue[0:4]), int(factvalue[4:6]), int(factvalue[6:8]), int(factvalue[8:10]), int(factvalue[10:12]), int(factvalue[12:14]), 0, 0, 0)) elif factname == 'size': result.size = long(factvalue) elif factname == 'sizd': # some FTP servers report directory size with sizd result.size = long(factvalue) elif factname == 'type': if factvalue.lower() == 'file': result.try_retr = True elif factvalue.lower() in ['dir', 'cdir', 'pdir']: result.try_cwd = True else: # dunno if it's file or directory result.try_retr = True result.try_cwd = True return result # --------------------------------------------------------------------------- # Public Functions # --------------------------------------------------------------------------- def parse_ftp_list_line(ftp_list_line, is_mlst=False): """ Convenience function that instantiates an `FTPListDataParser` object and passes ``ftp_list_line`` to the object's ``parse_line()`` method, returning the result. :Parameters: ftp_list_line : str The line of output :rtype: `FTPListData` :return: An `FTPListData` object describing the parsed line, or ``None`` if the line could not be parsed. Note that it's possible for this method to return a partially-filled `FTPListData` object (e.g., one without a name). """ if is_mlst: return FTPMlstDataParser().parse_line(ftp_list_line) else: return FTPListDataParser().parse_line(ftp_list_line) # --------------------------------------------------------------------------- # Private Functions # --------------------------------------------------------------------------- def _skip(s, i, c): while s[i] == c: i += 1 if i == len(s): raise IndexError return i def fileftperrors(f): @wraps(f) def deco(self, *args, **kwargs): self._lock.acquire() try: try: ret = f(self, *args, **kwargs) except Exception, e: self.ftpfs._translate_exception(args[0] if args else '', e) finally: self._lock.release() return ret return deco class _FTPFile(object): """ A file-like that provides access to a file being streamed over ftp.""" blocksize = 1024 * 64 def __init__(self, ftpfs, ftp, path, mode): if not hasattr(self, '_lock'): self._lock = threading.RLock() self.ftpfs = ftpfs self.ftp = ftp self.path = normpath(path) self.mode = mode self.read_pos = 0 self.write_pos = 0 self.closed = False self.file_size = None if 'r' in mode or 'a' in mode: self.file_size = ftpfs.getsize(path) self.conn = None self._start_file(mode, _encode(self.path)) @fileftperrors def _start_file(self, mode, path): self.read_pos = 0 self.write_pos = 0 if 'r' in mode: self.ftp.voidcmd('TYPE I') self.conn = self.ftp.transfercmd('RETR ' + path, None) else:#if 'w' in mode or 'a' in mode: self.ftp.voidcmd('TYPE I') if 'a' in mode: self.write_pos = self.file_size self.conn = self.ftp.transfercmd('APPE ' + path) else: self.conn = self.ftp.transfercmd('STOR ' + path) @fileftperrors def read(self, size=None): if self.conn is None: return b('') chunks = [] if size is None or size < 0: while 1: data = self.conn.recv(self.blocksize) if not data: self.conn.close() self.conn = None self.ftp.voidresp() break chunks.append(data) self.read_pos += len(data) return b('').join(chunks) remaining_bytes = size while remaining_bytes: read_size = min(remaining_bytes, self.blocksize) data = self.conn.recv(read_size) if not data: self.conn.close() self.conn = None self.ftp.voidresp() break chunks.append(data) self.read_pos += len(data) remaining_bytes -= len(data) return b('').join(chunks) @fileftperrors def write(self, data): data_pos = 0 remaining_data = len(data) while remaining_data: chunk_size = min(remaining_data, self.blocksize) self.conn.sendall(data[data_pos:data_pos+chunk_size]) data_pos += chunk_size remaining_data -= chunk_size self.write_pos += chunk_size def __enter__(self): return self def __exit__(self,exc_type,exc_value,traceback): self.close() @fileftperrors def flush(self): self.ftpfs._on_file_written(self.path) @fileftperrors def seek(self, pos, where=fs.SEEK_SET): # Ftp doesn't support a real seek, so we close the transfer and resume # it at the new position with the REST command # I'm not sure how reliable this method is! if self.file_size is None: raise ValueError("Seek only works with files open for read") self._lock.acquire() try: current = self.tell() new_pos = None if where == fs.SEEK_SET: new_pos = pos elif where == fs.SEEK_CUR: new_pos = current + pos elif where == fs.SEEK_END: new_pos = self.file_size + pos if new_pos < 0: raise ValueError("Can't seek before start of file") if self.conn is not None: self.conn.close() finally: self._lock.release() self.close() self._lock.acquire() try: self.ftp = self.ftpfs._open_ftp() self.ftp.sendcmd('TYPE I') self.ftp.sendcmd('REST %i' % (new_pos)) self.__init__(self.ftpfs, self.ftp, self.path, self.mode) self.read_pos = new_pos finally: self._lock.release() #raise UnsupportedError('ftp seek') @fileftperrors def tell(self): if 'r' in self.mode: return self.read_pos else: return self.write_pos @fileftperrors def truncate(self, size=None): self.ftpfs._on_file_written(self.path) # Inefficient, but I don't know how else to implement this if size is None: size = self.tell() if self.conn is not None: self.conn.close() self.close() read_f = None try: read_f = self.ftpfs.open(self.path, 'rb') data = read_f.read(size) finally: if read_f is not None: read_f.close() self.ftp = self.ftpfs._open_ftp() self.mode = 'w' self.__init__(self.ftpfs, self.ftp, _encode(self.path), self.mode) #self._start_file(self.mode, self.path) self.write(data) if len(data) < size: self.write('\0' * (size - len(data))) @fileftperrors def close(self): if 'w' in self.mode or 'a' in self.mode or '+' in self.mode: self.ftpfs._on_file_written(self.path) if self.conn is not None: try: self.conn.close() self.conn = None self.ftp.voidresp() except error_temp, error_perm: pass if self.ftp is not None: try: self.ftp.close() except error_temp, error_perm: pass self.closed = True def next(self): return self.readline() def readline(self, size=None): return next(iotools.line_iterator(self, size)) def __iter__(self): return iotools.line_iterator(self) def ftperrors(f): @wraps(f) def deco(self, *args, **kwargs): self._lock.acquire() try: self._enter_dircache() try: try: ret = f(self, *args, **kwargs) except Exception, e: self._translate_exception(args[0] if args else '', e) finally: self._leave_dircache() finally: self._lock.release() return ret return deco def _encode(s): if isinstance(s, unicode): return s.encode('utf-8') return s class _DirCache(dict): def __init__(self): super(_DirCache, self).__init__() self.count = 0 def addref(self): self.count += 1 return self.count def decref(self): self.count -= 1 return self.count class FTPFS(FS): _meta = { 'thread_safe' : True, 'network' : True, 'virtual': False, 'read_only' : False, 'unicode_paths' : True, 'case_insensitive_paths' : False, 'atomic.move' : True, 'atomic.copy' : True, 'atomic.makedir' : True, 'atomic.rename' : True, 'atomic.setcontents' : False, 'file.read_and_write' : False, } def __init__(self, host='', user='', passwd='', acct='', timeout=_GLOBAL_DEFAULT_TIMEOUT, port=21, dircache=True, follow_symlinks=False): """Connect to a FTP server. :param host: Host to connect to :param user: Username, or a blank string for anonymous :param passwd: Password, if required :param acct: Accounting information (few servers require this) :param timeout: Timeout in seconds :param port: Port to connection (default is 21) :param dircache: If True then directory information will be cached, speeding up operations such as `getinfo`, `isdir`, `isfile`, but changes to the ftp file structure will not be visible until :meth:`~fs.ftpfs.FTPFS.clear_dircache` is called """ super(FTPFS, self).__init__() self.host = host self.port = port self.user = user self.passwd = passwd self.acct = acct self.timeout = timeout self.default_timeout = timeout is _GLOBAL_DEFAULT_TIMEOUT self.use_dircache = dircache self.follow_symlinks = follow_symlinks self.use_mlst = False self._lock = threading.RLock() self._init_dircache() self._cache_hint = False try: self.ftp except FSError: self.closed = True raise def _init_dircache(self): self.dircache = _DirCache() @synchronize def cache_hint(self, enabled): self._cache_hint = bool(enabled) def _enter_dircache(self): self.dircache.addref() def _leave_dircache(self): self.dircache.decref() if self.use_dircache: if not self.dircache.count and not self._cache_hint: self.clear_dircache() else: self.clear_dircache() assert self.dircache.count >= 0, "dircache count should never be negative" @synchronize def _on_file_written(self, path): self.refresh_dircache(dirname(path)) @synchronize def _readdir(self, path): path = abspath(normpath(path)) if self.dircache.count: cached_dirlist = self.dircache.get(path) if cached_dirlist is not None: return cached_dirlist dirlist = {} def _get_FEAT(ftp): features = dict() try: response = ftp.sendcmd("FEAT") if response[:3] == "211": for line in response.splitlines()[1:]: if line[3] == "211": break if line[0] != ' ': break parts = line[1:].partition(' ') features[parts[0].upper()] = parts[2] except error_perm: # some FTP servers may not support FEAT pass return features def on_line(line): if not isinstance(line, unicode): line = line.decode('utf-8') info = parse_ftp_list_line(line, self.use_mlst) if info: info = info.__dict__ if info['name'] not in ('.', '..'): dirlist[info['name']] = info try: encoded_path = _encode(path) ftp_features = _get_FEAT(self.ftp) if 'MLST' in ftp_features: self.use_mlst = True try: # only request the facts we need self.ftp.sendcmd("OPTS MLST type;unique;size;modify;") except error_perm: # some FTP servers don't support OPTS MLST pass # need to send MLST first to discover if it's file or dir response = self.ftp.sendcmd("MLST " + encoded_path) lines = response.splitlines() if lines[0][:3] == "250": list_line = lines[1] # MLST line is preceded by space if list_line[0] == ' ': on_line(list_line[1:]) else: # Matrix FTP server has bug on_line(list_line) # if it's a dir, then we can send a MLSD if dirlist[dirlist.keys()[0]]['try_cwd']: dirlist = {} self.ftp.retrlines("MLSD " + encoded_path, on_line) else: self.ftp.dir(encoded_path, on_line) except error_reply: pass self.dircache[path] = dirlist def is_symlink(info): return info['try_retr'] and info['try_cwd'] and info.has_key('target') def resolve_symlink(linkpath): linkinfo = self.getinfo(linkpath) if not linkinfo.has_key('resolved'): linkinfo['resolved'] = linkpath if is_symlink(linkinfo): target = linkinfo['target'] base, fname = pathsplit(linkpath) return resolve_symlink(pathjoin(base, target)) else: return linkinfo if self.follow_symlinks: for name in dirlist: if is_symlink(dirlist[name]): target = dirlist[name]['target'] linkinfo = resolve_symlink(pathjoin(path, target)) for key in linkinfo: if key != 'name': dirlist[name][key] = linkinfo[key] del dirlist[name]['target'] return dirlist @synchronize def clear_dircache(self, *paths): """ Clear cached directory information. :param path: Path of directory to clear cache for, or all directories if None (the default) """ if not paths: self.dircache.clear() else: dircache = self.dircache paths = [normpath(abspath(path)) for path in paths] for cached_path in dircache.keys(): for path in paths: if isbase(cached_path, path): dircache.pop(cached_path, None) break @synchronize def refresh_dircache(self, *paths): for path in paths: path = abspath(normpath(path)) self.dircache.pop(path, None) @synchronize def _check_path(self, path): path = normpath(path) base, fname = pathsplit(abspath(path)) dirlist = self._readdir(base) if fname and fname not in dirlist: raise ResourceNotFoundError(path) return dirlist, fname def _get_dirlist(self, path): path = normpath(path) base, fname = pathsplit(abspath(path)) dirlist = self._readdir(base) return dirlist, fname @ftperrors def get_ftp(self): if self.closed: return None if not getattr(self, '_ftp', None): self._ftp = self._open_ftp() return self._ftp ftp = property(get_ftp) @ftperrors def _open_ftp(self): try: ftp = FTP() if self.default_timeout or sys.version_info < (2,6,): ftp.connect(self.host, self.port) else: ftp.connect(self.host, self.port, self.timeout) ftp.login(self.user, self.passwd, self.acct) except socket_error, e: raise RemoteConnectionError(str(e), details=e) return ftp def __getstate__(self): state = super(FTPFS, self).__getstate__() del state['_lock'] state.pop('_ftp', None) return state def __setstate__(self,state): super(FTPFS, self).__setstate__(state) self._init_dircache() self._lock = threading.RLock() #self._ftp = None #self.ftp def __str__(self): return '' % self.host def __unicode__(self): return u'' % self.host @convert_os_errors def _translate_exception(self, path, exception): """ Translates exceptions that my be thrown by the ftp code in to FS exceptions TODO: Flesh this out with more specific exceptions """ if isinstance(exception, socket_error): self._ftp = None raise RemoteConnectionError(str(exception), details=exception) elif isinstance(exception, error_temp): code, message = str(exception).split(' ', 1) self._ftp = None raise RemoteConnectionError(str(exception), path=path, msg="FTP error: %s" % str(exception), details=exception) elif isinstance(exception, error_perm): code, message = str(exception).split(' ', 1) code = int(code) if code == 550: pass if code == 552: raise StorageSpaceError raise PermissionDeniedError(str(exception), path=path, msg="FTP error: %s" % str(exception), details=exception) raise exception @ftperrors def close(self): if not self.closed: try: self.ftp.close() except FSError: pass self.closed = True def getpathurl(self, path, allow_none=False): path = normpath(path) credentials = '%s:%s' % (self.user, self.passwd) if credentials == ':': url = 'ftp://%s%s' % (self.host.rstrip('/'), abspath(path)) else: url = 'ftp://%s@%s%s' % (credentials, self.host.rstrip('/'), abspath(path)) return url @iotools.filelike_to_stream @ftperrors def open(self, path, mode, buffering=-1, encoding=None, errors=None, newline=None, line_buffering=False, **kwargs): path = normpath(path) mode = mode.lower() if self.isdir(path): raise ResourceInvalidError(path) if 'r' in mode or 'a' in mode: if not self.isfile(path): raise ResourceNotFoundError(path) if 'w' in mode or 'a' in mode or '+' in mode: self.refresh_dircache(dirname(path)) ftp = self._open_ftp() f = _FTPFile(self, ftp, normpath(path), mode) return f @ftperrors def setcontents(self, path, data=b'', encoding=None, errors=None, chunk_size=1024*64): path = normpath(path) data = iotools.make_bytes_io(data, encoding=encoding, errors=errors) self.refresh_dircache(dirname(path)) self.ftp.storbinary('STOR %s' % _encode(path), data, blocksize=chunk_size) @ftperrors def getcontents(self, path, mode="rb", encoding=None, errors=None, newline=None): path = normpath(path) contents = StringIO() self.ftp.retrbinary('RETR %s' % _encode(path), contents.write, blocksize=1024*64) data = contents.getvalue() if 'b' in data: return data return iotools.decode_binary(data, encoding=encoding, errors=errors) @ftperrors def exists(self, path): path = normpath(path) if path in ('', '/'): return True dirlist, fname = self._get_dirlist(path) return fname in dirlist @ftperrors def isdir(self, path): path = normpath(path) if path in ('', '/'): return True dirlist, fname = self._get_dirlist(path) info = dirlist.get(fname) if info is None: return False return info['try_cwd'] @ftperrors def isfile(self, path): path = normpath(path) if path in ('', '/'): return False dirlist, fname = self._get_dirlist(path) info = dirlist.get(fname) if info is None: return False return not info['try_cwd'] @ftperrors def listdir(self, path="./", wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False): path = normpath(path) #self.clear_dircache(path) if not self.exists(path): raise ResourceNotFoundError(path) if not self.isdir(path): raise ResourceInvalidError(path) paths = self._readdir(path).keys() return self._listdir_helper(path, paths, wildcard, full, absolute, dirs_only, files_only) @ftperrors def listdirinfo(self, path="./", wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False): path = normpath(path) def getinfo(p): try: if full or absolute: return self.getinfo(p) else: return self.getinfo(pathjoin(path,p)) except FSError: return {} return [(p, getinfo(p)) for p in self.listdir(path, wildcard=wildcard, full=full, absolute=absolute, dirs_only=dirs_only, files_only=files_only)] @ftperrors def makedir(self, path, recursive=False, allow_recreate=False): path = normpath(path) if path in ('', '/'): return def checkdir(path): if not self.isdir(path): self.clear_dircache(dirname(path)) try: self.ftp.mkd(_encode(path)) except error_reply: return except error_perm, e: if recursive or allow_recreate: return if str(e).split(' ', 1)[0]=='550': raise DestinationExistsError(path) else: raise if recursive: for p in recursepath(path): checkdir(p) else: base = dirname(path) if not self.exists(base): raise ParentDirectoryMissingError(path) if not allow_recreate: if self.exists(path): if self.isfile(path): raise ResourceInvalidError(path) raise DestinationExistsError(path) checkdir(path) @ftperrors def remove(self, path): if not self.exists(path): raise ResourceNotFoundError(path) if not self.isfile(path): raise ResourceInvalidError(path) self.refresh_dircache(dirname(path)) self.ftp.delete(_encode(path)) @ftperrors def removedir(self, path, recursive=False, force=False): path = abspath(normpath(path)) if not self.exists(path): raise ResourceNotFoundError(path) if self.isfile(path): raise ResourceInvalidError(path) if normpath(path) in ('', '/'): raise RemoveRootError(path) if not force: for _checkpath in self.listdir(path): raise DirectoryNotEmptyError(path) try: if force: for rpath in self.listdir(path, full=True): try: if self.isfile(rpath): self.remove(rpath) elif self.isdir(rpath): self.removedir(rpath, force=force) except FSError: pass self.clear_dircache(dirname(path)) self.ftp.rmd(_encode(path)) except error_reply: pass if recursive: try: if dirname(path) not in ('', '/'): self.removedir(dirname(path), recursive=True) except DirectoryNotEmptyError: pass self.clear_dircache(dirname(path), path) @ftperrors def rename(self, src, dst): try: self.refresh_dircache(dirname(src), dirname(dst)) self.ftp.rename(_encode(src), _encode(dst)) except error_perm, exception: code, message = str(exception).split(' ', 1) if code == "550": if not self.exists(dirname(dst)): raise ParentDirectoryMissingError(dst) raise except error_reply: pass @ftperrors def getinfo(self, path): dirlist, fname = self._check_path(path) if not fname: return {} info = dirlist[fname].copy() info['modified_time'] = datetime.datetime.fromtimestamp(info['mtime']) info['created_time'] = info['modified_time'] return info @ftperrors def getsize(self, path): size = None if self.dircache.count: dirlist, fname = self._check_path(path) size = dirlist[fname].get('size') if size is not None: return size self.ftp.sendcmd('TYPE I') size = self.ftp.size(_encode(path)) if size is None: dirlist, fname = self._check_path(path) size = dirlist[fname].get('size') if size is None: raise OperationFailedError('getsize', path) return size @ftperrors def desc(self, path): path = normpath(path) url = self.getpathurl(path, allow_none=True) if url: return url dirlist, fname = self._check_path(path) if fname not in dirlist: raise ResourceNotFoundError(path) return dirlist[fname].get('raw_line', 'No description available') @ftperrors def move(self, src, dst, overwrite=False, chunk_size=16384): if not overwrite and self.exists(dst): raise DestinationExistsError(dst) #self.refresh_dircache(dirname(src), dirname(dst)) try: self.rename(src, dst) except: self.copy(src, dst, overwrite=overwrite) self.remove(src) finally: self.refresh_dircache(src, dirname(src), dst, dirname(dst)) @ftperrors def copy(self, src, dst, overwrite=False, chunk_size=1024*64): if not self.isfile(src): if self.isdir(src): raise ResourceInvalidError(src, msg="Source is not a file: %(path)s") raise ResourceNotFoundError(src) if not overwrite and self.exists(dst): raise DestinationExistsError(dst) dst = normpath(dst) src_file = None try: src_file = self.open(src, "rb") ftp = self._open_ftp() ftp.voidcmd('TYPE I') ftp.storbinary('STOR %s' % _encode(normpath(dst)), src_file, blocksize=chunk_size) finally: self.refresh_dircache(dirname(dst)) if src_file is not None: src_file.close() @ftperrors def movedir(self, src, dst, overwrite=False, ignore_errors=False, chunk_size=16384): self.clear_dircache(dirname(src), dirname(dst)) super(FTPFS, self).movedir(src, dst, overwrite, ignore_errors, chunk_size) @ftperrors def copydir(self, src, dst, overwrite=False, ignore_errors=False, chunk_size=16384): self.clear_dircache(dirname(dst)) super(FTPFS, self).copydir(src, dst, overwrite, ignore_errors, chunk_size) if __name__ == "__main__": ftp_fs = FTPFS('ftp.ncsa.uiuc.edu') ftp_fs.cache_hint(True) from fs.browsewin import browse browse(ftp_fs) #ftp_fs = FTPFS('127.0.0.1', 'user', '12345', dircache=True) #f = ftp_fs.open('testout.txt', 'w') #f.write("Testing writing to an ftp file!") #f.write("\nHai!") #f.close() #ftp_fs.createfile(u"\N{GREEK CAPITAL LETTER KAPPA}", 'unicode!') #kappa = u"\N{GREEK CAPITAL LETTER KAPPA}" #ftp_fs.makedir(kappa) #print repr(ftp_fs.listdir()) #print repr(ftp_fs.listdir()) #ftp_fs.makedir('a/b/c/d', recursive=True) #print ftp_fs.getsize('/testout.txt') #print f.read() #for p in ftp_fs: # print p #from fs.utils import print_fs #print_fs(ftp_fs) #print ftp_fs.getsize('test.txt') #from fs.browsewin import browse #browse(ftp_fs) fs-0.5.4/fs/osfs/0000755000000000000000000000000012621617365013461 5ustar rootroot00000000000000fs-0.5.4/fs/osfs/xattrs.py0000664000175000017500000000323112512525115015351 0ustar willwill00000000000000""" fs.osfs.xattrs ============== Extended-attribute support for OSFS """ import os import sys import errno from fs.errors import * from fs.path import * from fs.base import FS try: import xattr except ImportError: xattr = None if xattr is not None: class OSFSXAttrMixin(object): """Mixin providing extended-attribute support via the 'xattr' module""" def __init__(self, *args, **kwargs): super(OSFSXAttrMixin, self).__init__(*args, **kwargs) @convert_os_errors def setxattr(self, path, key, value): xattr.xattr(self.getsyspath(path))[key]=value @convert_os_errors def getxattr(self, path, key, default=None): try: return xattr.xattr(self.getsyspath(path)).get(key) except KeyError: return default @convert_os_errors def delxattr(self, path, key): try: del xattr.xattr(self.getsyspath(path))[key] except KeyError: pass @convert_os_errors def listxattrs(self, path): return xattr.xattr(self.getsyspath(path)).keys() else: class OSFSXAttrMixin(object): """Mixin disable extended-attribute support.""" def __init__(self, *args, **kwargs): super(OSFSXAttrMixin, self).__init__(*args, **kwargs) def getxattr(self,path,key,default=None): raise UnsupportedError def setxattr(self,path,key,value): raise UnsupportedError def delxattr(self,path,key): raise UnsupportedError def listxattrs(self,path): raise UnsupportedError fs-0.5.4/fs/osfs/watch_inotify.py0000664000175000017500000002443712512525115016706 0ustar willwill00000000000000""" fs.osfs.watch_inotify ============= Change watcher support for OSFS, backed by pyinotify. """ import os import sys import errno import select import threading from fs.errors import * from fs.path import * from fs.watch import * try: import pyinotify except Exception, e: # pyinotify sometimes raises its own custom errors on import. # How on earth are we supposed to catch them when we can't import them? if isinstance(e,ImportError): raise raise ImportError("could not import pyinotify") try: pyinotify.WatchManager.get_fd except AttributeError: raise ImportError("pyinotify version is too old") class OSFSWatchMixin(WatchableFSMixin): """Mixin providing change-watcher support via pyinotify.""" __watch_lock = threading.Lock() __watch_thread = None def close(self): super(OSFSWatchMixin,self).close() self.notify_watchers(CLOSED) for watcher_list in self._watchers.values(): for watcher in watcher_list: self.del_watcher(watcher) self.__watch_lock.acquire() try: wt = self.__watch_thread if wt is not None and not wt.watchers: wt.stop() wt.join() OSFSWatchMixin.__watch_thread = None finally: self.__watch_lock.release() @convert_os_errors def add_watcher(self,callback,path="/",events=None,recursive=True): super_add_watcher = super(OSFSWatchMixin,self).add_watcher w = super_add_watcher(callback,path,events,recursive) w._pyinotify_id = None syspath = self.getsyspath(path) if isinstance(syspath,unicode): syspath = syspath.encode(sys.getfilesystemencoding()) # Each watch gets its own WatchManager, since it's tricky to make # a single WatchManager handle multiple callbacks with different # events for a single path. This means we pay one file descriptor # for each watcher added to the filesystem. That's not too bad. w._pyinotify_WatchManager = wm = pyinotify.WatchManager() # Each individual notifier gets multiplexed by a single shared thread. w._pyinotify_Notifier = pyinotify.Notifier(wm) evtmask = self.__get_event_mask(events) def process_events(event): self.__route_event(w,event) kwds = dict(rec=recursive,auto_add=recursive,quiet=False) try: wids = wm.add_watch(syspath,evtmask,process_events,**kwds) except pyinotify.WatchManagerError, e: raise OperationFailedError("add_watcher",details=e) w._pyinotify_id = wids[syspath] self.__watch_lock.acquire() try: wt = self.__get_watch_thread() wt.add_watcher(w) finally: self.__watch_lock.release() return w @convert_os_errors def del_watcher(self,watcher_or_callback): if isinstance(watcher_or_callback,Watcher): watchers = [watcher_or_callback] else: watchers = self._find_watchers(watcher_or_callback) for watcher in watchers: wm = watcher._pyinotify_WatchManager wm.rm_watch(watcher._pyinotify_id,rec=watcher.recursive) super(OSFSWatchMixin,self).del_watcher(watcher) self.__watch_lock.acquire() try: wt = self.__get_watch_thread() for watcher in watchers: wt.del_watcher(watcher) finally: self.__watch_lock.release() def __get_event_mask(self,events): """Convert the given set of events into a pyinotify event mask.""" if events is None: events = (EVENT,) mask = 0 for evt in events: if issubclass(ACCESSED,evt): mask |= pyinotify.IN_ACCESS if issubclass(CREATED,evt): mask |= pyinotify.IN_CREATE if issubclass(REMOVED,evt): mask |= pyinotify.IN_DELETE mask |= pyinotify.IN_DELETE_SELF if issubclass(MODIFIED,evt): mask |= pyinotify.IN_ATTRIB mask |= pyinotify.IN_MODIFY mask |= pyinotify.IN_CLOSE_WRITE if issubclass(MOVED_SRC,evt): mask |= pyinotify.IN_MOVED_FROM mask |= pyinotify.IN_MOVED_TO if issubclass(MOVED_DST,evt): mask |= pyinotify.IN_MOVED_FROM mask |= pyinotify.IN_MOVED_TO if issubclass(OVERFLOW,evt): mask |= pyinotify.IN_Q_OVERFLOW if issubclass(CLOSED,evt): mask |= pyinotify.IN_UNMOUNT return mask def __route_event(self,watcher,inevt): """Convert pyinotify event into fs.watch event, then handle it.""" try: path = self.unsyspath(inevt.pathname) except ValueError: return try: src_path = inevt.src_pathname if src_path is not None: src_path = self.unsyspath(src_path) except (AttributeError,ValueError): src_path = None if inevt.mask & pyinotify.IN_ACCESS: watcher.handle_event(ACCESSED(self,path)) if inevt.mask & pyinotify.IN_CREATE: watcher.handle_event(CREATED(self,path)) # Recursive watching of directories in pyinotify requires # the creation of a new watch for each subdir, resulting in # a race condition whereby events in the subdir are missed. # We'd prefer to duplicate events than to miss them. if inevt.mask & pyinotify.IN_ISDIR: try: # pyinotify does this for dirs itself, we only. # need to worry about newly-created files. for child in self.listdir(path,files_only=True): cpath = pathjoin(path,child) self.notify_watchers(CREATED,cpath) self.notify_watchers(MODIFIED,cpath,True) except FSError: pass if inevt.mask & pyinotify.IN_DELETE: watcher.handle_event(REMOVED(self,path)) if inevt.mask & pyinotify.IN_DELETE_SELF: watcher.handle_event(REMOVED(self,path)) if inevt.mask & pyinotify.IN_ATTRIB: watcher.handle_event(MODIFIED(self,path,False)) if inevt.mask & pyinotify.IN_MODIFY: watcher.handle_event(MODIFIED(self,path,True)) if inevt.mask & pyinotify.IN_CLOSE_WRITE: watcher.handle_event(MODIFIED(self,path,True, closed=True)) if inevt.mask & pyinotify.IN_MOVED_FROM: # Sorry folks, I'm not up for decoding the destination path. watcher.handle_event(MOVED_SRC(self,path,None)) if inevt.mask & pyinotify.IN_MOVED_TO: if getattr(inevt,"src_pathname",None): watcher.handle_event(MOVED_SRC(self,src_path,path)) watcher.handle_event(MOVED_DST(self,path,src_path)) else: watcher.handle_event(MOVED_DST(self,path,None)) if inevt.mask & pyinotify.IN_Q_OVERFLOW: watcher.handle_event(OVERFLOW(self)) if inevt.mask & pyinotify.IN_UNMOUNT: watcher.handle_event(CLOSED(self)) def __get_watch_thread(self): """Get the shared watch thread, initializing if necessary. This method must only be called while holding self.__watch_lock, or multiple notifiers could be created. """ if OSFSWatchMixin.__watch_thread is None: OSFSWatchMixin.__watch_thread = SharedThreadedNotifier() OSFSWatchMixin.__watch_thread.start() return OSFSWatchMixin.__watch_thread class SharedThreadedNotifier(threading.Thread): """pyinotifer Notifier that can manage multiple WatchManagers. Each watcher added to an OSFS corresponds to a new pyinotify.WatchManager instance. Rather than run a notifier thread for each manager, we run a single thread that multiplexes between them all. """ def __init__(self): super(SharedThreadedNotifier,self).__init__() self.daemon = True self.running = True self._pipe_r, self._pipe_w = os.pipe() self._poller = select.poll() self._poller.register(self._pipe_r,select.POLLIN) self.watchers = {} def add_watcher(self,watcher): fd = watcher._pyinotify_WatchManager.get_fd() self.watchers[fd] = watcher self._poller.register(fd,select.POLLIN) # Bump the poll object so it recognises the new fd. os.write(self._pipe_w,b"H") def del_watcher(self,watcher): fd = watcher._pyinotify_WatchManager.get_fd() try: del self.watchers[fd] except KeyError: pass else: self._poller.unregister(fd) def run(self): # Grab some attributes of the select module, so they're available # even when shutting down the interpreter. _select_error = select.error _select_POLLIN = select.POLLIN # Loop until stopped, dispatching to individual notifiers. while self.running: try: ready_fds = self._poller.poll() except _select_error, e: if e[0] != errno.EINTR: raise else: for (fd,event) in ready_fds: # Ignore all events other than "input ready". if not event & _select_POLLIN: continue # For signals on our internal pipe, just read and discard. if fd == self._pipe_r: os.read(self._pipe_r,1) # For notifier fds, dispath to the notifier methods. else: try: notifier = self.watchers[fd]._pyinotify_Notifier except KeyError: pass else: notifier.read_events() try: notifier.process_events() except EnvironmentError: pass def stop(self): if self.running: self.running = False os.write(self._pipe_w,"S") os.close(self._pipe_w) fs-0.5.4/fs/osfs/watch.py0000664000175000017500000000165112512525115015136 0ustar willwill00000000000000""" fs.osfs.watch ============= Change watcher support for OSFS """ import os import sys import errno import threading from fs.errors import * from fs.path import * from fs.watch import * OSFSWatchMixin = None # Try using native implementation on win32 if sys.platform == "win32": try: from fs.osfs.watch_win32 import OSFSWatchMixin except ImportError: pass # Try using pyinotify if available if OSFSWatchMixin is None: try: from fs.osfs.watch_inotify import OSFSWatchMixin except ImportError: pass # Fall back to raising UnsupportedError if OSFSWatchMixin is None: class OSFSWatchMixin(object): def __init__(self, *args, **kwargs): super(OSFSWatchMixin, self).__init__(*args, **kwargs) def add_watcher(self,*args,**kwds): raise UnsupportedError def del_watcher(self,watcher_or_callback): raise UnsupportedError fs-0.5.4/fs/osfs/watch_win32.py0000664000175000017500000004115712512525115016165 0ustar willwill00000000000000""" fs.osfs.watch_win32 =================== Change watcher support for OSFS, using ReadDirectoryChangesW on win32. """ import os import sys import errno import threading import Queue import stat import struct import ctypes import ctypes.wintypes import traceback import weakref try: LPVOID = ctypes.wintypes.LPVOID except AttributeError: # LPVOID wasn't defined in Py2.5, guess it was introduced in Py2.6 LPVOID = ctypes.c_void_p from fs.errors import * from fs.path import * from fs.watch import * INVALID_HANDLE_VALUE = 0xFFFFFFFF FILE_NOTIFY_CHANGE_FILE_NAME = 0x01 FILE_NOTIFY_CHANGE_DIR_NAME = 0x02 FILE_NOTIFY_CHANGE_ATTRIBUTES = 0x04 FILE_NOTIFY_CHANGE_SIZE = 0x08 FILE_NOTIFY_CHANGE_LAST_WRITE = 0x010 FILE_NOTIFY_CHANGE_LAST_ACCESS = 0x020 FILE_NOTIFY_CHANGE_CREATION = 0x040 FILE_NOTIFY_CHANGE_SECURITY = 0x0100 FILE_LIST_DIRECTORY = 0x01 FILE_SHARE_READ = 0x01 FILE_SHARE_WRITE = 0x02 OPEN_EXISTING = 3 FILE_FLAG_BACKUP_SEMANTICS = 0x02000000 FILE_FLAG_OVERLAPPED = 0x40000000 THREAD_TERMINATE = 0x0001 FILE_ACTION_ADDED = 1 FILE_ACTION_REMOVED = 2 FILE_ACTION_MODIFIED = 3 FILE_ACTION_RENAMED_OLD_NAME = 4 FILE_ACTION_RENAMED_NEW_NAME = 5 FILE_ACTION_OVERFLOW = 0xFFFF WAIT_ABANDONED = 0x00000080 WAIT_IO_COMPLETION = 0x000000C0 WAIT_OBJECT_0 = 0x00000000 WAIT_TIMEOUT = 0x00000102 def _errcheck_bool(value,func,args): if not value: raise ctypes.WinError() return args def _errcheck_handle(value,func,args): if not value: raise ctypes.WinError() if value == INVALID_HANDLE_VALUE: raise ctypes.WinError() return args def _errcheck_dword(value,func,args): if value == 0xFFFFFFFF: raise ctypes.WinError() return args class OVERLAPPED(ctypes.Structure): _fields_ = [('Internal', LPVOID), ('InternalHigh', LPVOID), ('Offset', ctypes.wintypes.DWORD), ('OffsetHigh', ctypes.wintypes.DWORD), ('Pointer', LPVOID), ('hEvent', ctypes.wintypes.HANDLE), ] try: ReadDirectoryChangesW = ctypes.windll.kernel32.ReadDirectoryChangesW except AttributeError: raise ImportError("ReadDirectoryChangesW is not available") ReadDirectoryChangesW.restype = ctypes.wintypes.BOOL ReadDirectoryChangesW.errcheck = _errcheck_bool ReadDirectoryChangesW.argtypes = ( ctypes.wintypes.HANDLE, # hDirectory LPVOID, # lpBuffer ctypes.wintypes.DWORD, # nBufferLength ctypes.wintypes.BOOL, # bWatchSubtree ctypes.wintypes.DWORD, # dwNotifyFilter ctypes.POINTER(ctypes.wintypes.DWORD), # lpBytesReturned ctypes.POINTER(OVERLAPPED), # lpOverlapped LPVOID #FileIOCompletionRoutine # lpCompletionRoutine ) CreateFileW = ctypes.windll.kernel32.CreateFileW CreateFileW.restype = ctypes.wintypes.HANDLE CreateFileW.errcheck = _errcheck_handle CreateFileW.argtypes = ( ctypes.wintypes.LPCWSTR, # lpFileName ctypes.wintypes.DWORD, # dwDesiredAccess ctypes.wintypes.DWORD, # dwShareMode LPVOID, # lpSecurityAttributes ctypes.wintypes.DWORD, # dwCreationDisposition ctypes.wintypes.DWORD, # dwFlagsAndAttributes ctypes.wintypes.HANDLE # hTemplateFile ) CloseHandle = ctypes.windll.kernel32.CloseHandle CloseHandle.restype = ctypes.wintypes.BOOL CloseHandle.argtypes = ( ctypes.wintypes.HANDLE, # hObject ) CreateEvent = ctypes.windll.kernel32.CreateEventW CreateEvent.restype = ctypes.wintypes.HANDLE CreateEvent.errcheck = _errcheck_handle CreateEvent.argtypes = ( LPVOID, # lpEventAttributes ctypes.wintypes.BOOL, # bManualReset ctypes.wintypes.BOOL, # bInitialState ctypes.wintypes.LPCWSTR, #lpName ) SetEvent = ctypes.windll.kernel32.SetEvent SetEvent.restype = ctypes.wintypes.BOOL SetEvent.errcheck = _errcheck_bool SetEvent.argtypes = ( ctypes.wintypes.HANDLE, # hEvent ) WaitForSingleObjectEx = ctypes.windll.kernel32.WaitForSingleObjectEx WaitForSingleObjectEx.restype = ctypes.wintypes.DWORD WaitForSingleObjectEx.errcheck = _errcheck_dword WaitForSingleObjectEx.argtypes = ( ctypes.wintypes.HANDLE, # hObject ctypes.wintypes.DWORD, # dwMilliseconds ctypes.wintypes.BOOL, # bAlertable ) CreateIoCompletionPort = ctypes.windll.kernel32.CreateIoCompletionPort CreateIoCompletionPort.restype = ctypes.wintypes.HANDLE CreateIoCompletionPort.errcheck = _errcheck_handle CreateIoCompletionPort.argtypes = ( ctypes.wintypes.HANDLE, # FileHandle ctypes.wintypes.HANDLE, # ExistingCompletionPort LPVOID, # CompletionKey ctypes.wintypes.DWORD, # NumberOfConcurrentThreads ) GetQueuedCompletionStatus = ctypes.windll.kernel32.GetQueuedCompletionStatus GetQueuedCompletionStatus.restype = ctypes.wintypes.BOOL GetQueuedCompletionStatus.errcheck = _errcheck_bool GetQueuedCompletionStatus.argtypes = ( ctypes.wintypes.HANDLE, # CompletionPort LPVOID, # lpNumberOfBytesTransferred LPVOID, # lpCompletionKey ctypes.POINTER(OVERLAPPED), # lpOverlapped ctypes.wintypes.DWORD, # dwMilliseconds ) PostQueuedCompletionStatus = ctypes.windll.kernel32.PostQueuedCompletionStatus PostQueuedCompletionStatus.restype = ctypes.wintypes.BOOL PostQueuedCompletionStatus.errcheck = _errcheck_bool PostQueuedCompletionStatus.argtypes = ( ctypes.wintypes.HANDLE, # CompletionPort ctypes.wintypes.DWORD, # lpNumberOfBytesTransferred ctypes.wintypes.DWORD, # lpCompletionKey ctypes.POINTER(OVERLAPPED), # lpOverlapped ) class WatchedDirectory(object): def __init__(self,callback,path,flags,recursive=True): self.path = path self.flags = flags self.callback = callback self.recursive = recursive self.handle = None self.error = None self.handle = CreateFileW(path, FILE_LIST_DIRECTORY, FILE_SHARE_READ | FILE_SHARE_WRITE, None, OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS|FILE_FLAG_OVERLAPPED, None) self.result = ctypes.create_string_buffer(1024) self.overlapped = overlapped = OVERLAPPED() self.ready = threading.Event() def __del__(self): self.close() def close(self): if self.handle is not None: CloseHandle(self.handle) self.handle = None def post(self): overlapped = self.overlapped overlapped.Internal = 0 overlapped.InternalHigh = 0 overlapped.Offset = 0 overlapped.OffsetHigh = 0 overlapped.Pointer = 0 overlapped.hEvent = 0 try: ReadDirectoryChangesW(self.handle, ctypes.byref(self.result),len(self.result), self.recursive,self.flags,None, overlapped,None) except WindowsError, e: self.error = e self.close() def complete(self,nbytes): if nbytes == 0: self.callback(None,0) else: res = self.result.raw[:nbytes] for (name,action) in self._extract_change_info(res): if self.callback: self.callback(os.path.join(self.path,name),action) def _extract_change_info(self,buffer): """Extract the information out of a FILE_NOTIFY_INFORMATION structure.""" pos = 0 while pos < len(buffer): jump, action, namelen = struct.unpack("iii",buffer[pos:pos+12]) # TODO: this may return a shortname or a longname, with no way # to tell which. Normalise them somehow? name = buffer[pos+12:pos+12+namelen].decode("utf16") yield (name,action) if not jump: break pos += jump class WatchThread(threading.Thread): """Thread for watching filesystem changes.""" def __init__(self): super(WatchThread,self).__init__() self.closed = False self.watched_directories = {} self.ready = threading.Event() self._iocp = None self._new_watches = Queue.Queue() def close(self): if not self.closed: self.closed = True if self._iocp: PostQueuedCompletionStatus(self._iocp,0,1,None) def add_watcher(self,callback,path,events,recursive): if os.path.isfile(path): path = os.path.dirname(path) watched_dirs = [] for w in self._get_watched_dirs(callback,path,events,recursive): self.attach_watched_directory(w) watched_dirs.append(w) return watched_dirs def del_watcher(self,w): w = self.watched_directories.pop(hash(w)) w.callback = None w.close() def _get_watched_dirs(self,callback,path,events,recursive): do_access = False do_change = False flags = 0 for evt in events: if issubclass(ACCESSED,evt): do_access = True if issubclass(MODIFIED,evt): do_change = True flags |= FILE_NOTIFY_CHANGE_ATTRIBUTES flags |= FILE_NOTIFY_CHANGE_CREATION flags |= FILE_NOTIFY_CHANGE_SECURITY if issubclass(CREATED,evt): flags |= FILE_NOTIFY_CHANGE_FILE_NAME flags |= FILE_NOTIFY_CHANGE_DIR_NAME if issubclass(REMOVED,evt): flags |= FILE_NOTIFY_CHANGE_FILE_NAME flags |= FILE_NOTIFY_CHANGE_DIR_NAME if issubclass(MOVED_SRC,evt): flags |= FILE_NOTIFY_CHANGE_FILE_NAME flags |= FILE_NOTIFY_CHANGE_DIR_NAME if issubclass(MOVED_DST,evt): flags |= FILE_NOTIFY_CHANGE_FILE_NAME flags |= FILE_NOTIFY_CHANGE_DIR_NAME if do_access: # Separately capture FILE_NOTIFY_CHANGE_LAST_ACCESS events # so we can reliably generate ACCESSED events. def on_access_event(path,action): if action == FILE_ACTION_OVERFLOW: callback(OVERFLOW,path) else: callback(ACCESSED,path) yield WatchedDirectory(on_access_event,path, FILE_NOTIFY_CHANGE_LAST_ACCESS,recursive) if do_change: # Separately capture FILE_NOTIFY_CHANGE_LAST_WRITE events # so we can generate MODIFIED(data_changed=True) events. cflags = FILE_NOTIFY_CHANGE_LAST_WRITE | FILE_NOTIFY_CHANGE_SIZE def on_change_event(path,action): if action == FILE_ACTION_OVERFLOW: callback(OVERFLOW,path) else: callback(MODIFIED,path,True) yield WatchedDirectory(on_change_event,path,cflags,recursive) if flags: # All other events we can route through a common handler. old_name = [None] def on_misc_event(path,action): if action == FILE_ACTION_OVERFLOW: callback(OVERFLOW,path) elif action == FILE_ACTION_ADDED: callback(CREATED,path) elif action == FILE_ACTION_REMOVED: callback(REMOVED,path) elif action == FILE_ACTION_MODIFIED: callback(MODIFIED,path) elif action == FILE_ACTION_RENAMED_OLD_NAME: old_name[0] = path elif action == FILE_ACTION_RENAMED_NEW_NAME: callback(MOVED_DST,path,old_name[0]) callback(MOVED_SRC,old_name[0],path) old_name[0] = None yield WatchedDirectory(on_misc_event,path,flags,recursive) def run(self): try: self._iocp = CreateIoCompletionPort(INVALID_HANDLE_VALUE,None,0,1) self.ready.set() nbytes = ctypes.wintypes.DWORD() iocpkey = ctypes.wintypes.DWORD() overlapped = OVERLAPPED() while not self.closed: try: GetQueuedCompletionStatus(self._iocp, ctypes.byref(nbytes), ctypes.byref(iocpkey), ctypes.byref(overlapped), -1) except WindowsError: traceback.print_exc() else: if iocpkey.value > 1: try: w = self.watched_directories[iocpkey.value] except KeyError: pass else: w.complete(nbytes.value) w.post() elif not self.closed: try: while True: w = self._new_watches.get_nowait() if w.handle is not None: CreateIoCompletionPort(w.handle, self._iocp, hash(w),0) w.post() w.ready.set() except Queue.Empty: pass finally: self.ready.set() for w in self.watched_directories.itervalues(): w.close() if self._iocp: CloseHandle(self._iocp) def attach_watched_directory(self,w): self.watched_directories[hash(w)] = w self._new_watches.put(w) PostQueuedCompletionStatus(self._iocp,0,1,None) w.ready.wait() class OSFSWatchMixin(WatchableFSMixin): """Mixin providing change-watcher support via pyinotify.""" __watch_lock = threading.Lock() __watch_thread = None def close(self): super(OSFSWatchMixin,self).close() self.__shutdown_watch_thread(force=True) self.notify_watchers(CLOSED) @convert_os_errors def add_watcher(self,callback,path="/",events=None,recursive=True): w = super(OSFSWatchMixin,self).add_watcher(callback,path,events,recursive) syspath = self.getsyspath(path) wt = self.__get_watch_thread() # Careful not to create a reference cycle here. weak_self = weakref.ref(self) def handle_event(event_class,path,*args,**kwds): selfref = weak_self() if selfref is None: return try: path = selfref.unsyspath(path) except ValueError: pass else: if event_class in (MOVED_SRC,MOVED_DST) and args and args[0]: args = (selfref.unsyspath(args[0]),) + args[1:] event = event_class(selfref,path,*args,**kwds) w.handle_event(event) w._watch_objs = wt.add_watcher(handle_event,syspath,w.events,w.recursive) for wd in w._watch_objs: if wd.error is not None: self.del_watcher(w) raise wd.error return w @convert_os_errors def del_watcher(self,watcher_or_callback): wt = self.__get_watch_thread() if isinstance(watcher_or_callback,Watcher): watchers = [watcher_or_callback] else: watchers = self._find_watchers(watcher_or_callback) for watcher in watchers: for wobj in watcher._watch_objs: wt.del_watcher(wobj) super(OSFSWatchMixin,self).del_watcher(watcher) if not wt.watched_directories: self.__shutdown_watch_thread() def __get_watch_thread(self): """Get the shared watch thread, initializing if necessary.""" if self.__watch_thread is None: self.__watch_lock.acquire() try: if self.__watch_thread is None: wt = WatchThread() wt.start() wt.ready.wait() OSFSWatchMixin.__watch_thread = wt finally: self.__watch_lock.release() return self.__watch_thread def __shutdown_watch_thread(self,force=False): """Stop the shared watch manager, if there are no watches left.""" self.__watch_lock.acquire() try: if OSFSWatchMixin.__watch_thread is None: return if not force and OSFSWatchMixin.__watch_thread.watched_directories: return try: OSFSWatchMixin.__watch_thread.close() except EnvironmentError: pass else: OSFSWatchMixin.__watch_thread.join() OSFSWatchMixin.__watch_thread = None finally: self.__watch_lock.release() fs-0.5.4/fs/osfs/__init__.py0000664000175000017500000003630212621460675015602 0ustar willwill00000000000000""" fs.osfs ======= Exposes the OS Filesystem as an FS object. For example, to print all the files and directories in the OS root:: >>> from fs.osfs import OSFS >>> home_fs = OSFS('/') >>> print home_fs.listdir() """ import os import os.path from os.path import exists as _exists, isdir as _isdir, isfile as _isfile import sys import errno import datetime import platform import io import shutil scandir = None try: scandir = os.scandir except AttributeError: try: from scandir import scandir except ImportError: pass from fs.base import * from fs.path import * from fs.errors import * from fs import _thread_synchronize_default from fs.osfs.xattrs import OSFSXAttrMixin from fs.osfs.watch import OSFSWatchMixin @convert_os_errors def _os_stat(path): """Replacement for os.stat that raises FSError subclasses.""" return os.stat(path) @convert_os_errors def _os_mkdir(name, mode=0777): """Replacement for os.mkdir that raises FSError subclasses.""" return os.mkdir(name, mode) @convert_os_errors def _os_makedirs(name, mode=0777): """Replacement for os.makdirs that raises FSError subclasses. This implementation also correctly handles win32 long filenames (those starting with "\\\\?\\") which can confuse os.makedirs(). The difficulty is that a long-name drive reference like "\\\\?\\C:\\" must end with a backslash to be considered a valid path, but os.makedirs() strips them. """ head, tail = os.path.split(name) while not tail: head, tail = os.path.split(head) if sys.platform == "win32" and len(head) == 6: if head.startswith("\\\\?\\"): head = head + "\\" if head and tail and not os.path.exists(head): try: _os_makedirs(head, mode) except OSError, e: if e.errno != errno.EEXIST: raise if tail == os.curdir: return os.mkdir(name, mode) class OSFS(OSFSXAttrMixin, OSFSWatchMixin, FS): """Expose the underlying operating-system filesystem as an FS object. This is the most basic of filesystems, which simply shadows the underlaying filesystem of the OS. Most of its methods simply defer to the matching methods in the os and os.path modules. """ _meta = {'thread_safe': True, 'network': False, 'virtual': False, 'read_only': False, 'unicode_paths': os.path.supports_unicode_filenames, 'case_insensitive_paths': os.path.normcase('Aa') == 'aa', 'atomic.makedir': True, 'atomic.rename': True, 'atomic.setcontents': False} if platform.system() == 'Windows': _meta["invalid_path_chars"] = ''.join(chr(n) for n in xrange(31)) + '\\:*?"<>|' else: _meta["invalid_path_chars"] = '\0' def __init__(self, root_path, thread_synchronize=_thread_synchronize_default, encoding=None, create=False, dir_mode=0700, use_long_paths=True): """ Creates an FS object that represents the OS Filesystem under a given root path :param root_path: The root OS path :param thread_synchronize: If True, this object will be thread-safe by use of a threading.Lock object :param encoding: The encoding method for path strings :param create: If True, then root_path will be created if it doesn't already exist :param dir_mode: The mode to use when creating the directory """ super(OSFS, self).__init__(thread_synchronize=thread_synchronize) self.encoding = encoding or sys.getfilesystemencoding() or 'utf-8' self.dir_mode = dir_mode self.use_long_paths = use_long_paths root_path = os.path.expanduser(os.path.expandvars(root_path)) root_path = os.path.normpath(os.path.abspath(root_path)) # Enable long pathnames on win32 if sys.platform == "win32": if use_long_paths and not root_path.startswith("\\\\?\\"): if not root_path.startswith("\\"): root_path = u"\\\\?\\" + root_path else: # Explicitly mark UNC paths, seems to work better. if root_path.startswith("\\\\"): root_path = u"\\\\?\\UNC\\" + root_path[2:] else: root_path = u"\\\\?" + root_path # If it points at the root of a drive, it needs a trailing slash. if len(root_path) == 6 and not root_path.endswith("\\"): root_path = root_path + "\\" if create: try: _os_makedirs(root_path, mode=dir_mode) except (OSError, DestinationExistsError): pass if not os.path.exists(root_path): raise ResourceNotFoundError(root_path, msg="Root directory does not exist: %(path)s") if not os.path.isdir(root_path): raise ResourceInvalidError(root_path, msg="Root path is not a directory: %(path)s") self.root_path = root_path self.dir_mode = dir_mode def __str__(self): return "" % self.root_path def __repr__(self): return "" % self.root_path def __unicode__(self): return u"" % self.root_path def _decode_path(self, p): if isinstance(p, unicode): return p return p.decode(self.encoding, 'replace') def getsyspath(self, path, allow_none=False): self.validatepath(path) path = relpath(normpath(path)).replace(u"/", os.sep) path = os.path.join(self.root_path, path) if not path.startswith(self.root_path): raise PathError(path, msg="OSFS given path outside root: %(path)s") path = self._decode_path(path) return path def unsyspath(self, path): """Convert a system-level path into an FS-level path. This basically the reverse of getsyspath(). If the path does not refer to a location within this filesystem, ValueError is raised. :param path: a system path :returns: a path within this FS object :rtype: string """ # TODO: HAve a closer look at this method path = os.path.normpath(os.path.abspath(path)) path = self._decode_path(path) if sys.platform == "win32": if len(path) == 6 and not path.endswith("\\"): path = path + "\\" prefix = os.path.normcase(self.root_path) if not prefix.endswith(os.path.sep): prefix += os.path.sep if not os.path.normcase(path).startswith(prefix): raise ValueError("path not within this FS: %s (%s)" % (os.path.normcase(path), prefix)) return normpath(path[len(self.root_path):]) def getmeta(self, meta_name, default=NoDefaultMeta): if meta_name == 'free_space': if platform.system() == 'Windows': try: import ctypes free_bytes = ctypes.c_ulonglong(0) ctypes.windll.kernel32.GetDiskFreeSpaceExW(ctypes.c_wchar_p(self.root_path), None, None, ctypes.pointer(free_bytes)) return free_bytes.value except ImportError: # Fall through to call the base class pass else: stat = os.statvfs(self.root_path) return stat.f_bfree * stat.f_bsize elif meta_name == 'total_space': if platform.system() == 'Windows': try: import ctypes total_bytes = ctypes.c_ulonglong(0) ctypes.windll.kernel32.GetDiskFreeSpaceExW(ctypes.c_wchar_p(self.root_path), None, ctypes.pointer(total_bytes), None) return total_bytes.value except ImportError: # Fall through to call the base class pass else: stat = os.statvfs(self.root_path) return stat.f_blocks * stat.f_bsize return super(OSFS, self).getmeta(meta_name, default) @convert_os_errors def open(self, path, mode='r', buffering=-1, encoding=None, errors=None, newline=None, line_buffering=False, **kwargs): mode = ''.join(c for c in mode if c in 'rwabt+') sys_path = self.getsyspath(path) if not encoding and 'b' not in mode: encoding = encoding or 'utf-8' try: return io.open(sys_path, mode=mode, buffering=buffering, encoding=encoding, errors=errors, newline=newline) except EnvironmentError, e: # Win32 gives EACCES when opening a directory. if sys.platform == "win32" and e.errno in (errno.EACCES,): if self.isdir(path): raise ResourceInvalidError(path) raise @convert_os_errors def setcontents(self, path, data=b'', encoding=None, errors=None, chunk_size=64 * 1024): return super(OSFS, self).setcontents(path, data, encoding=encoding, errors=errors, chunk_size=chunk_size) @convert_os_errors def exists(self, path): return _exists(self.getsyspath(path)) @convert_os_errors def isdir(self, path): return _isdir(self.getsyspath(path)) @convert_os_errors def isfile(self, path): return _isfile(self.getsyspath(path)) @convert_os_errors def listdir(self, path="./", wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False): _decode_path = self._decode_path sys_path = self.getsyspath(path) if scandir is None: listing = os.listdir(sys_path) paths = [_decode_path(p) for p in listing] return self._listdir_helper(path, paths, wildcard, full, absolute, dirs_only, files_only) else: if dirs_only and files_only: raise ValueError("dirs_only and files_only can not both be True") # Use optimized scandir if present scan = scandir(sys_path) if dirs_only: paths = [_decode_path(dir_entry.name) for dir_entry in scan if dir_entry.is_dir()] elif files_only: paths = [_decode_path(dir_entry.name) for dir_entry in scan if dir_entry.is_file()] else: paths = [_decode_path(dir_entry.name) for dir_entry in scan] return self._listdir_helper(path, paths, wildcard, full, absolute, False, False) @convert_os_errors def makedir(self, path, recursive=False, allow_recreate=False): sys_path = self.getsyspath(path) try: if recursive: _os_makedirs(sys_path, self.dir_mode) else: _os_mkdir(sys_path, self.dir_mode) except DestinationExistsError: if self.isfile(path): raise ResourceInvalidError(path, msg="Cannot create directory, there's already a file of that name: %(path)s") if not allow_recreate: raise DestinationExistsError(path, msg="Can not create a directory that already exists (try allow_recreate=True): %(path)s") except ResourceNotFoundError: raise ParentDirectoryMissingError(path) @convert_os_errors def remove(self, path): sys_path = self.getsyspath(path) try: os.remove(sys_path) except OSError, e: if e.errno == errno.EACCES and sys.platform == "win32": # sometimes windows says this for attempts to remove a dir if os.path.isdir(sys_path): raise ResourceInvalidError(path) if e.errno == errno.EPERM and sys.platform == "darwin": # sometimes OSX says this for attempts to remove a dir if os.path.isdir(sys_path): raise ResourceInvalidError(path) raise @convert_os_errors def removedir(self, path, recursive=False, force=False): # Don't remove the root directory of this FS if path in ('', '/'): raise RemoveRootError(path) sys_path = self.getsyspath(path) if force: # shutil implementation handles concurrency better shutil.rmtree(sys_path, ignore_errors=True) else: os.rmdir(sys_path) # Using os.removedirs() for this can result in dirs being # removed outside the root of this FS, so we recurse manually. if recursive: try: if dirname(path) not in ('', '/'): self.removedir(dirname(path), recursive=True) except DirectoryNotEmptyError: pass @convert_os_errors def rename(self, src, dst): path_src = self.getsyspath(src) path_dst = self.getsyspath(dst) try: os.rename(path_src, path_dst) except OSError, e: if e.errno: # POSIX rename() can rename over an empty directory but gives # ENOTEMPTY if the dir has contents. Raise UnsupportedError # instead of DirectoryEmptyError in this case. if e.errno == errno.ENOTEMPTY: raise UnsupportedError("rename") # Linux (at least) gives ENOENT when trying to rename into # a directory that doesn't exist. We want ParentMissingError # in this case. if e.errno == errno.ENOENT: if not os.path.exists(os.path.dirname(path_dst)): raise ParentDirectoryMissingError(dst) raise def _stat(self, path): """Stat the given path, normalising error codes.""" sys_path = self.getsyspath(path) try: return _os_stat(sys_path) except ResourceInvalidError: raise ResourceNotFoundError(path) @convert_os_errors def getinfo(self, path): stats = self._stat(path) info = dict((k, getattr(stats, k)) for k in dir(stats) if k.startswith('st_')) info['size'] = info['st_size'] # TODO: this doesn't actually mean 'creation time' on unix fromtimestamp = datetime.datetime.fromtimestamp ct = info.get('st_ctime', None) if ct is not None: info['created_time'] = fromtimestamp(ct) at = info.get('st_atime', None) if at is not None: info['accessed_time'] = fromtimestamp(at) mt = info.get('st_mtime', None) if mt is not None: info['modified_time'] = fromtimestamp(mt) return info @convert_os_errors def getinfokeys(self, path, *keys): info = {} stats = self._stat(path) fromtimestamp = datetime.datetime.fromtimestamp for key in keys: try: if key == 'size': info[key] = stats.st_size elif key == 'modified_time': info[key] = fromtimestamp(stats.st_mtime) elif key == 'created_time': info[key] = fromtimestamp(stats.st_ctime) elif key == 'accessed_time': info[key] = fromtimestamp(stats.st_atime) else: info[key] = getattr(stats, key) except AttributeError: continue return info @convert_os_errors def getsize(self, path): return self._stat(path).st_size fs-0.5.4/fs/multifs.py0000664000175000017500000002463212512525115014545 0ustar willwill00000000000000""" fs.multifs ========== A MultiFS is a filesystem composed of a sequence of other filesystems, where the directory structure of each filesystem is overlaid over the previous filesystem. When you attempt to access a file from the MultiFS it will try each 'child' FS in order, until it either finds a path that exists or raises a ResourceNotFoundError. One use for such a filesystem would be to selectively override a set of files, to customize behavior. For example, to create a filesystem that could be used to *theme* a web application. We start with the following directories:: `-- templates |-- snippets | `-- panel.html |-- index.html |-- profile.html `-- base.html `-- theme |-- snippets | |-- widget.html | `-- extra.html |-- index.html `-- theme.html And we want to create a single filesystem that looks for files in `templates` if they don't exist in `theme`. We can do this with the following code:: from fs.osfs import OSFS from fs.multifs import MultiFS themed_template_fs = MultiFS() themed_template_fs.addfs('templates', OSFS('templates')) themed_template_fs.addfs('theme', OSFS('theme')) Now we have a `themed_template_fs` FS object presents a single view of both directories:: |-- snippets | |-- panel.html | |-- widget.html | `-- extra.html |-- index.html |-- profile.html |-- base.html `-- theme.html A MultiFS is generally read-only, and any operation that may modify data (including opening files for writing) will fail. However, you can set a writeable fs with the `setwritefs` method -- which does not have to be one of the FS objects set with `addfs`. The reason that only one FS object is ever considered for write access is that otherwise it would be ambiguous as to which filesystem you would want to modify. If you need to be able to modify more than one FS in the MultiFS, you can always access them directly. """ from fs.base import FS, synchronize from fs.path import * from fs.errors import * from fs import _thread_synchronize_default class MultiFS(FS): """A filesystem that delegates to a sequence of other filesystems. Operations on the MultiFS will try each 'child' filesystem in order, until it succeeds. In effect, creating a filesystem that combines the files and dirs of its children. """ _meta = { 'virtual': True, 'read_only' : False, 'unicode_paths' : True, 'case_insensitive_paths' : False } def __init__(self, auto_close=True): """ :param auto_close: If True the child filesystems will be closed when the MultiFS is closed """ super(MultiFS, self).__init__(thread_synchronize=_thread_synchronize_default) self.auto_close = auto_close self.fs_sequence = [] self.fs_lookup = {} self.fs_priorities = {} self.writefs = None @synchronize def __str__(self): return "" % ", ".join(str(fs) for fs in self.fs_sequence) __repr__ = __str__ @synchronize def __unicode__(self): return u"" % ", ".join(unicode(fs) for fs in self.fs_sequence) def _get_priority(self, name): return self.fs_priorities[name] @synchronize def close(self): # Explicitly close if requested if self.auto_close: for fs in self.fs_sequence: fs.close() if self.writefs is not None: self.writefs.close() # Discard any references del self.fs_sequence[:] self.fs_lookup.clear() self.fs_priorities.clear() self.writefs = None super(MultiFS, self).close() def _priority_sort(self): """Sort filesystems by priority order""" priority_order = sorted(self.fs_lookup.keys(), key=lambda n: self.fs_priorities[n], reverse=True) self.fs_sequence = [self.fs_lookup[name] for name in priority_order] @synchronize def addfs(self, name, fs, write=False, priority=0): """Adds a filesystem to the MultiFS. :param name: A unique name to refer to the filesystem being added. The filesystem can later be retrieved by using this name as an index to the MultiFS, i.e. multifs['myfs'] :param fs: The filesystem to add :param write: If this value is True, then the `fs` will be used as the writeable FS :param priority: A number that gives the priorty of the filesystem being added. Filesystems will be searched in descending priority order and then by the reverse order they were added. So by default, the most recently added filesystem will be looked at first """ if name in self.fs_lookup: raise ValueError("Name already exists.") priority = (priority, len(self.fs_sequence)) self.fs_priorities[name] = priority self.fs_sequence.append(fs) self.fs_lookup[name] = fs self._priority_sort() if write: self.setwritefs(fs) @synchronize def setwritefs(self, fs): """Sets the filesystem to use when write access is required. Without a writeable FS, any operations that could modify data (including opening files for writing / appending) will fail. :param fs: An FS object that will be used to open writeable files """ self.writefs = fs @synchronize def clearwritefs(self): """Clears the writeable filesystem (operations that modify the multifs will fail)""" self.writefs = None @synchronize def removefs(self, name): """Removes a filesystem from the sequence. :param name: The name of the filesystem, as used in addfs """ if name not in self.fs_lookup: raise ValueError("No filesystem called '%s'" % name) fs = self.fs_lookup[name] self.fs_sequence.remove(fs) del self.fs_lookup[name] self._priority_sort() @synchronize def __getitem__(self, name): return self.fs_lookup[name] @synchronize def __iter__(self): return iter(self.fs_sequence[:]) def _delegate_search(self, path): for fs in self: if fs.exists(path): return fs return None @synchronize def which(self, path, mode='r'): """Retrieves the filesystem that a given path would delegate to. Returns a tuple of the filesystem's name and the filesystem object itself. :param path: A path in MultiFS """ if 'w' in mode or '+' in mode or 'a' in mode: return self.writefs for fs in self: if fs.exists(path): for fs_name, fs_object in self.fs_lookup.iteritems(): if fs is fs_object: return fs_name, fs raise ResourceNotFoundError(path, msg="Path does not map to any filesystem: %(path)s") @synchronize def getsyspath(self, path, allow_none=False): fs = self._delegate_search(path) if fs is not None: return fs.getsyspath(path, allow_none=allow_none) if allow_none: return None raise ResourceNotFoundError(path) @synchronize def desc(self, path): if not self.exists(path): raise ResourceNotFoundError(path) name, fs = self.which(path) if name is None: return "" return "%s (in %s)" % (fs.desc(path), name) @synchronize def open(self, path, mode='r', buffering=-1, encoding=None, errors=None, newline=None, line_buffering=False, **kwargs): if 'w' in mode or '+' in mode or 'a' in mode: if self.writefs is None: raise OperationFailedError('open', path=path, msg="No writeable FS set") return self.writefs.open(path, mode=mode, buffering=buffering, encoding=encoding, errors=errors, newline=newline, line_buffering=line_buffering, **kwargs) for fs in self: if fs.exists(path): fs_file = fs.open(path, mode=mode, buffering=buffering, encoding=encoding, errors=errors, newline=newline, line_buffering=line_buffering, **kwargs) return fs_file raise ResourceNotFoundError(path) @synchronize def exists(self, path): return self._delegate_search(path) is not None @synchronize def isdir(self, path): fs = self._delegate_search(path) if fs is not None: return fs.isdir(path) return False @synchronize def isfile(self, path): fs = self._delegate_search(path) if fs is not None: return fs.isfile(path) return False @synchronize def listdir(self, path="./", *args, **kwargs): paths = [] for fs in self: try: paths += fs.listdir(path, *args, **kwargs) except FSError: pass return list(set(paths)) @synchronize def makedir(self, path, recursive=False, allow_recreate=False): if self.writefs is None: raise OperationFailedError('makedir', path=path, msg="No writeable FS set") self.writefs.makedir(path, recursive=recursive, allow_recreate=allow_recreate) @synchronize def remove(self, path): if self.writefs is None: raise OperationFailedError('remove', path=path, msg="No writeable FS set") self.writefs.remove(path) @synchronize def removedir(self, path, recursive=False, force=False): if self.writefs is None: raise OperationFailedError('removedir', path=path, msg="No writeable FS set") if normpath(path) in ('', '/'): raise RemoveRootError(path) self.writefs.removedir(path, recursive=recursive, force=force) @synchronize def rename(self, src, dst): if self.writefs is None: raise OperationFailedError('rename', path=src, msg="No writeable FS set") self.writefs.rename(src, dst) @synchronize def settimes(self, path, accessed_time=None, modified_time=None): if self.writefs is None: raise OperationFailedError('settimes', path=path, msg="No writeable FS set") self.writefs.settimes(path, accessed_time, modified_time) @synchronize def getinfo(self, path): for fs in self: if fs.exists(path): return fs.getinfo(path) raise ResourceNotFoundError(path) fs-0.5.4/fs/s3fs.py0000664000175000017500000007113212621410706013735 0ustar willwill00000000000000""" fs.s3fs ======= **Currently only avaiable on Python2 due to boto not being available for Python3** FS subclass accessing files in Amazon S3 This module provides the class 'S3FS', which implements the FS filesystem interface for objects stored in Amazon Simple Storage Service (S3). """ import os import datetime import tempfile from fnmatch import fnmatch import stat as statinfo import boto.s3.connection from boto.s3.prefix import Prefix from boto.exception import S3ResponseError from fs.base import * from fs.path import * from fs.errors import * from fs.remote import * from fs.filelike import LimitBytesFile from fs import iotools import six # Boto is not thread-safe, so we need to use a per-thread S3 connection. if hasattr(threading,"local"): thread_local = threading.local else: class thread_local(object): def __init__(self): self._map = {} def __getattr__(self,attr): try: return self._map[(threading.currentThread(),attr)] except KeyError: raise AttributeError, attr def __setattr__(self,attr,value): self._map[(threading.currentThread(),attr)] = value class S3FS(FS): """A filesystem stored in Amazon S3. This class provides the FS interface for files stored in Amazon's Simple Storage Service (S3). It should be instantiated with the name of the S3 bucket to use, and optionally a prefix under which the files should be stored. Local temporary files are used when opening files from this filesystem, and any changes are only pushed back into S3 when the files are closed or flushed. """ _meta = {'thread_safe': True, 'virtual': False, 'read_only': False, 'unicode_paths': True, 'case_insensitive_paths': False, 'network': True, 'atomic.move': True, 'atomic.copy': True, 'atomic.makedir': True, 'atomic.rename': False, 'atomic.setcontents': True } class meta: PATH_MAX = None NAME_MAX = None def __init__(self, bucket, prefix="", aws_access_key=None, aws_secret_key=None, separator="/", thread_synchronize=True, key_sync_timeout=1): """Constructor for S3FS objects. S3FS objects require the name of the S3 bucket in which to store files, and can optionally be given a prefix under which the files should be stored. The AWS public and private keys may be specified as additional arguments; if they are not specified they will be read from the two environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. The keyword argument 'key_sync_timeout' specifies the maximum time in seconds that the filesystem will spend trying to confirm that a newly-uploaded S3 key is available for reading. For no timeout set it to zero. To disable these checks entirely (and thus reduce the filesystem's consistency guarantees to those of S3's "eventual consistency" model) set it to None. By default the path separator is "/", but this can be overridden by specifying the keyword 'separator' in the constructor. """ self._bucket_name = bucket self._access_keys = (aws_access_key,aws_secret_key) self._separator = separator self._key_sync_timeout = key_sync_timeout # Normalise prefix to this form: path/to/files/ prefix = normpath(prefix) while prefix.startswith(separator): prefix = prefix[1:] if not prefix.endswith(separator) and prefix != "": prefix = prefix + separator if isinstance(prefix,unicode): prefix = prefix.encode("utf8") if aws_access_key is None: if "AWS_ACCESS_KEY_ID" not in os.environ: raise CreateFailedError("AWS_ACCESS_KEY_ID not set") if aws_secret_key is None: if "AWS_SECRET_ACCESS_KEY" not in os.environ: raise CreateFailedError("AWS_SECRET_ACCESS_KEY not set") self._prefix = prefix self._tlocal = thread_local() super(S3FS, self).__init__(thread_synchronize=thread_synchronize) # Make _s3conn and _s3bukt properties that are created on demand, # since they cannot be stored during pickling. def _s3conn(self): try: (c,ctime) = self._tlocal.s3conn if time.time() - ctime > 60: raise AttributeError return c except AttributeError: c = boto.s3.connection.S3Connection(*self._access_keys) self._tlocal.s3conn = (c,time.time()) return c _s3conn = property(_s3conn) def _s3bukt(self): try: (b,ctime) = self._tlocal.s3bukt if time.time() - ctime > 60: raise AttributeError return b except AttributeError: try: # Validate by listing the bucket if there is no prefix. # If there is a prefix, validate by listing only the prefix # itself, to avoid errors when an IAM policy has been applied. if self._prefix: b = self._s3conn.get_bucket(self._bucket_name, validate=0) b.get_key(self._prefix) else: b = self._s3conn.get_bucket(self._bucket_name, validate=1) except S3ResponseError, e: if "404 Not Found" not in str(e): raise b = self._s3conn.create_bucket(self._bucket_name) self._tlocal.s3bukt = (b,time.time()) return b _s3bukt = property(_s3bukt) def __getstate__(self): state = super(S3FS,self).__getstate__() del state['_tlocal'] return state def __setstate__(self,state): super(S3FS,self).__setstate__(state) self._tlocal = thread_local() def __repr__(self): args = (self.__class__.__name__,self._bucket_name,self._prefix) return '<%s: %s:%s>' % args __str__ = __repr__ def _s3path(self,path): """Get the absolute path to a file stored in S3.""" path = relpath(normpath(path)) path = self._separator.join(iteratepath(path)) s3path = self._prefix + path if s3path and s3path[-1] == self._separator: s3path = s3path[:-1] if isinstance(s3path,unicode): s3path = s3path.encode("utf8") return s3path def _uns3path(self,s3path,roots3path=None): """Get the local path for a file stored in S3. This is essentially the opposite of self._s3path(). """ if roots3path is None: roots3path = self._s3path("") i = len(roots3path) return s3path[i:] def _sync_key(self,k): """Synchronise on contents of the given key. Since S3 only offers "eventual consistency" of data, it is possible to create a key but be unable to read it back straight away. This method works around that limitation by polling the key until it reads back the value expected by the given key. Note that this could easily fail if the key is modified by another program, meaning the content will never be as specified in the given key. This is the reason for the timeout argument to the construtcor. """ timeout = self._key_sync_timeout if timeout is None: return k k2 = self._s3bukt.get_key(k.name) t = time.time() while k2 is None or k2.etag != k.etag: if timeout > 0: if t + timeout < time.time(): break time.sleep(0.1) k2 = self._s3bukt.get_key(k.name) return k2 def _sync_set_contents(self,key,contents): """Synchronously set the contents of a key.""" if isinstance(key,basestring): key = self._s3bukt.new_key(key) if isinstance(contents,basestring): key.set_contents_from_string(contents) elif hasattr(contents,"md5"): hexmd5 = contents.md5 b64md5 = hexmd5.decode("hex").encode("base64").strip() key.set_contents_from_file(contents,md5=(hexmd5,b64md5)) else: try: contents.seek(0) except (AttributeError,EnvironmentError): tf = tempfile.TemporaryFile() data = contents.read(524288) while data: tf.write(data) data = contents.read(524288) tf.seek(0) key.set_contents_from_file(tf) else: key.set_contents_from_file(contents) return self._sync_key(key) def makepublic(self, path): """Mark given path as publicly accessible using HTTP(S)""" s3path = self._s3path(path) k = self._s3bukt.get_key(s3path) k.make_public() def getpathurl(self, path, allow_none=False, expires=3600): """Returns a url that corresponds to the given path.""" s3path = self._s3path(path) k = self._s3bukt.get_key(s3path) # Is there AllUsers group with READ permissions? is_public = True in [grant.permission == 'READ' and grant.uri == 'http://acs.amazonaws.com/groups/global/AllUsers' for grant in k.get_acl().acl.grants] url = k.generate_url(expires, force_http=is_public) if url == None: if not allow_none: raise NoPathURLError(path=path) return None if is_public: # Strip time token; it has no sense for public resource url = url.split('?')[0] return url def setcontents(self, path, data=b'', encoding=None, errors=None, chunk_size=64*1024): s3path = self._s3path(path) if isinstance(data, six.text_type): data = data.encode(encoding=encoding, errors=errors) self._sync_set_contents(s3path, data) @iotools.filelike_to_stream def open(self, path, mode='r', buffering=-1, encoding=None, errors=None, newline=None, line_buffering=False, **kwargs): """Open the named file in the given mode. This method downloads the file contents into a local temporary file so that it can be worked on efficiently. Any changes made to the file are only sent back to S3 when the file is flushed or closed. """ if self.isdir(path): raise ResourceInvalidError(path) s3path = self._s3path(path) # Truncate the file if requested if "w" in mode: k = self._sync_set_contents(s3path,"") else: k = self._s3bukt.get_key(s3path) if k is None: # Create the file if it's missing if "w" not in mode and "a" not in mode: raise ResourceNotFoundError(path) if not self.isdir(dirname(path)): raise ParentDirectoryMissingError(path) k = self._sync_set_contents(s3path,"") # Make sure nothing tries to read past end of socket data f = LimitBytesFile(k.size,k,"r") # For streaming reads, return the key object directly if mode == "r-": return f # For everything else, use a RemoteFileBuffer. # This will take care of closing the socket when it's done. return RemoteFileBuffer(self,path,mode,f) def exists(self,path): """Check whether a path exists.""" s3path = self._s3path(path) s3pathD = s3path + self._separator # The root directory always exists if self._prefix.startswith(s3path): return True ks = self._s3bukt.list(prefix=s3path,delimiter=self._separator) for k in ks: # A regular file if _eq_utf8(k.name,s3path): return True # A directory if _eq_utf8(k.name,s3pathD): return True return False def isdir(self,path): """Check whether a path exists and is a directory.""" s3path = self._s3path(path) + self._separator # Root is always a directory if s3path == "/" or s3path == self._prefix: return True # Use a list request so that we return true if there are any files # in that directory. This avoids requiring a special file for the # the directory itself, which other tools may not create. ks = self._s3bukt.list(prefix=s3path,delimiter=self._separator) try: iter(ks).next() except StopIteration: return False else: return True def isfile(self,path): """Check whether a path exists and is a regular file.""" s3path = self._s3path(path) # Root is never a file if self._prefix.startswith(s3path): return False k = self._s3bukt.get_key(s3path) if k is not None: return True return False def listdir(self,path="./",wildcard=None,full=False,absolute=False, dirs_only=False,files_only=False): """List contents of a directory.""" return list(self.ilistdir(path,wildcard,full,absolute, dirs_only,files_only)) def listdirinfo(self,path="./",wildcard=None,full=False,absolute=False, dirs_only=False,files_only=False): return list(self.ilistdirinfo(path,wildcard,full,absolute, dirs_only,files_only)) def ilistdir(self,path="./",wildcard=None,full=False,absolute=False, dirs_only=False,files_only=False): """List contents of a directory.""" keys = self._iter_keys(path) entries = self._filter_keys(path,keys,wildcard,full,absolute, dirs_only,files_only) return (nm for (nm,k) in entries) def ilistdirinfo(self,path="./",wildcard=None,full=False,absolute=False, dirs_only=False,files_only=False): keys = self._iter_keys(path) entries = self._filter_keys(path,keys,wildcard,full,absolute, dirs_only,files_only) return ((nm,self._get_key_info(k,nm)) for (nm,k) in entries) def _iter_keys(self,path): """Iterator over keys contained in the given directory. This generator yields (name,key) pairs for each entry in the given directory. If the path is not a directory, it raises the approprate error. """ s3path = self._s3path(path) + self._separator if s3path == "/": s3path = "" isDir = False for k in self._s3bukt.list(prefix=s3path,delimiter=self._separator): if not isDir: isDir = True # Skip over the entry for the directory itself, if it exists name = self._uns3path(k.name,s3path) if name != "": if not isinstance(name,unicode): name = name.decode("utf8") if name.endswith(self._separator): name = name[:-1] yield (name,k) if not isDir: if s3path != self._prefix: if self.isfile(path): msg = "that's not a directory: %(path)s" raise ResourceInvalidError(path,msg=msg) raise ResourceNotFoundError(path) def _key_is_dir(self, k): if isinstance(k,Prefix): return True if k.name.endswith(self._separator): return True return False def _filter_keys(self,path,keys,wildcard,full,absolute, dirs_only,files_only): """Filter out keys not matching the given criteria. Given a (name,key) iterator as returned by _iter_keys, this method applies the given filtering criteria and returns a filtered iterator. """ sep = self._separator if dirs_only and files_only: raise ValueError("dirs_only and files_only can not both be True") if dirs_only: keys = ((nm,k) for (nm,k) in keys if self._key_is_dir(k)) elif files_only: keys = ((nm,k) for (nm,k) in keys if not self._key_is_dir(k)) if wildcard is not None: if callable(wildcard): keys = ((nm,k) for (nm,k) in keys if wildcard(nm)) else: keys = ((nm,k) for (nm,k) in keys if fnmatch(nm,wildcard)) if full: return ((relpath(pathjoin(path, nm)),k) for (nm,k) in keys) elif absolute: return ((abspath(pathjoin(path, nm)),k) for (nm,k) in keys) return keys def makedir(self,path,recursive=False,allow_recreate=False): """Create a directory at the given path. The 'mode' argument is accepted for compatibility with the standard FS interface, but is currently ignored. """ s3path = self._s3path(path) s3pathD = s3path + self._separator if s3pathD == self._prefix: if allow_recreate: return msg = "Can not create a directory that already exists"\ " (try allow_recreate=True): %(path)s" raise DestinationExistsError(path, msg=msg) s3pathP = self._s3path(dirname(path)) if s3pathP: s3pathP = s3pathP + self._separator # Check various preconditions using list of parent dir ks = self._s3bukt.list(prefix=s3pathP,delimiter=self._separator) if s3pathP == self._prefix: parentExists = True else: parentExists = False for k in ks: if not parentExists: parentExists = True if _eq_utf8(k.name,s3path): # It's already a file msg = "Destination exists as a regular file: %(path)s" raise ResourceInvalidError(path, msg=msg) if _eq_utf8(k.name,s3pathD): # It's already a directory if allow_recreate: return msg = "Can not create a directory that already exists"\ " (try allow_recreate=True): %(path)s" raise DestinationExistsError(path, msg=msg) # Create parent if required if not parentExists: if recursive: self.makedir(dirname(path),recursive,allow_recreate) else: msg = "Parent directory does not exist: %(path)s" raise ParentDirectoryMissingError(path, msg=msg) # Create an empty file representing the directory if s3pathD not in ('/', ''): self._sync_set_contents(s3pathD,"") def remove(self,path): """Remove the file at the given path.""" s3path = self._s3path(path) ks = self._s3bukt.list(prefix=s3path,delimiter=self._separator) for k in ks: if _eq_utf8(k.name,s3path): break if _startswith_utf8(k.name,s3path + "/"): msg = "that's not a file: %(path)s" raise ResourceInvalidError(path,msg=msg) else: raise ResourceNotFoundError(path) self._s3bukt.delete_key(s3path) k = self._s3bukt.get_key(s3path) while k: k = self._s3bukt.get_key(s3path) def removedir(self,path,recursive=False,force=False): """Remove the directory at the given path.""" if normpath(path) in ('', '/'): raise RemoveRootError(path) s3path = self._s3path(path) if s3path != self._prefix: s3path = s3path + self._separator if force: # If we will be forcibly removing any directory contents, we # might as well get the un-delimited list straight away. ks = self._s3bukt.list(prefix=s3path) else: ks = self._s3bukt.list(prefix=s3path,delimiter=self._separator) # Fail if the directory is not empty, or remove them if forced found = False for k in ks: found = True if not _eq_utf8(k.name,s3path): if not force: raise DirectoryNotEmptyError(path) self._s3bukt.delete_key(k.name) if not found: if self.isfile(path): msg = "removedir() called on a regular file: %(path)s" raise ResourceInvalidError(path,msg=msg) if path not in ("","/"): raise ResourceNotFoundError(path) self._s3bukt.delete_key(s3path) if recursive and path not in ("","/"): pdir = dirname(path) try: self.removedir(pdir,recursive=True,force=False) except DirectoryNotEmptyError: pass def rename(self,src,dst): """Rename the file at 'src' to 'dst'.""" # Actually, in S3 'rename' is exactly the same as 'move' if self.isfile(src): self.move(src,dst) else: self.movedir(src,dst) def getinfo(self,path): s3path = self._s3path(path) if path in ("","/"): k = Prefix(bucket=self._s3bukt,name="/") else: k = self._s3bukt.get_key(s3path) if k is None: ks = self._s3bukt.list(prefix=s3path,delimiter=self._separator) for k in ks: if isinstance(k,Prefix): break else: raise ResourceNotFoundError(path) return self._get_key_info(k,path) def _get_key_info(self,key,name=None): info = {} if name is not None: info["name"] = basename(name) else: info["name"] = basename(self._uns3key(k.name)) if self._key_is_dir(key): info["st_mode"] = 0700 | statinfo.S_IFDIR else: info["st_mode"] = 0700 | statinfo.S_IFREG if hasattr(key,"size"): info['size'] = int(key.size) etag = getattr(key,"etag",None) if etag is not None: if isinstance(etag,unicode): etag = etag.encode("utf8") info['etag'] = etag.strip('"').strip("'") if hasattr(key,"last_modified"): # TODO: does S3 use any other formats? fmt = "%a, %d %b %Y %H:%M:%S %Z" try: mtime = datetime.datetime.strptime(key.last_modified,fmt) info['modified_time'] = mtime except ValueError: pass return info def desc(self,path): return "No description available" def copy(self,src,dst,overwrite=False,chunk_size=16384): """Copy a file from 'src' to 'dst'. src -- The source path dst -- The destination path overwrite -- If True, then the destination may be overwritten (if a file exists at that location). If False then an exception will be thrown if the destination exists chunk_size -- Size of chunks to use in copy (ignored by S3) """ s3path_dst = self._s3path(dst) s3path_dstD = s3path_dst + self._separator # Check for various preconditions. ks = self._s3bukt.list(prefix=s3path_dst,delimiter=self._separator) dstOK = False for k in ks: # It exists as a regular file if _eq_utf8(k.name,s3path_dst): if not overwrite: raise DestinationExistsError(dst) dstOK = True break # Check if it refers to a directory. If so, we copy *into* it. # Since S3 lists in lexicographic order, subsequent iterations # of the loop will check for the existence of the new filename. if _eq_utf8(k.name,s3path_dstD): nm = basename(src) dst = pathjoin(dirname(dst),nm) s3path_dst = s3path_dstD + nm dstOK = True if not dstOK and not self.isdir(dirname(dst)): msg = "Destination directory does not exist: %(path)s" raise ParentDirectoryMissingError(dst,msg=msg) # OK, now we can copy the file. s3path_src = self._s3path(src) try: self._s3bukt.copy_key(s3path_dst,self._bucket_name,s3path_src) except S3ResponseError, e: if "404 Not Found" in str(e): msg = "Source is not a file: %(path)s" raise ResourceInvalidError(src, msg=msg) raise e else: k = self._s3bukt.get_key(s3path_dst) while k is None: k = self._s3bukt.get_key(s3path_dst) self._sync_key(k) def move(self,src,dst,overwrite=False,chunk_size=16384): """Move a file from one location to another.""" self.copy(src,dst,overwrite=overwrite) self._s3bukt.delete_key(self._s3path(src)) def walkfiles(self, path="/", wildcard=None, dir_wildcard=None, search="breadth", ignore_errors=False ): if search != "breadth" or dir_wildcard is not None: args = (wildcard,dir_wildcard,search,ignore_errors) for item in super(S3FS,self).walkfiles(path,*args): yield item else: prefix = self._s3path(path) for k in self._s3bukt.list(prefix=prefix): name = relpath(self._uns3path(k.name,prefix)) if name != "": if not isinstance(name,unicode): name = name.decode("utf8") if not k.name.endswith(self._separator): if wildcard is not None: if callable(wildcard): if not wildcard(basename(name)): continue else: if not fnmatch(basename(name),wildcard): continue yield pathjoin(path,name) def walkinfo(self, path="/", wildcard=None, dir_wildcard=None, search="breadth", ignore_errors=False ): if search != "breadth" or dir_wildcard is not None: args = (wildcard,dir_wildcard,search,ignore_errors) for item in super(S3FS,self).walkfiles(path,*args): yield (item,self.getinfo(item)) else: prefix = self._s3path(path) for k in self._s3bukt.list(prefix=prefix): name = relpath(self._uns3path(k.name,prefix)) if name != "": if not isinstance(name,unicode): name = name.decode("utf8") if wildcard is not None: if callable(wildcard): if not wildcard(basename(name)): continue else: if not fnmatch(basename(name),wildcard): continue yield (pathjoin(path,name),self._get_key_info(k,name)) def walkfilesinfo(self, path="/", wildcard=None, dir_wildcard=None, search="breadth", ignore_errors=False ): if search != "breadth" or dir_wildcard is not None: args = (wildcard,dir_wildcard,search,ignore_errors) for item in super(S3FS,self).walkfiles(path,*args): yield (item,self.getinfo(item)) else: prefix = self._s3path(path) for k in self._s3bukt.list(prefix=prefix): name = relpath(self._uns3path(k.name,prefix)) if name != "": if not isinstance(name,unicode): name = name.decode("utf8") if not k.name.endswith(self._separator): if wildcard is not None: if callable(wildcard): if not wildcard(basename(name)): continue else: if not fnmatch(basename(name),wildcard): continue yield (pathjoin(path,name),self._get_key_info(k,name)) def _eq_utf8(name1,name2): if isinstance(name1,unicode): name1 = name1.encode("utf8") if isinstance(name2,unicode): name2 = name2.encode("utf8") return name1 == name2 def _startswith_utf8(name1,name2): if isinstance(name1,unicode): name1 = name1.encode("utf8") if isinstance(name2,unicode): name2 = name2.encode("utf8") return name1.startswith(name2) fs-0.5.4/fs/watch.py0000664000175000017500000005602212512525115014166 0ustar willwill00000000000000""" fs.watch ======== Change notification support for FS. This module defines a standard interface for FS subclasses that support change notification callbacks. It also offers some WrapFS subclasses that can simulate such an ability on top of an ordinary FS object. An FS object that wants to be "watchable" must provide the following methods: * ``add_watcher(callback,path="/",events=None,recursive=True)`` Request that the given callback be executed in response to changes to the given path. A specific set of change events can be specified. This method returns a Watcher object. * ``del_watcher(watcher_or_callback)`` Remove the given watcher object, or any watchers associated with the given callback. If you would prefer to read changes from a filesystem in a blocking fashion rather than using callbacks, you can use the function 'iter_changes' to obtain an iterator over the change events. """ import sys import weakref import threading import Queue import traceback from fs.path import * from fs.errors import * from fs.wrapfs import WrapFS from fs.base import FS from fs.filelike import FileWrapper from six import b class EVENT(object): """Base class for change notification events.""" def __init__(self,fs,path): super(EVENT, self).__init__() self.fs = fs if path is not None: path = abspath(normpath(path)) self.path = path def __str__(self): return unicode(self).encode("utf8") def __unicode__(self): return u"" % (self.__class__.__name__,self.path,hex(id(self))) def clone(self,fs=None,path=None): if fs is None: fs = self.fs if path is None: path = self.path return self.__class__(fs,path) class ACCESSED(EVENT): """Event fired when a file's contents are accessed.""" pass class CREATED(EVENT): """Event fired when a new file or directory is created.""" pass class REMOVED(EVENT): """Event fired when a file or directory is removed.""" pass class MODIFIED(EVENT): """Event fired when a file or directory is modified.""" def __init__(self,fs,path,data_changed=False, closed=False): super(MODIFIED,self).__init__(fs,path) self.data_changed = data_changed self.closed = closed def clone(self,fs=None,path=None,data_changed=None): evt = super(MODIFIED,self).clone(fs,path) if data_changed is None: data_changed = self.data_changed evt.data_changed = data_changed return evt class MOVED_DST(EVENT): """Event fired when a file or directory is the target of a move.""" def __init__(self,fs,path,source=None): super(MOVED_DST,self).__init__(fs,path) if source is not None: source = abspath(normpath(source)) self.source = source def __unicode__(self): return u"" % (self.__class__.__name__,self.path,self.source,hex(id(self))) def clone(self,fs=None,path=None,source=None): evt = super(MOVED_DST,self).clone(fs,path) if source is None: source = self.source evt.source = source return evt class MOVED_SRC(EVENT): """Event fired when a file or directory is the source of a move.""" def __init__(self,fs,path,destination=None): super(MOVED_SRC,self).__init__(fs,path) if destination is not None: destination = abspath(normpath(destination)) self.destination = destination def __unicode__(self): return u"" % (self.__class__.__name__,self.path,self.destination,hex(id(self))) def clone(self,fs=None,path=None,destination=None): evt = super(MOVED_SRC,self).clone(fs,path) if destination is None: destination = self.destination evt.destination = destination return evt class CLOSED(EVENT): """Event fired when the filesystem is closed.""" pass class ERROR(EVENT): """Event fired when some miscellaneous error occurs.""" pass class OVERFLOW(ERROR): """Event fired when some events could not be processed.""" pass class Watcher(object): """Object encapsulating filesystem watch info.""" def __init__(self,fs,callback,path="/",events=None,recursive=True): if events is None: events = (EVENT,) else: events = tuple(events) # Since the FS probably holds a reference to the Watcher, keeping # a reference back to the FS would create a cycle containing a # __del__ method. Use a weakref to avoid this. self._w_fs = weakref.ref(fs) self.callback = callback self.path = abspath(normpath(path)) self.events = events self.recursive = recursive @property def fs(self): return self._w_fs() def delete(self): fs = self.fs if fs is not None: fs.del_watcher(self) def handle_event(self,event): if not isinstance(event,self.events): return if event.path is not None: if not isprefix(self.path,event.path): return if not self.recursive: if event.path != self.path: if dirname(event.path) != self.path: return try: self.callback(event) except Exception: print >>sys.stderr, "error in FS watcher callback", self.callback traceback.print_exc() class WatchableFSMixin(FS): """Mixin class providing watcher management functions.""" def __init__(self,*args,**kwds): self._watchers = PathMap() super(WatchableFSMixin,self).__init__(*args,**kwds) def __getstate__(self): state = super(WatchableFSMixin,self).__getstate__() state.pop("_watchers",None) return state def __setstate__(self,state): super(WatchableFSMixin,self).__setstate__(state) self._watchers = PathMap() def add_watcher(self,callback,path="/",events=None,recursive=True): """Add a watcher callback to the FS.""" w = Watcher(self,callback,path,events,recursive=recursive) self._watchers.setdefault(path,[]).append(w) return w def del_watcher(self,watcher_or_callback): """Delete a watcher callback from the FS.""" if isinstance(watcher_or_callback,Watcher): self._watchers[watcher_or_callback.path].remove(watcher_or_callback) else: for watchers in self._watchers.itervalues(): for i,watcher in enumerate(watchers): if watcher.callback is watcher_or_callback: del watchers[i] break def _find_watchers(self,callback): """Find watchers registered with the given callback.""" for watchers in self._watchers.itervalues(): for watcher in watchers: if watcher.callback is callback: yield watcher def notify_watchers(self,event_or_class,path=None,*args,**kwds): """Notify watchers of the given event data.""" if isinstance(event_or_class,EVENT): event = event_or_class else: event = event_or_class(self,path,*args,**kwds) if path is None: path = event.path if path is None: for watchers in self._watchers.itervalues(): for watcher in watchers: watcher.handle_event(event) else: for prefix in recursepath(path): if prefix in self._watchers: for watcher in self._watchers[prefix]: watcher.handle_event(event) class WatchedFile(FileWrapper): """File wrapper for use with WatchableFS. This file wrapper provides access to a file opened from a WatchableFS instance, and fires MODIFIED events when the file is modified. """ def __init__(self,file,fs,path,mode=None): super(WatchedFile,self).__init__(file,mode) self.fs = fs self.path = path self.was_modified = False def _write(self,string,flushing=False): self.was_modified = True return super(WatchedFile,self)._write(string,flushing=flushing) def _truncate(self,size): self.was_modified = True return super(WatchedFile,self)._truncate(size) def flush(self): super(WatchedFile,self).flush() # Don't bother if python if being torn down if Watcher is not None: if self.was_modified: self.fs.notify_watchers(MODIFIED,self.path,True) def close(self): super(WatchedFile,self).close() # Don't bother if python if being torn down if Watcher is not None: if self.was_modified: self.fs.notify_watchers(MODIFIED,self.path,True) class WatchableFS(WatchableFSMixin,WrapFS): """FS wrapper simulating watcher callbacks. This FS wrapper intercepts method calls that modify the underlying FS and generates appropriate notification events. It thus allows watchers to monitor changes made through the underlying FS object, but not changes that might be made through other interfaces to the same filesystem. """ def __init__(self, *args, **kwds): super(WatchableFS, self).__init__(*args, **kwds) def close(self): super(WatchableFS, self).close() self.notify_watchers(CLOSED) def open(self, path, mode='r', buffering=-1, encoding=None, errors=None, newline=None, line_buffering=False, **kwargs): existed = self.wrapped_fs.isfile(path) f = super(WatchableFS, self).open(path, mode=mode, buffering=buffering, encoding=encoding, errors=errors, newline=newline, line_buffering=line_buffering, **kwargs) if not existed: self.notify_watchers(CREATED, path) self.notify_watchers(ACCESSED, path) return WatchedFile(f, self, path, mode) def setcontents(self, path, data=b'', encoding=None, errors=None, chunk_size=64*1024): existed = self.wrapped_fs.isfile(path) ret = super(WatchableFS, self).setcontents(path, data=data, encoding=encoding, errors=errors, chunk_size=chunk_size) if not existed: self.notify_watchers(CREATED, path) self.notify_watchers(ACCESSED, path) if data: self.notify_watchers(MODIFIED, path, True) return ret def createfile(self, path, wipe=False): existed = self.wrapped_fs.isfile(path) ret = super(WatchableFS, self).createfile(path, wipe=wipe) if not existed: self.notify_watchers(CREATED,path) self.notify_watchers(ACCESSED,path) return ret def makedir(self,path,recursive=False,allow_recreate=False): existed = self.wrapped_fs.isdir(path) try: super(WatchableFS,self).makedir(path,allow_recreate=allow_recreate) except ParentDirectoryMissingError: if not recursive: raise parent = dirname(path) if parent != path: self.makedir(dirname(path),recursive=True,allow_recreate=True) super(WatchableFS,self).makedir(path,allow_recreate=allow_recreate) if not existed: self.notify_watchers(CREATED,path) def remove(self,path): super(WatchableFS,self).remove(path) self.notify_watchers(REMOVED,path) def removedir(self,path,recursive=False,force=False): if not force: for nm in self.listdir(path): raise DirectoryNotEmptyError(path) else: for nm in self.listdir(path,dirs_only=True): try: self.removedir(pathjoin(path,nm),force=True) except ResourceNotFoundError: pass for nm in self.listdir(path,files_only=True): try: self.remove(pathjoin(path,nm)) except ResourceNotFoundError: pass super(WatchableFS,self).removedir(path) self.notify_watchers(REMOVED,path) if recursive: parent = dirname(path) while parent and not self.listdir(parent): super(WatchableFS,self).removedir(parent) self.notify_watchers(REMOVED,parent) parent = dirname(parent) def rename(self,src,dst): d_existed = self.wrapped_fs.exists(dst) super(WatchableFS,self).rename(src,dst) if d_existed: self.notify_watchers(REMOVED,dst) self.notify_watchers(MOVED_DST,dst,src) self.notify_watchers(MOVED_SRC,src,dst) def copy(self,src,dst,**kwds): d = self._pre_copy(src,dst) super(WatchableFS,self).copy(src,dst,**kwds) self._post_copy(src,dst,d) def copydir(self,src,dst,**kwds): d = self._pre_copy(src,dst) super(WatchableFS,self).copydir(src,dst,**kwds) self._post_copy(src,dst,d) def move(self,src,dst,**kwds): d = self._pre_copy(src,dst) super(WatchableFS,self).move(src,dst,**kwds) self._post_copy(src,dst,d) self._post_move(src,dst,d) def movedir(self,src,dst,**kwds): d = self._pre_copy(src,dst) super(WatchableFS,self).movedir(src,dst,**kwds) self._post_copy(src,dst,d) self._post_move(src,dst,d) def _pre_copy(self,src,dst): dst_paths = {} try: for (dirnm,filenms) in self.wrapped_fs.walk(dst): dirnm = dirnm[len(dst)+1:] dst_paths[dirnm] = True for filenm in filenms: dst_paths[filenm] = False except ResourceNotFoundError: pass except ResourceInvalidError: dst_paths[""] = False src_paths = {} try: for (dirnm,filenms) in self.wrapped_fs.walk(src): dirnm = dirnm[len(src)+1:] src_paths[dirnm] = True for filenm in filenms: src_paths[pathjoin(dirnm,filenm)] = False except ResourceNotFoundError: pass except ResourceInvalidError: src_paths[""] = False return (src_paths,dst_paths) def _post_copy(self,src,dst,data): (src_paths,dst_paths) = data for src_path,isdir in sorted(src_paths.items()): path = pathjoin(dst,src_path) if src_path in dst_paths: self.notify_watchers(MODIFIED,path,not isdir) else: self.notify_watchers(CREATED,path) for dst_path,isdir in sorted(dst_paths.items()): path = pathjoin(dst,dst_path) if not self.wrapped_fs.exists(path): self.notify_watchers(REMOVED,path) def _post_move(self,src,dst,data): (src_paths,dst_paths) = data for src_path,isdir in sorted(src_paths.items(),reverse=True): path = pathjoin(src,src_path) self.notify_watchers(REMOVED,path) def setxattr(self,path,name,value): super(WatchableFS,self).setxattr(path,name,value) self.notify_watchers(MODIFIED,path,False) def delxattr(self,path,name): super(WatchableFS,self).delxattr(path,name) self.notify_watchers(MODIFIED,path,False) class PollingWatchableFS(WatchableFS): """FS wrapper simulating watcher callbacks by periodic polling. This FS wrapper augments the functionality of WatchableFS by periodically polling the underlying FS for changes. It is thus capable of detecting changes made to the underlying FS via other interfaces, albeit with a (configurable) delay to account for the polling interval. """ def __init__(self,wrapped_fs,poll_interval=60*5): super(PollingWatchableFS,self).__init__(wrapped_fs) self.poll_interval = poll_interval self.add_watcher(self._on_path_modify,"/",(CREATED,MOVED_DST,)) self.add_watcher(self._on_path_modify,"/",(MODIFIED,ACCESSED,)) self.add_watcher(self._on_path_delete,"/",(REMOVED,MOVED_SRC,)) self._path_info = PathMap() self._poll_thread = threading.Thread(target=self._poll_for_changes) self._poll_cond = threading.Condition() self._poll_close_event = threading.Event() self._poll_thread.start() def close(self): self._poll_close_event.set() self._poll_thread.join() super(PollingWatchableFS,self).close() def _on_path_modify(self,event): path = event.path try: try: self._path_info[path] = self.wrapped_fs.getinfo(path) except ResourceNotFoundError: self._path_info.clear(path) except FSError: pass def _on_path_delete(self,event): self._path_info.clear(event.path) def _poll_for_changes(self): try: while not self._poll_close_event.isSet(): # Walk all directories looking for changes. # Come back to any that give us an error. error_paths = set() for dirnm in self.wrapped_fs.walkdirs(): if self._poll_close_event.isSet(): break try: self._check_for_changes(dirnm) except FSError: error_paths.add(dirnm) # Retry the directories that gave us an error, until # we have successfully updated them all while error_paths and not self._poll_close_event.isSet(): dirnm = error_paths.pop() if self.wrapped_fs.isdir(dirnm): try: self._check_for_changes(dirnm) except FSError: error_paths.add(dirnm) # Notify that we have completed a polling run self._poll_cond.acquire() self._poll_cond.notifyAll() self._poll_cond.release() # Sleep for the specified interval, or until closed. self._poll_close_event.wait(timeout=self.poll_interval) except FSError: if not self.closed: raise def _check_for_changes(self,dirnm): # Check the metadata for the directory itself. new_info = self.wrapped_fs.getinfo(dirnm) try: old_info = self._path_info[dirnm] except KeyError: self.notify_watchers(CREATED,dirnm) else: if new_info != old_info: self.notify_watchers(MODIFIED,dirnm,False) # Check the metadata for each file in the directory. # We assume that if the file's data changes, something in its # metadata will also change; don't want to read through each file! # Subdirectories will be handled by the outer polling loop. for filenm in self.wrapped_fs.listdir(dirnm,files_only=True): if self._poll_close_event.isSet(): return fpath = pathjoin(dirnm,filenm) new_info = self.wrapped_fs.getinfo(fpath) try: old_info = self._path_info[fpath] except KeyError: self.notify_watchers(CREATED,fpath) else: was_accessed = False was_modified = False for (k,v) in new_info.iteritems(): if k not in old_info: was_modified = True break elif old_info[k] != v: if k in ("accessed_time","st_atime",): was_accessed = True elif k: was_modified = True break else: for k in old_info: if k not in new_info: was_modified = True break if was_modified: self.notify_watchers(MODIFIED,fpath,True) elif was_accessed: self.notify_watchers(ACCESSED,fpath) # Check for deletion of cached child entries. for childnm in self._path_info.iternames(dirnm): if self._poll_close_event.isSet(): return cpath = pathjoin(dirnm,childnm) if not self.wrapped_fs.exists(cpath): self.notify_watchers(REMOVED,cpath) def ensure_watchable(fs,wrapper_class=PollingWatchableFS,*args,**kwds): """Ensure that the given fs supports watching, simulating it if necessary. Given an FS object, this function returns an equivalent FS that has support for watcher callbacks. This may be the original object if it supports them natively, or a wrapper class if they must be simulated. """ if isinstance(fs,wrapper_class): return fs try: w = fs.add_watcher(lambda e: None,"/",recursive=False) except (AttributeError,FSError): return wrapper_class(fs,*args,**kwds) else: fs.del_watcher(w) return fs class iter_changes(object): """Blocking iterator over the change events produced by an FS. This class can be used to transform the callback-based watcher mechanism into a blocking stream of events. It operates by having the callbacks push events onto a queue as they come in, then reading them off one at a time. """ def __init__(self,fs=None,path="/",events=None,**kwds): self.closed = False self._queue = Queue.Queue() self._watching = set() if fs is not None: self.add_watcher(fs,path,events,**kwds) def __iter__(self): return self def __del__(self): self.close() def next(self,timeout=None): if not self._watching: raise StopIteration try: event = self._queue.get(timeout=timeout) except Queue.Empty: raise StopIteration if event is None: raise StopIteration if isinstance(event,CLOSED): event.fs.del_watcher(self._enqueue) self._watching.remove(event.fs) return event def close(self): if not self.closed: self.closed = True for fs in self._watching: fs.del_watcher(self._enqueue) self._queue.put(None) def add_watcher(self,fs,path="/",events=None,**kwds): w = fs.add_watcher(self._enqueue,path,events,**kwds) self._watching.add(fs) return w def _enqueue(self,event): self._queue.put(event) def del_watcher(self,watcher): for fs in self._watching: try: fs.del_watcher(watcher) break except ValueError: pass else: raise ValueError("watcher not found: %s" % (watcher,)) fs-0.5.4/fs/tests/0000755000000000000000000000000012621617365013651 5ustar rootroot00000000000000fs-0.5.4/fs/tests/test_fs.py0000664000175000017500000000732512512525115015673 0ustar willwill00000000000000""" fs.tests.test_fs: testcases for basic FS implementations """ from fs.tests import FSTestCases, ThreadingTestCases from fs.path import * from fs import errors import unittest import os import sys import shutil import tempfile from fs import osfs class TestOSFS(unittest.TestCase,FSTestCases,ThreadingTestCases): def setUp(self): self.temp_dir = tempfile.mkdtemp(u"fstest") self.fs = osfs.OSFS(self.temp_dir) def tearDown(self): shutil.rmtree(self.temp_dir) self.fs.close() def check(self, p): return os.path.exists(os.path.join(self.temp_dir, relpath(p))) def test_invalid_chars(self): super(TestOSFS, self).test_invalid_chars() self.assertRaises(errors.InvalidCharsInPathError, self.fs.open, 'invalid\0file', 'wb') self.assertFalse(self.fs.isvalidpath('invalid\0file')) self.assert_(self.fs.isvalidpath('validfile')) self.assert_(self.fs.isvalidpath('completely_valid/path/foo.bar')) class TestSubFS(unittest.TestCase,FSTestCases,ThreadingTestCases): def setUp(self): self.temp_dir = tempfile.mkdtemp(u"fstest") self.parent_fs = osfs.OSFS(self.temp_dir) self.parent_fs.makedir("foo/bar", recursive=True) self.fs = self.parent_fs.opendir("foo/bar") def tearDown(self): shutil.rmtree(self.temp_dir) self.fs.close() def check(self, p): p = os.path.join("foo/bar", relpath(p)) full_p = os.path.join(self.temp_dir, p) return os.path.exists(full_p) from fs import memoryfs class TestMemoryFS(unittest.TestCase,FSTestCases,ThreadingTestCases): def setUp(self): self.fs = memoryfs.MemoryFS() from fs import mountfs class TestMountFS(unittest.TestCase,FSTestCases,ThreadingTestCases): def setUp(self): self.mount_fs = mountfs.MountFS() self.mem_fs = memoryfs.MemoryFS() self.mount_fs.mountdir("mounted/memfs", self.mem_fs) self.fs = self.mount_fs.opendir("mounted/memfs") def tearDown(self): self.fs.close() def check(self, p): return self.mount_fs.exists(pathjoin("mounted/memfs", relpath(p))) class TestMountFS_atroot(unittest.TestCase,FSTestCases,ThreadingTestCases): def setUp(self): self.mem_fs = memoryfs.MemoryFS() self.fs = mountfs.MountFS() self.fs.mountdir("", self.mem_fs) def tearDown(self): self.fs.close() def check(self, p): return self.mem_fs.exists(p) class TestMountFS_stacked(unittest.TestCase,FSTestCases,ThreadingTestCases): def setUp(self): self.mem_fs1 = memoryfs.MemoryFS() self.mem_fs2 = memoryfs.MemoryFS() self.mount_fs = mountfs.MountFS() self.mount_fs.mountdir("mem", self.mem_fs1) self.mount_fs.mountdir("mem/two", self.mem_fs2) self.fs = self.mount_fs.opendir("/mem/two") def tearDown(self): self.fs.close() def check(self, p): return self.mount_fs.exists(pathjoin("mem/two", relpath(p))) from fs import tempfs class TestTempFS(unittest.TestCase,FSTestCases,ThreadingTestCases): def setUp(self): self.fs = tempfs.TempFS() def tearDown(self): td = self.fs._temp_dir self.fs.close() self.assert_(not os.path.exists(td)) def check(self, p): td = self.fs._temp_dir return os.path.exists(os.path.join(td, relpath(p))) def test_invalid_chars(self): super(TestTempFS, self).test_invalid_chars() self.assertRaises(errors.InvalidCharsInPathError, self.fs.open, 'invalid\0file', 'wb') self.assertFalse(self.fs.isvalidpath('invalid\0file')) self.assert_(self.fs.isvalidpath('validfile')) self.assert_(self.fs.isvalidpath('completely_valid/path/foo.bar')) fs-0.5.4/fs/tests/test_opener.py0000664000175000017500000000104112512525115016540 0ustar willwill00000000000000""" fs.tests.test_opener: testcases for FS opener """ import unittest import tempfile import shutil from fs.opener import opener from fs import path class TestOpener(unittest.TestCase): def setUp(self): self.temp_dir = tempfile.mkdtemp(u"fstest_opener") def tearDown(self): shutil.rmtree(self.temp_dir) def testOpen(self): filename = path.join(self.temp_dir, 'foo.txt') file_object = opener.open(filename, 'wb') file_object.close() self.assertTrue(file_object.closed) fs-0.5.4/fs/tests/zipfs_binary_test.py0000664000175000017500000000210112512525115017745 0ustar willwill00000000000000""" Test case for ZipFS binary file reading/writing Passes ok on Linux, fails on Windows (tested: Win7, 64-bit): AssertionError: ' \r\n' != ' \n' """ import unittest from fs.zipfs import ZipFS import os from six import b class ZipFsBinaryWriteRead(unittest.TestCase): test_content = b(chr(32) + chr(10)) def setUp(self): self.z = ZipFS('test.zip', 'w') def tearDown(self): try: os.remove('test.zip') except: pass def test_binary_write_read(self): # GIVEN zipfs z = self.z # WHEN binary data is written to a test file in zipfs f = z.open('test.data', 'wb') f.write(self.test_content) f.close() z.close() # THEN the same binary data is retrieved when opened again z = ZipFS('test.zip', 'r') f = z.open('test.data', 'rb') content = f.read() f.close() z.close() self.assertEqual(content, self.test_content) if __name__ == '__main__': unittest.main() fs-0.5.4/fs/tests/test_rpcfs.py0000664000175000017500000000512412512525115016373 0ustar willwill00000000000000 import unittest import sys import os, os.path import socket import threading import time from fs.tests import FSTestCases, ThreadingTestCases from fs.tempfs import TempFS from fs.osfs import OSFS from fs.memoryfs import MemoryFS from fs.path import * from fs.errors import * from fs import rpcfs from fs.expose.xmlrpc import RPCFSServer import six from six import PY3, b class TestRPCFS(unittest.TestCase, FSTestCases, ThreadingTestCases): def makeServer(self,fs,addr): return RPCFSServer(fs,addr,logRequests=False) def startServer(self): port = 3000 self.temp_fs = TempFS() self.server = None self.serve_more_requests = True self.server_thread = threading.Thread(target=self.runServer) self.server_thread.setDaemon(True) self.start_event = threading.Event() self.end_event = threading.Event() self.server_thread.start() self.start_event.wait() def runServer(self): """Run the server, swallowing shutdown-related execptions.""" port = 3000 while not self.server: try: self.server = self.makeServer(self.temp_fs,("127.0.0.1",port)) except socket.error, e: if e.args[1] == "Address already in use": port += 1 else: raise self.server_addr = ("127.0.0.1", port) self.server.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) self.start_event.set() try: #self.server.serve_forever() while self.serve_more_requests: self.server.handle_request() except Exception, e: pass self.end_event.set() def setUp(self): self.startServer() self.fs = rpcfs.RPCFS("http://%s:%d" % self.server_addr) def tearDown(self): self.serve_more_requests = False try: self.bump() self.server.server_close() except Exception: pass #self.server_thread.join() self.temp_fs.close() def bump(self): host, port = self.server_addr for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM): af, socktype, proto, cn, sa = res sock = None try: sock = socket.socket(af, socktype, proto) sock.settimeout(.1) sock.connect(sa) sock.send(b("\n")) except socket.error, e: pass finally: if sock is not None: sock.close() fs-0.5.4/fs/tests/test_importhook.py0000664000175000017500000001113512512525115017450 0ustar willwill00000000000000 import sys import unittest import marshal import imp import struct from textwrap import dedent from fs.expose.importhook import FSImportHook from fs.tempfs import TempFS from fs.zipfs import ZipFS from six import b class TestFSImportHook(unittest.TestCase): def setUp(self): pass def tearDown(self): for mph in list(sys.meta_path): if isinstance(mph,FSImportHook): sys.meta_path.remove(mph) for ph in list(sys.path_hooks): if issubclass(ph,FSImportHook): sys.path_hooks.remove(mph) for (k,v) in sys.modules.items(): if k.startswith("fsih_"): del sys.modules[k] elif hasattr(v,"__loader__"): if isinstance(v.__loader__,FSImportHook): del sys.modules[k] sys.path_importer_cache.clear() def _init_modules(self,fs): fs.setcontents("fsih_hello.py",b(dedent(""" message = 'hello world!' """))) fs.makedir("fsih_pkg") fs.setcontents("fsih_pkg/__init__.py",b(dedent(""" a = 42 """))) fs.setcontents("fsih_pkg/sub1.py",b(dedent(""" import fsih_pkg from fsih_hello import message a = fsih_pkg.a """))) fs.setcontents("fsih_pkg/sub2.pyc",self._getpyc(b(dedent(""" import fsih_pkg from fsih_hello import message a = fsih_pkg.a * 2 """)))) def _getpyc(self,src): """Get the .pyc contents to match th given .py source code.""" code = imp.get_magic() + struct.pack("") from fs.path import * from fs import ftpfs ftp_port = 30000 class TestFTPFS(unittest.TestCase, FSTestCases, ThreadingTestCases): __test__ = not PY3 def setUp(self): global ftp_port ftp_port += 1 use_port = str(ftp_port) #ftp_port = 10000 self.temp_dir = tempfile.mkdtemp(u"ftpfstests") file_path = __file__ if ':' not in file_path: file_path = abspath(file_path) # Apparently Windows requires values from default environment, so copy the exisiting os.environ env = os.environ.copy() env['PYTHONPATH'] = os.getcwd() + os.pathsep + env.get('PYTHONPATH', '') self.ftp_server = subprocess.Popen([sys.executable, file_path, self.temp_dir, use_port], stdout=subprocess.PIPE, env=env) # Block until the server writes a line to stdout self.ftp_server.stdout.readline() # Poll until a connection can be made start_time = time.time() while time.time() - start_time < 5: try: ftpurl = urllib.urlopen('ftp://127.0.0.1:%s' % use_port) except IOError: time.sleep(0) else: ftpurl.read() ftpurl.close() break else: # Avoid a possible infinite loop raise Exception("Unable to connect to ftp server") self.fs = ftpfs.FTPFS('127.0.0.1', 'user', '12345', dircache=True, port=use_port, timeout=5.0) self.fs.cache_hint(True) def tearDown(self): #self.ftp_server.terminate() if sys.platform == 'win32': os.popen('TASKKILL /PID '+str(self.ftp_server.pid)+' /F') else: os.system('kill '+str(self.ftp_server.pid)) shutil.rmtree(self.temp_dir) self.fs.close() def check(self, p): check_path = self.temp_dir.rstrip(os.sep) + os.sep + p return os.path.exists(check_path.encode('utf-8')) if __name__ == "__main__": # Run an ftp server that exposes a given directory import sys authorizer = DummyAuthorizer() authorizer.add_user("user", "12345", sys.argv[1], perm="elradfmw") authorizer.add_anonymous(sys.argv[1]) #def nolog(*args): # pass #ftpserver.log = nolog #ftpserver.logline = nolog handler = FTPHandler handler.authorizer = authorizer address = ("127.0.0.1", int(sys.argv[2])) #print address ftpd = FTPServer(address, handler) sys.stdout.write('serving\n') sys.stdout.flush() ftpd.serve_forever() fs-0.5.4/fs/tests/test_sqlitefs.py0000664000175000017500000000060112512525115017103 0ustar willwill00000000000000try: from fs.contrib.sqlitefs import SqliteFS except ImportError: SqliteFS = None from fs.tests import FSTestCases import unittest import os if SqliteFS: class TestSqliteFS(unittest.TestCase, FSTestCases): def setUp(self): self.fs = SqliteFS("sqlitefs.db") def tearDown(self): os.remove('sqlitefs.db') fs-0.5.4/fs/tests/test_utils.py0000664000175000017500000000751212512525115016421 0ustar willwill00000000000000import unittest from fs.tempfs import TempFS from fs.memoryfs import MemoryFS from fs import utils from six import b class TestUtils(unittest.TestCase): def _make_fs(self, fs): fs.setcontents("f1", b("file 1")) fs.setcontents("f2", b("file 2")) fs.setcontents("f3", b("file 3")) fs.makedir("foo/bar", recursive=True) fs.setcontents("foo/bar/fruit", b("apple")) def _check_fs(self, fs): self.assert_(fs.isfile("f1")) self.assert_(fs.isfile("f2")) self.assert_(fs.isfile("f3")) self.assert_(fs.isdir("foo/bar")) self.assert_(fs.isfile("foo/bar/fruit")) self.assertEqual(fs.getcontents("f1", "rb"), b("file 1")) self.assertEqual(fs.getcontents("f2", "rb"), b("file 2")) self.assertEqual(fs.getcontents("f3", "rb"), b("file 3")) self.assertEqual(fs.getcontents("foo/bar/fruit", "rb"), b("apple")) def test_copydir_root(self): """Test copydir from root""" fs1 = MemoryFS() self._make_fs(fs1) fs2 = MemoryFS() utils.copydir(fs1, fs2) self._check_fs(fs2) fs1 = TempFS() self._make_fs(fs1) fs2 = TempFS() utils.copydir(fs1, fs2) self._check_fs(fs2) def test_copydir_indir(self): """Test copydir in a directory""" fs1 = MemoryFS() fs2 = MemoryFS() self._make_fs(fs1) utils.copydir(fs1, (fs2, "copy")) self._check_fs(fs2.opendir("copy")) fs1 = TempFS() fs2 = TempFS() self._make_fs(fs1) utils.copydir(fs1, (fs2, "copy")) self._check_fs(fs2.opendir("copy")) def test_movedir_indir(self): """Test movedir in a directory""" fs1 = MemoryFS() fs2 = MemoryFS() fs1sub = fs1.makeopendir("from") self._make_fs(fs1sub) utils.movedir((fs1, "from"), (fs2, "copy")) self.assert_(not fs1.exists("from")) self._check_fs(fs2.opendir("copy")) fs1 = TempFS() fs2 = TempFS() fs1sub = fs1.makeopendir("from") self._make_fs(fs1sub) utils.movedir((fs1, "from"), (fs2, "copy")) self.assert_(not fs1.exists("from")) self._check_fs(fs2.opendir("copy")) def test_movedir_root(self): """Test movedir to root dir""" fs1 = MemoryFS() fs2 = MemoryFS() fs1sub = fs1.makeopendir("from") self._make_fs(fs1sub) utils.movedir((fs1, "from"), fs2) self.assert_(not fs1.exists("from")) self._check_fs(fs2) fs1 = TempFS() fs2 = TempFS() fs1sub = fs1.makeopendir("from") self._make_fs(fs1sub) utils.movedir((fs1, "from"), fs2) self.assert_(not fs1.exists("from")) self._check_fs(fs2) def test_remove_all(self): """Test remove_all function""" fs = TempFS() fs.setcontents("f1", b("file 1")) fs.setcontents("f2", b("file 2")) fs.setcontents("f3", b("file 3")) fs.makedir("foo/bar", recursive=True) fs.setcontents("foo/bar/fruit", b("apple")) fs.setcontents("foo/baz", b("baz")) utils.remove_all(fs, "foo/bar") self.assert_(not fs.exists("foo/bar/fruit")) self.assert_(fs.exists("foo/bar")) self.assert_(fs.exists("foo/baz")) utils.remove_all(fs, "") self.assert_(not fs.exists("foo/bar/fruit")) self.assert_(not fs.exists("foo/bar/baz")) self.assert_(not fs.exists("foo/baz")) self.assert_(not fs.exists("foo")) self.assert_(not fs.exists("f1")) self.assert_(fs.isdirempty('/')) fs-0.5.4/fs/tests/test_wrapfs.py0000664000175000017500000000617312512525115016565 0ustar willwill00000000000000""" fs.tests.test_wrapfs: testcases for FS wrapper implementations """ import unittest from fs.tests import FSTestCases, ThreadingTestCases import os import sys import shutil import tempfile from fs import osfs from fs.errors import * from fs.path import * from fs.utils import remove_all from fs import wrapfs import six from six import PY3, b class TestWrapFS(unittest.TestCase, FSTestCases, ThreadingTestCases): #__test__ = False def setUp(self): self.temp_dir = tempfile.mkdtemp(u"fstest") self.fs = wrapfs.WrapFS(osfs.OSFS(self.temp_dir)) def tearDown(self): shutil.rmtree(self.temp_dir) self.fs.close() def check(self, p): return os.path.exists(os.path.join(self.temp_dir, relpath(p))) from fs.wrapfs.lazyfs import LazyFS class TestLazyFS(unittest.TestCase, FSTestCases, ThreadingTestCases): def setUp(self): self.temp_dir = tempfile.mkdtemp(u"fstest") self.fs = LazyFS((osfs.OSFS,(self.temp_dir,))) def tearDown(self): shutil.rmtree(self.temp_dir) self.fs.close() def check(self, p): return os.path.exists(os.path.join(self.temp_dir, relpath(p))) from fs.wrapfs.limitsizefs import LimitSizeFS class TestLimitSizeFS(TestWrapFS): _dont_retest = TestWrapFS._dont_retest + ("test_big_file",) def setUp(self): super(TestLimitSizeFS,self).setUp() self.fs = LimitSizeFS(self.fs,1024*1024*2) # 2MB limit def tearDown(self): remove_all(self.fs, "/") self.assertEquals(self.fs.cur_size,0) super(TestLimitSizeFS,self).tearDown() self.fs.close() def test_storage_error(self): total_written = 0 for i in xrange(1024*2): try: total_written += 1030 self.fs.setcontents("file %i" % i, b("C")*1030) except StorageSpaceError: self.assertTrue(total_written > 1024*1024*2) self.assertTrue(total_written < 1024*1024*2 + 1030) break else: self.assertTrue(False,"StorageSpaceError not raised") from fs.wrapfs.hidedotfilesfs import HideDotFilesFS class TestHideDotFilesFS(unittest.TestCase): def setUp(self): self.temp_dir = tempfile.mkdtemp(u"fstest") open(os.path.join(self.temp_dir, u".dotfile"), 'w').close() open(os.path.join(self.temp_dir, u"regularfile"), 'w').close() os.mkdir(os.path.join(self.temp_dir, u".dotdir")) os.mkdir(os.path.join(self.temp_dir, u"regulardir")) self.fs = HideDotFilesFS(osfs.OSFS(self.temp_dir)) def tearDown(self): shutil.rmtree(self.temp_dir) self.fs.close() def test_hidden(self): self.assertEquals(len(self.fs.listdir(hidden=False)), 2) self.assertEquals(len(list(self.fs.ilistdir(hidden=False))), 2) def test_nonhidden(self): self.assertEquals(len(self.fs.listdir(hidden=True)), 4) self.assertEquals(len(list(self.fs.ilistdir(hidden=True))), 4) def test_default(self): self.assertEquals(len(self.fs.listdir()), 2) self.assertEquals(len(list(self.fs.ilistdir())), 2) fs-0.5.4/fs/tests/test_remote.py0000664000175000017500000003072012512525115016551 0ustar willwill00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- """ fs.tests.test_remote: testcases for FS remote support utilities """ from fs.tests import FSTestCases, ThreadingTestCases import unittest import threading import random import time import sys from fs.remote import * from fs import SEEK_END from fs.wrapfs import WrapFS, wrap_fs_methods from fs.tempfs import TempFS from fs.path import * from fs.local_functools import wraps from six import PY3, b class RemoteTempFS(TempFS): """ Simple filesystem implementing setfilecontents for RemoteFileBuffer tests """ def __repr__(self): return '' % self._temp_dir def open(self, path, mode='rb', write_on_flush=True, **kwargs): if 'a' in mode or 'r' in mode or '+' in mode: f = super(RemoteTempFS, self).open(path, mode='rb', **kwargs) f = TellAfterCloseFile(f) else: f = None return RemoteFileBuffer(self, path, mode, f, write_on_flush=write_on_flush) def setcontents(self, path, data, encoding=None, errors=None, chunk_size=64*1024): f = super(RemoteTempFS, self).open(path, 'wb', encoding=encoding, errors=errors, chunk_size=chunk_size) if getattr(data, 'read', False): f.write(data.read()) else: f.write(data) f.close() class TellAfterCloseFile(object): """File-like object that allows calling tell() after it's been closed.""" def __init__(self, file): self._finalpos = None self.file = file def close(self): if self._finalpos is None: self._finalpos = self.file.tell() self.file.close() def tell(self): if self._finalpos is not None: return self._finalpos return self.file.tell() def __getattr__(self, attr): return getattr(self.file, attr) class TestRemoteFileBuffer(unittest.TestCase, FSTestCases, ThreadingTestCases): class FakeException(Exception): pass def setUp(self): self.fs = RemoteTempFS() self.original_setcontents = self.fs.setcontents def tearDown(self): self.fs.close() self.fakeOff() def fake_setcontents(self, path, content=b(''), chunk_size=16*1024): ''' Fake replacement for RemoteTempFS setcontents() ''' raise self.FakeException("setcontents should not be called here!") def fakeOn(self): ''' Turn on fake_setcontents(). When setcontents on RemoteTempFS is called, FakeException is raised and nothing is stored. ''' self.fs.setcontents = self.fake_setcontents def fakeOff(self): ''' Switch off fake_setcontents(). ''' self.fs.setcontents = self.original_setcontents def test_ondemand(self): ''' Tests on-demand loading of remote content in RemoteFileBuffer ''' contents = b("Tristatricettri stribrnych strikacek strikalo") + \ b("pres tristatricettri stribrnych strech.") f = self.fs.open('test.txt', 'wb') f.write(contents) f.close() # During following tests, no setcontents() should be called. self.fakeOn() f = self.fs.open('test.txt', 'rb') self.assertEquals(f.read(10), contents[:10]) f.wrapped_file.seek(0, SEEK_END) self.assertEquals(f._rfile.tell(), 10) f.seek(20) self.assertEquals(f.tell(), 20) self.assertEquals(f._rfile.tell(), 20) f.seek(0, SEEK_END) self.assertEquals(f._rfile.tell(), len(contents)) f.close() f = self.fs.open('test.txt', 'ab') self.assertEquals(f.tell(), len(contents)) f.close() self.fakeOff() # Writing over the rfile edge f = self.fs.open('test.txt', 'wb+') self.assertEquals(f.tell(), 0) f.seek(len(contents) - 5) # Last 5 characters not loaded from remote file self.assertEquals(f._rfile.tell(), len(contents) - 5) # Confirm that last 5 characters are still in rfile buffer self.assertEquals(f._rfile.read(), contents[-5:]) # Rollback position 5 characters before eof f._rfile.seek(len(contents[:-5])) # Write 10 new characters (will make contents longer for 5 chars) f.write(b('1234567890')) f.flush() # We are on the end of file (and buffer not serve anything anymore) self.assertEquals(f.read(), b('')) f.close() self.fakeOn() # Check if we wrote everything OK from # previous writing over the remote buffer edge f = self.fs.open('test.txt', 'rb') self.assertEquals(f.read(), contents[:-5] + b('1234567890')) f.close() self.fakeOff() def test_writeonflush(self): ''' Test 'write_on_flush' switch of RemoteFileBuffer. When True, flush() should call setcontents and store to remote destination. When False, setcontents should be called only on close(). ''' self.fakeOn() f = self.fs.open('test.txt', 'wb', write_on_flush=True) f.write(b('Sample text')) self.assertRaises(self.FakeException, f.flush) f.write(b('Second sample text')) self.assertRaises(self.FakeException, f.close) self.fakeOff() f.close() self.fakeOn() f = self.fs.open('test.txt', 'wb', write_on_flush=False) f.write(b('Sample text')) # FakeException is not raised, because setcontents is not called f.flush() f.write(b('Second sample text')) self.assertRaises(self.FakeException, f.close) self.fakeOff() def test_flush_and_continue(self): ''' This tests if partially loaded remote buffer can be flushed back to remote destination and opened file is still in good condition. ''' contents = b("Zlutoucky kun upel dabelske ody.") contents2 = b('Ententyky dva spaliky cert vyletel z elektriky') f = self.fs.open('test.txt', 'wb') f.write(contents) f.close() f = self.fs.open('test.txt', 'rb+') # Check if we read just 10 characters self.assertEquals(f.read(10), contents[:10]) self.assertEquals(f._rfile.tell(), 10) # Write garbage to file to mark it as _changed f.write(b('x')) # This should read the rest of file and store file back to again. f.flush() f.seek(0) # Try if we have unocrrupted file locally... self.assertEquals(f.read(), contents[:10] + b('x') + contents[11:]) f.close() # And if we have uncorrupted file also on storage f = self.fs.open('test.txt', 'rb') self.assertEquals(f.read(), contents[:10] + b('x') + contents[11:]) f.close() # Now try it again, but write garbage behind edge of remote file f = self.fs.open('test.txt', 'rb+') self.assertEquals(f.read(10), contents[:10]) # Write garbage to file to mark it as _changed f.write(contents2) f.flush() f.seek(0) # Try if we have unocrrupted file locally... self.assertEquals(f.read(), contents[:10] + contents2) f.close() # And if we have uncorrupted file also on storage f = self.fs.open('test.txt', 'rb') self.assertEquals(f.read(), contents[:10] + contents2) f.close() class TestCacheFS(unittest.TestCase,FSTestCases,ThreadingTestCases): """Test simple operation of CacheFS""" def setUp(self): self._check_interval = sys.getcheckinterval() sys.setcheckinterval(10) self.wrapped_fs = TempFS() self.fs = CacheFS(self.wrapped_fs,cache_timeout=0.01) def tearDown(self): self.fs.close() sys.setcheckinterval(self._check_interval) def test_values_are_used_from_cache(self): old_timeout = self.fs.cache_timeout self.fs.cache_timeout = None try: self.assertFalse(self.fs.isfile("hello")) self.wrapped_fs.setcontents("hello",b("world")) self.assertTrue(self.fs.isfile("hello")) self.wrapped_fs.remove("hello") self.assertTrue(self.fs.isfile("hello")) self.fs.clear_cache() self.assertFalse(self.fs.isfile("hello")) finally: self.fs.cache_timeout = old_timeout def test_values_are_updated_in_cache(self): old_timeout = self.fs.cache_timeout self.fs.cache_timeout = None try: self.assertFalse(self.fs.isfile("hello")) self.wrapped_fs.setcontents("hello",b("world")) self.assertTrue(self.fs.isfile("hello")) self.wrapped_fs.remove("hello") self.assertTrue(self.fs.isfile("hello")) self.wrapped_fs.setcontents("hello",b("world")) self.assertTrue(self.fs.isfile("hello")) self.fs.remove("hello") self.assertFalse(self.fs.isfile("hello")) finally: self.fs.cache_timeout = old_timeout class TestConnectionManagerFS(unittest.TestCase,FSTestCases):#,ThreadingTestCases): """Test simple operation of ConnectionManagerFS""" def setUp(self): self._check_interval = sys.getcheckinterval() sys.setcheckinterval(10) self.fs = ConnectionManagerFS(TempFS()) def tearDown(self): self.fs.close() sys.setcheckinterval(self._check_interval) class DisconnectingFS(WrapFS): """FS subclass that raises lots of RemoteConnectionErrors.""" def __init__(self,fs=None): if fs is None: fs = TempFS() self._connected = True self._continue = True self._bounce_thread = None super(DisconnectingFS,self).__init__(fs) if random.choice([True,False]): raise RemoteConnectionError("") self._bounce_thread = threading.Thread(target=self._bounce) self._bounce_thread.daemon = True self._bounce_thread.start() def __getstate__(self): state = super(DisconnectingFS,self).__getstate__() del state["_bounce_thread"] return state def __setstate__(self,state): super(DisconnectingFS,self).__setstate__(state) self._bounce_thread = threading.Thread(target=self._bounce) self._bounce_thread.daemon = True self._bounce_thread.start() def _bounce(self): while self._continue: time.sleep(random.random()*0.1) self._connected = not self._connected def setcontents(self, path, data=b(''), encoding=None, errors=None, chunk_size=64*1024): return self.wrapped_fs.setcontents(path, data, encoding=encoding, errors=errors, chunk_size=chunk_size) def close(self): if not self.closed: self._continue = False if self._bounce_thread is not None: self._bounce_thread.join() self._connected = True super(DisconnectingFS,self).close() def disconnecting_wrapper(func): """Method wrapper to raise RemoteConnectionError if not connected.""" @wraps(func) def wrapper(self,*args,**kwds): if not self._connected: raise RemoteConnectionError("") return func(self,*args,**kwds) return wrapper DisconnectingFS = wrap_fs_methods(disconnecting_wrapper,DisconnectingFS,exclude=["close"]) class DisconnectRecoveryFS(WrapFS): """FS subclass that recovers from RemoteConnectionErrors by waiting.""" pass def recovery_wrapper(func): """Method wrapper to recover from RemoteConnectionErrors by waiting.""" @wraps(func) def wrapper(self,*args,**kwds): while True: try: return func(self,*args,**kwds) except RemoteConnectionError: self.wrapped_fs.wait_for_connection() return wrapper # this also checks that wrap_fs_methods works as a class decorator DisconnectRecoveryFS = wrap_fs_methods(recovery_wrapper)(DisconnectRecoveryFS) class TestConnectionManagerFS_disconnect(TestConnectionManagerFS): """Test ConnectionManagerFS's ability to wait for reconnection.""" def setUp(self): self._check_interval = sys.getcheckinterval() sys.setcheckinterval(10) c_fs = ConnectionManagerFS(DisconnectingFS,poll_interval=0.1) self.fs = DisconnectRecoveryFS(c_fs) def tearDown(self): self.fs.close() sys.setcheckinterval(self._check_interval) if __name__ == '__main__': unittest.main() fs-0.5.4/fs/tests/test_iotools.py0000664000175000017500000000273312512525115016751 0ustar willwill00000000000000from __future__ import unicode_literals from fs import iotools import io import unittest from os.path import dirname, join, abspath try: unicode except NameError: unicode = str class OpenFilelike(object): def __init__(self, make_f): self.make_f = make_f @iotools.filelike_to_stream def open(self, path, mode='r', buffering=-1, encoding=None, errors=None, newline=None, line_buffering=False, **kwargs): return self.make_f() def __enter__(self): return self def __exit__(self, *args, **kwargs): self.f.close() class TestIOTools(unittest.TestCase): def get_bin_file(self): path = join(dirname(abspath(__file__)), 'data/UTF-8-demo.txt') return io.open(path, 'rb') def test_make_stream(self): """Test make_stream""" with self.get_bin_file() as f: text = f.read() self.assert_(isinstance(text, bytes)) with self.get_bin_file() as f: with iotools.make_stream("data/UTF-8-demo.txt", f, 'rt') as f2: text = f2.read() self.assert_(isinstance(text, unicode)) def test_decorator(self): """Test filelike_to_stream decorator""" o = OpenFilelike(self.get_bin_file) with o.open('file', 'rb') as f: text = f.read() self.assert_(isinstance(text, bytes)) with o.open('file', 'rt') as f: text = f.read() self.assert_(isinstance(text, unicode)) fs-0.5.4/fs/tests/test_errors.py0000664000175000017500000000146312512525115016574 0ustar willwill00000000000000# -*- encoding: utf-8 -*- """ fs.tests.test_errors: testcases for the fs error classes functions """ import unittest import fs.tests from fs.errors import * import pickle from fs.path import * class TestErrorPickling(unittest.TestCase): def test_pickling(self): def assert_dump_load(e): e2 = pickle.loads(pickle.dumps(e)) self.assertEqual(e.__dict__,e2.__dict__) assert_dump_load(FSError()) assert_dump_load(PathError("/some/path")) assert_dump_load(ResourceNotFoundError("/some/other/path")) assert_dump_load(UnsupportedError("makepony")) class TestFSError(unittest.TestCase): def test_unicode_representation_of_error_with_non_ascii_characters(self): path_error = PathError('/Shïrê/Frødø') _ = unicode(path_error)fs-0.5.4/fs/tests/test_expose.py0000664000175000017500000001246212512525115016564 0ustar willwill00000000000000""" fs.tests.test_expose: testcases for fs.expose and associated FS classes """ import unittest import sys import os import os.path import socket import threading import time from fs.tests import FSTestCases, ThreadingTestCases from fs.tempfs import TempFS from fs.osfs import OSFS from fs.memoryfs import MemoryFS from fs.path import * from fs.errors import * from fs import rpcfs from fs.expose.xmlrpc import RPCFSServer import six from six import PY3, b from fs.tests.test_rpcfs import TestRPCFS try: from fs import sftpfs from fs.expose.sftp import BaseSFTPServer except ImportError: if not PY3: raise import logging logging.getLogger('paramiko').setLevel(logging.ERROR) logging.getLogger('paramiko.transport').setLevel(logging.ERROR) class TestSFTPFS(TestRPCFS): __test__ = not PY3 def makeServer(self,fs,addr): return BaseSFTPServer(addr,fs) def setUp(self): self.startServer() self.fs = sftpfs.SFTPFS(self.server_addr, no_auth=True) def bump(self): # paramiko doesn't like being bumped, just wait for it to timeout. # TODO: do this using a paramiko.Transport() connection pass try: from fs.expose import fuse except ImportError: pass else: from fs.osfs import OSFS class TestFUSE(unittest.TestCase, FSTestCases, ThreadingTestCases): def setUp(self): self.temp_fs = TempFS() self.temp_fs.makedir("root") self.temp_fs.makedir("mount") self.mounted_fs = self.temp_fs.opendir("root") self.mount_point = self.temp_fs.getsyspath("mount") self.fs = OSFS(self.temp_fs.getsyspath("mount")) self.mount_proc = fuse.mount(self.mounted_fs, self.mount_point) def tearDown(self): self.mount_proc.unmount() try: self.temp_fs.close() except OSError: # Sometimes FUSE hangs onto the mountpoint if mount_proc is # forcibly killed. Shell out to fusermount to make sure. fuse.unmount(self.mount_point) self.temp_fs.close() def check(self, p): return self.mounted_fs.exists(p) from fs.expose import dokan if dokan.is_available: from fs.osfs import OSFS class DokanTestCases(FSTestCases): """Specialised testcases for filesystems exposed via Dokan. This modifies some of the standard tests to work around apparent bugs in the current Dokan implementation. """ def test_remove(self): self.fs.createfile("a.txt") self.assertTrue(self.check("a.txt")) self.fs.remove("a.txt") self.assertFalse(self.check("a.txt")) self.assertRaises(ResourceNotFoundError,self.fs.remove,"a.txt") self.fs.makedir("dir1") # This appears to be a bug in Dokan - DeleteFile will happily # delete an empty directory. #self.assertRaises(ResourceInvalidError,self.fs.remove,"dir1") self.fs.createfile("/dir1/a.txt") self.assertTrue(self.check("dir1/a.txt")) self.fs.remove("dir1/a.txt") self.assertFalse(self.check("/dir1/a.txt")) def test_open_on_directory(self): # Dokan seems quite happy to ask me to open a directory and # then treat it like a file. pass def test_settimes(self): # Setting the times does actually work, but there's some sort # of caching effect which prevents them from being read back # out. Disabling the test for now. pass def test_safety_wrapper(self): rawfs = MemoryFS() safefs = dokan.Win32SafetyFS(rawfs) rawfs.setcontents("autoRun.inf", b("evilcodeevilcode")) self.assertTrue(safefs.exists("_autoRun.inf")) self.assertTrue("autoRun.inf" not in safefs.listdir("/")) safefs.setcontents("file:stream",b("test")) self.assertFalse(rawfs.exists("file:stream")) self.assertTrue(rawfs.exists("file__colon__stream")) self.assertTrue("file:stream" in safefs.listdir("/")) class TestDokan(unittest.TestCase,DokanTestCases,ThreadingTestCases): def setUp(self): self.temp_fs = TempFS() self.drive = "K" while os.path.exists(self.drive+":\\") and self.drive <= "Z": self.drive = chr(ord(self.drive) + 1) if self.drive > "Z": raise RuntimeError("no free drive letters") fs_to_mount = OSFS(self.temp_fs.getsyspath("/")) self.mount_proc = dokan.mount(fs_to_mount,self.drive)#,flags=dokan.DOKAN_OPTION_DEBUG|dokan.DOKAN_OPTION_STDERR,numthreads=1) self.fs = OSFS(self.mount_proc.path) def tearDown(self): self.mount_proc.unmount() for _ in xrange(10): try: if self.mount_proc.poll() is None: self.mount_proc.terminate() except EnvironmentError: time.sleep(0.1) else: break else: if self.mount_proc.poll() is None: self.mount_proc.terminate() self.temp_fs.close() if __name__ == '__main__': unittest.main() fs-0.5.4/fs/tests/data/0000755000000000000000000000000012621617365014562 5ustar rootroot00000000000000fs-0.5.4/fs/tests/data/UTF-8-demo.txt0000664000175000017500000003334412512525115017051 0ustar willwill00000000000000 UTF-8 encoded sample plain-text file ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ Markus Kuhn [ˈmaʳkʊs kuːn] — 2002-07-25 The ASCII compatible UTF-8 encoding used in this plain-text file is defined in Unicode, ISO 10646-1, and RFC 2279. Using Unicode/UTF-8, you can write in emails and source code things such as Mathematics and sciences: ∮ E⋅da = Q, n → ∞, ∑ f(i) = ∏ g(i), ⎧⎡⎛┌─────┐⎞⎤⎫ ⎪⎢⎜│a²+b³ ⎟⎥⎪ ∀x∈ℝ: ⌈x⌉ = −⌊−x⌋, α ∧ ¬β = ¬(¬α ∨ β), ⎪⎢⎜│───── ⎟⎥⎪ ⎪⎢⎜⎷ c₈ ⎟⎥⎪ ℕ ⊆ ℕ₀ ⊂ ℤ ⊂ ℚ ⊂ ℝ ⊂ ℂ, ⎨⎢⎜ ⎟⎥⎬ ⎪⎢⎜ ∞ ⎟⎥⎪ ⊥ < a ≠ b ≡ c ≤ d ≪ ⊤ ⇒ (⟦A⟧ ⇔ ⟪B⟫), ⎪⎢⎜ ⎲ ⎟⎥⎪ ⎪⎢⎜ ⎳aⁱ-bⁱ⎟⎥⎪ 2H₂ + O₂ ⇌ 2H₂O, R = 4.7 kΩ, ⌀ 200 mm ⎩⎣⎝i=1 ⎠⎦⎭ Linguistics and dictionaries: ði ıntəˈnæʃənəl fəˈnɛtık əsoʊsiˈeıʃn Y [ˈʏpsilɔn], Yen [jɛn], Yoga [ˈjoːgɑ] APL: ((V⍳V)=⍳⍴V)/V←,V ⌷←⍳→⍴∆∇⊃‾⍎⍕⌈ Nicer typography in plain text files: ╔══════════════════════════════════════════╗ ║ ║ ║ • ‘single’ and “double” quotes ║ ║ ║ ║ • Curly apostrophes: “We’ve been here” ║ ║ ║ ║ • Latin-1 apostrophe and accents: '´` ║ ║ ║ ║ • ‚deutsche‘ „Anführungszeichen“ ║ ║ ║ ║ • †, ‡, ‰, •, 3–4, —, −5/+5, ™, … ║ ║ ║ ║ • ASCII safety test: 1lI|, 0OD, 8B ║ ║ ╭─────────╮ ║ ║ • the euro symbol: │ 14.95 € │ ║ ║ ╰─────────╯ ║ ╚══════════════════════════════════════════╝ Combining characters: STARGΛ̊TE SG-1, a = v̇ = r̈, a⃑ ⊥ b⃑ Greek (in Polytonic): The Greek anthem: Σὲ γνωρίζω ἀπὸ τὴν κόψη τοῦ σπαθιοῦ τὴν τρομερή, σὲ γνωρίζω ἀπὸ τὴν ὄψη ποὺ μὲ βία μετράει τὴ γῆ. ᾿Απ᾿ τὰ κόκκαλα βγαλμένη τῶν ῾Ελλήνων τὰ ἱερά καὶ σὰν πρῶτα ἀνδρειωμένη χαῖρε, ὦ χαῖρε, ᾿Ελευθεριά! From a speech of Demosthenes in the 4th century BC: Οὐχὶ ταὐτὰ παρίσταταί μοι γιγνώσκειν, ὦ ἄνδρες ᾿Αθηναῖοι, ὅταν τ᾿ εἰς τὰ πράγματα ἀποβλέψω καὶ ὅταν πρὸς τοὺς λόγους οὓς ἀκούω· τοὺς μὲν γὰρ λόγους περὶ τοῦ τιμωρήσασθαι Φίλιππον ὁρῶ γιγνομένους, τὰ δὲ πράγματ᾿ εἰς τοῦτο προήκοντα, ὥσθ᾿ ὅπως μὴ πεισόμεθ᾿ αὐτοὶ πρότερον κακῶς σκέψασθαι δέον. οὐδέν οὖν ἄλλο μοι δοκοῦσιν οἱ τὰ τοιαῦτα λέγοντες ἢ τὴν ὑπόθεσιν, περὶ ἧς βουλεύεσθαι, οὐχὶ τὴν οὖσαν παριστάντες ὑμῖν ἁμαρτάνειν. ἐγὼ δέ, ὅτι μέν ποτ᾿ ἐξῆν τῇ πόλει καὶ τὰ αὑτῆς ἔχειν ἀσφαλῶς καὶ Φίλιππον τιμωρήσασθαι, καὶ μάλ᾿ ἀκριβῶς οἶδα· ἐπ᾿ ἐμοῦ γάρ, οὐ πάλαι γέγονεν ταῦτ᾿ ἀμφότερα· νῦν μέντοι πέπεισμαι τοῦθ᾿ ἱκανὸν προλαβεῖν ἡμῖν εἶναι τὴν πρώτην, ὅπως τοὺς συμμάχους σώσομεν. ἐὰν γὰρ τοῦτο βεβαίως ὑπάρξῃ, τότε καὶ περὶ τοῦ τίνα τιμωρήσεταί τις καὶ ὃν τρόπον ἐξέσται σκοπεῖν· πρὶν δὲ τὴν ἀρχὴν ὀρθῶς ὑποθέσθαι, μάταιον ἡγοῦμαι περὶ τῆς τελευτῆς ὁντινοῦν ποιεῖσθαι λόγον. Δημοσθένους, Γ´ ᾿Ολυνθιακὸς Georgian: From a Unicode conference invitation: გთხოვთ ახლავე გაიაროთ რეგისტრაცია Unicode-ის მეათე საერთაშორისო კონფერენციაზე დასასწრებად, რომელიც გაიმართება 10-12 მარტს, ქ. მაინცში, გერმანიაში. კონფერენცია შეჰკრებს ერთად მსოფლიოს ექსპერტებს ისეთ დარგებში როგორიცაა ინტერნეტი და Unicode-ი, ინტერნაციონალიზაცია და ლოკალიზაცია, Unicode-ის გამოყენება ოპერაციულ სისტემებსა, და გამოყენებით პროგრამებში, შრიფტებში, ტექსტების დამუშავებასა და მრავალენოვან კომპიუტერულ სისტემებში. Russian: From a Unicode conference invitation: Зарегистрируйтесь сейчас на Десятую Международную Конференцию по Unicode, которая состоится 10-12 марта 1997 года в Майнце в Германии. Конференция соберет широкий круг экспертов по вопросам глобального Интернета и Unicode, локализации и интернационализации, воплощению и применению Unicode в различных операционных системах и программных приложениях, шрифтах, верстке и многоязычных компьютерных системах. Thai (UCS Level 2): Excerpt from a poetry on The Romance of The Three Kingdoms (a Chinese classic 'San Gua'): [----------------------------|------------------------] ๏ แผ่นดินฮั่นเสื่อมโทรมแสนสังเวช พระปกเกศกองบู๊กู้ขึ้นใหม่ สิบสองกษัตริย์ก่อนหน้าแลถัดไป สององค์ไซร้โง่เขลาเบาปัญญา ทรงนับถือขันทีเป็นที่พึ่ง บ้านเมืองจึงวิปริตเป็นนักหนา โฮจิ๋นเรียกทัพทั่วหัวเมืองมา หมายจะฆ่ามดชั่วตัวสำคัญ เหมือนขับไสไล่เสือจากเคหา รับหมาป่าเข้ามาเลยอาสัญ ฝ่ายอ้องอุ้นยุแยกให้แตกกัน ใช้สาวนั้นเป็นชนวนชื่นชวนใจ พลันลิฉุยกุยกีกลับก่อเหตุ ช่างอาเพศจริงหนาฟ้าร้องไห้ ต้องรบราฆ่าฟันจนบรรลัย ฤๅหาใครค้ำชูกู้บรรลังก์ ฯ (The above is a two-column text. If combining characters are handled correctly, the lines of the second column should be aligned with the | character above.) Ethiopian: Proverbs in the Amharic language: ሰማይ አይታረስ ንጉሥ አይከሰስ። ብላ ካለኝ እንደአባቴ በቆመጠኝ። ጌጥ ያለቤቱ ቁምጥና ነው። ደሀ በሕልሙ ቅቤ ባይጠጣ ንጣት በገደለው። የአፍ ወለምታ በቅቤ አይታሽም። አይጥ በበላ ዳዋ ተመታ። ሲተረጉሙ ይደረግሙ። ቀስ በቀስ፥ ዕንቁላል በእግሩ ይሄዳል። ድር ቢያብር አንበሳ ያስር። ሰው እንደቤቱ እንጅ እንደ ጉረቤቱ አይተዳደርም። እግዜር የከፈተውን ጉሮሮ ሳይዘጋው አይድርም። የጎረቤት ሌባ፥ ቢያዩት ይስቅ ባያዩት ያጠልቅ። ሥራ ከመፍታት ልጄን ላፋታት። ዓባይ ማደሪያ የለው፥ ግንድ ይዞ ይዞራል። የእስላም አገሩ መካ የአሞራ አገሩ ዋርካ። ተንጋሎ ቢተፉ ተመልሶ ባፉ። ወዳጅህ ማር ቢሆን ጨርስህ አትላሰው። እግርህን በፍራሽህ ልክ ዘርጋ። Runes: ᚻᛖ ᚳᚹᚫᚦ ᚦᚫᛏ ᚻᛖ ᛒᚢᛞᛖ ᚩᚾ ᚦᚫᛗ ᛚᚪᚾᛞᛖ ᚾᚩᚱᚦᚹᛖᚪᚱᛞᚢᛗ ᚹᛁᚦ ᚦᚪ ᚹᛖᛥᚫ (Old English, which transcribed into Latin reads 'He cwaeth that he bude thaem lande northweardum with tha Westsae.' and means 'He said that he lived in the northern land near the Western Sea.') Braille: ⡌⠁⠧⠑ ⠼⠁⠒ ⡍⠜⠇⠑⠹⠰⠎ ⡣⠕⠌ ⡍⠜⠇⠑⠹ ⠺⠁⠎ ⠙⠑⠁⠙⠒ ⠞⠕ ⠃⠑⠛⠔ ⠺⠊⠹⠲ ⡹⠻⠑ ⠊⠎ ⠝⠕ ⠙⠳⠃⠞ ⠱⠁⠞⠑⠧⠻ ⠁⠃⠳⠞ ⠹⠁⠞⠲ ⡹⠑ ⠗⠑⠛⠊⠌⠻ ⠕⠋ ⠙⠊⠎ ⠃⠥⠗⠊⠁⠇ ⠺⠁⠎ ⠎⠊⠛⠝⠫ ⠃⠹ ⠹⠑ ⠊⠇⠻⠛⠹⠍⠁⠝⠂ ⠹⠑ ⠊⠇⠻⠅⠂ ⠹⠑ ⠥⠝⠙⠻⠞⠁⠅⠻⠂ ⠁⠝⠙ ⠹⠑ ⠡⠊⠑⠋ ⠍⠳⠗⠝⠻⠲ ⡎⠊⠗⠕⠕⠛⠑ ⠎⠊⠛⠝⠫ ⠊⠞⠲ ⡁⠝⠙ ⡎⠊⠗⠕⠕⠛⠑⠰⠎ ⠝⠁⠍⠑ ⠺⠁⠎ ⠛⠕⠕⠙ ⠥⠏⠕⠝ ⠰⡡⠁⠝⠛⠑⠂ ⠋⠕⠗ ⠁⠝⠹⠹⠔⠛ ⠙⠑ ⠡⠕⠎⠑ ⠞⠕ ⠏⠥⠞ ⠙⠊⠎ ⠙⠁⠝⠙ ⠞⠕⠲ ⡕⠇⠙ ⡍⠜⠇⠑⠹ ⠺⠁⠎ ⠁⠎ ⠙⠑⠁⠙ ⠁⠎ ⠁ ⠙⠕⠕⠗⠤⠝⠁⠊⠇⠲ ⡍⠔⠙⠖ ⡊ ⠙⠕⠝⠰⠞ ⠍⠑⠁⠝ ⠞⠕ ⠎⠁⠹ ⠹⠁⠞ ⡊ ⠅⠝⠪⠂ ⠕⠋ ⠍⠹ ⠪⠝ ⠅⠝⠪⠇⠫⠛⠑⠂ ⠱⠁⠞ ⠹⠻⠑ ⠊⠎ ⠏⠜⠞⠊⠊⠥⠇⠜⠇⠹ ⠙⠑⠁⠙ ⠁⠃⠳⠞ ⠁ ⠙⠕⠕⠗⠤⠝⠁⠊⠇⠲ ⡊ ⠍⠊⠣⠞ ⠙⠁⠧⠑ ⠃⠑⠲ ⠔⠊⠇⠔⠫⠂ ⠍⠹⠎⠑⠇⠋⠂ ⠞⠕ ⠗⠑⠛⠜⠙ ⠁ ⠊⠕⠋⠋⠔⠤⠝⠁⠊⠇ ⠁⠎ ⠹⠑ ⠙⠑⠁⠙⠑⠌ ⠏⠊⠑⠊⠑ ⠕⠋ ⠊⠗⠕⠝⠍⠕⠝⠛⠻⠹ ⠔ ⠹⠑ ⠞⠗⠁⠙⠑⠲ ⡃⠥⠞ ⠹⠑ ⠺⠊⠎⠙⠕⠍ ⠕⠋ ⠳⠗ ⠁⠝⠊⠑⠌⠕⠗⠎ ⠊⠎ ⠔ ⠹⠑ ⠎⠊⠍⠊⠇⠑⠆ ⠁⠝⠙ ⠍⠹ ⠥⠝⠙⠁⠇⠇⠪⠫ ⠙⠁⠝⠙⠎ ⠩⠁⠇⠇ ⠝⠕⠞ ⠙⠊⠌⠥⠗⠃ ⠊⠞⠂ ⠕⠗ ⠹⠑ ⡊⠳⠝⠞⠗⠹⠰⠎ ⠙⠕⠝⠑ ⠋⠕⠗⠲ ⡹⠳ ⠺⠊⠇⠇ ⠹⠻⠑⠋⠕⠗⠑ ⠏⠻⠍⠊⠞ ⠍⠑ ⠞⠕ ⠗⠑⠏⠑⠁⠞⠂ ⠑⠍⠏⠙⠁⠞⠊⠊⠁⠇⠇⠹⠂ ⠹⠁⠞ ⡍⠜⠇⠑⠹ ⠺⠁⠎ ⠁⠎ ⠙⠑⠁⠙ ⠁⠎ ⠁ ⠙⠕⠕⠗⠤⠝⠁⠊⠇⠲ (The first couple of paragraphs of "A Christmas Carol" by Dickens) Compact font selection example text: ABCDEFGHIJKLMNOPQRSTUVWXYZ /0123456789 abcdefghijklmnopqrstuvwxyz £©µÀÆÖÞßéöÿ –—‘“”„†•…‰™œŠŸž€ ΑΒΓΔΩαβγδω АБВГДабвгд ∀∂∈ℝ∧∪≡∞ ↑↗↨↻⇣ ┐┼╔╘░►☺♀ fi�⑀₂ἠḂӥẄɐː⍎אԱა Greetings in various languages: Hello world, Καλημέρα κόσμε, コンニチハ Box drawing alignment tests: █ ▉ ╔══╦══╗ ┌──┬──┐ ╭──┬──╮ ╭──┬──╮ ┏━━┳━━┓ ┎┒┏┑ ╷ ╻ ┏┯┓ ┌┰┐ ▊ ╱╲╱╲╳╳╳ ║┌─╨─┐║ │╔═╧═╗│ │╒═╪═╕│ │╓─╁─╖│ ┃┌─╂─┐┃ ┗╃╄┙ ╶┼╴╺╋╸┠┼┨ ┝╋┥ ▋ ╲╱╲╱╳╳╳ ║│╲ ╱│║ │║ ║│ ││ │ ││ │║ ┃ ║│ ┃│ ╿ │┃ ┍╅╆┓ ╵ ╹ ┗┷┛ └┸┘ ▌ ╱╲╱╲╳╳╳ ╠╡ ╳ ╞╣ ├╢ ╟┤ ├┼─┼─┼┤ ├╫─╂─╫┤ ┣┿╾┼╼┿┫ ┕┛┖┚ ┌┄┄┐ ╎ ┏┅┅┓ ┋ ▍ ╲╱╲╱╳╳╳ ║│╱ ╲│║ │║ ║│ ││ │ ││ │║ ┃ ║│ ┃│ ╽ │┃ ░░▒▒▓▓██ ┊ ┆ ╎ ╏ ┇ ┋ ▎ ║└─╥─┘║ │╚═╤═╝│ │╘═╪═╛│ │╙─╀─╜│ ┃└─╂─┘┃ ░░▒▒▓▓██ ┊ ┆ ╎ ╏ ┇ ┋ ▏ ╚══╩══╝ └──┴──┘ ╰──┴──╯ ╰──┴──╯ ┗━━┻━━┛ ▗▄▖▛▀▜ └╌╌┘ ╎ ┗╍╍┛ ┋ ▁▂▃▄▅▆▇█ ▝▀▘▙▄▟ fs-0.5.4/fs/tests/test_xattr.py0000664000175000017500000001561312512525115016424 0ustar willwill00000000000000""" fs.tests.test_xattr: testcases for extended attribute support """ import unittest import os from fs.path import * from fs.errors import * from fs.tests import FSTestCases from six import b class XAttrTestCases: """Testcases for filesystems providing extended attribute support. This class should be used as a mixin to the unittest.TestCase class for filesystems that provide extended attribute support. """ def test_getsetdel(self): def do_getsetdel(p): self.assertEqual(self.fs.getxattr(p,"xattr1"),None) self.fs.setxattr(p,"xattr1","value1") self.assertEqual(self.fs.getxattr(p,"xattr1"),"value1") self.fs.delxattr(p,"xattr1") self.assertEqual(self.fs.getxattr(p,"xattr1"),None) self.fs.setcontents("test.txt",b("hello")) do_getsetdel("test.txt") self.assertRaises(ResourceNotFoundError,self.fs.getxattr,"test2.txt","xattr1") self.fs.makedir("mystuff") self.fs.setcontents("/mystuff/test.txt",b("")) do_getsetdel("mystuff") do_getsetdel("mystuff/test.txt") def test_list_xattrs(self): def do_list(p): self.assertEquals(sorted(self.fs.listxattrs(p)),[]) self.fs.setxattr(p,"xattr1","value1") self.assertEquals(self.fs.getxattr(p,"xattr1"),"value1") self.assertEquals(sorted(self.fs.listxattrs(p)),["xattr1"]) self.assertTrue(isinstance(self.fs.listxattrs(p)[0],unicode)) self.fs.setxattr(p,"attr2","value2") self.assertEquals(sorted(self.fs.listxattrs(p)),["attr2","xattr1"]) self.assertTrue(isinstance(self.fs.listxattrs(p)[0],unicode)) self.assertTrue(isinstance(self.fs.listxattrs(p)[1],unicode)) self.fs.delxattr(p,"xattr1") self.assertEquals(sorted(self.fs.listxattrs(p)),["attr2"]) self.fs.delxattr(p,"attr2") self.assertEquals(sorted(self.fs.listxattrs(p)),[]) self.fs.setcontents("test.txt",b("hello")) do_list("test.txt") self.fs.makedir("mystuff") self.fs.setcontents("/mystuff/test.txt",b("")) do_list("mystuff") do_list("mystuff/test.txt") def test_copy_xattrs(self): self.fs.setcontents("a.txt",b("content")) self.fs.setxattr("a.txt","myattr","myvalue") self.fs.setxattr("a.txt","testattr","testvalue") self.fs.makedir("stuff") self.fs.copy("a.txt","stuff/a.txt") self.assertTrue(self.fs.exists("stuff/a.txt")) self.assertEquals(self.fs.getxattr("stuff/a.txt","myattr"),"myvalue") self.assertEquals(self.fs.getxattr("stuff/a.txt","testattr"),"testvalue") self.assertEquals(self.fs.getxattr("a.txt","myattr"),"myvalue") self.assertEquals(self.fs.getxattr("a.txt","testattr"),"testvalue") self.fs.setxattr("stuff","dirattr","a directory") self.fs.copydir("stuff","stuff2") self.assertEquals(self.fs.getxattr("stuff2/a.txt","myattr"),"myvalue") self.assertEquals(self.fs.getxattr("stuff2/a.txt","testattr"),"testvalue") self.assertEquals(self.fs.getxattr("stuff2","dirattr"),"a directory") self.assertEquals(self.fs.getxattr("stuff","dirattr"),"a directory") def test_move_xattrs(self): self.fs.setcontents("a.txt",b("content")) self.fs.setxattr("a.txt","myattr","myvalue") self.fs.setxattr("a.txt","testattr","testvalue") self.fs.makedir("stuff") self.fs.move("a.txt","stuff/a.txt") self.assertTrue(self.fs.exists("stuff/a.txt")) self.assertEquals(self.fs.getxattr("stuff/a.txt","myattr"),"myvalue") self.assertEquals(self.fs.getxattr("stuff/a.txt","testattr"),"testvalue") self.fs.setxattr("stuff","dirattr","a directory") self.fs.movedir("stuff","stuff2") self.assertEquals(self.fs.getxattr("stuff2/a.txt","myattr"),"myvalue") self.assertEquals(self.fs.getxattr("stuff2/a.txt","testattr"),"testvalue") self.assertEquals(self.fs.getxattr("stuff2","dirattr"),"a directory") def test_remove_file(self): def listxattrs(path): return list(self.fs.listxattrs(path)) # Check that xattrs aren't preserved after a file is removed self.fs.createfile("myfile") self.assertEquals(listxattrs("myfile"),[]) self.fs.setxattr("myfile","testattr","testvalue") self.assertEquals(listxattrs("myfile"),["testattr"]) self.fs.remove("myfile") self.assertRaises(ResourceNotFoundError,listxattrs,"myfile") self.fs.createfile("myfile") self.assertEquals(listxattrs("myfile"),[]) self.fs.setxattr("myfile","testattr2","testvalue2") self.assertEquals(listxattrs("myfile"),["testattr2"]) self.assertEquals(self.fs.getxattr("myfile","testattr2"),"testvalue2") # Check that removing a file without xattrs still works self.fs.createfile("myfile2") self.fs.remove("myfile2") def test_remove_dir(self): def listxattrs(path): return list(self.fs.listxattrs(path)) # Check that xattrs aren't preserved after a dir is removed self.fs.makedir("mydir") self.assertEquals(listxattrs("mydir"),[]) self.fs.setxattr("mydir","testattr","testvalue") self.assertEquals(listxattrs("mydir"),["testattr"]) self.fs.removedir("mydir") self.assertRaises(ResourceNotFoundError,listxattrs,"mydir") self.fs.makedir("mydir") self.assertEquals(listxattrs("mydir"),[]) self.fs.setxattr("mydir","testattr2","testvalue2") self.assertEquals(listxattrs("mydir"),["testattr2"]) self.assertEquals(self.fs.getxattr("mydir","testattr2"),"testvalue2") # Check that removing a dir without xattrs still works self.fs.makedir("mydir2") self.fs.removedir("mydir2") # Check that forcibly removing a dir with xattrs still works self.fs.makedir("mydir3") self.fs.createfile("mydir3/testfile") self.fs.removedir("mydir3",force=True) self.assertFalse(self.fs.exists("mydir3")) from fs.xattrs import ensure_xattrs from fs import tempfs class TestXAttr_TempFS(unittest.TestCase,FSTestCases,XAttrTestCases): def setUp(self): fs = tempfs.TempFS() self.fs = ensure_xattrs(fs) def tearDown(self): try: td = self.fs._temp_dir except AttributeError: td = self.fs.wrapped_fs._temp_dir self.fs.close() self.assert_(not os.path.exists(td)) def check(self, p): try: td = self.fs._temp_dir except AttributeError: td = self.fs.wrapped_fs._temp_dir return os.path.exists(os.path.join(td, relpath(p))) from fs import memoryfs class TestXAttr_MemoryFS(unittest.TestCase,FSTestCases,XAttrTestCases): def setUp(self): self.fs = ensure_xattrs(memoryfs.MemoryFS()) def check(self, p): return self.fs.exists(p) fs-0.5.4/fs/tests/test_zipfs.py0000664000175000017500000001373012512525115016413 0ustar willwill00000000000000""" fs.tests.test_zipfs: testcases for the ZipFS class """ import unittest import os import random import zipfile import tempfile import shutil import fs.tests from fs.path import * from fs import zipfs from six import PY3, b class TestReadZipFS(unittest.TestCase): def setUp(self): self.temp_filename = "".join(random.choice("abcdefghijklmnopqrstuvwxyz") for _ in range(6))+".zip" self.temp_filename = os.path.join(tempfile.gettempdir(), self.temp_filename) self.zf = zipfile.ZipFile(self.temp_filename, "w") zf = self.zf zf.writestr("a.txt", b("Hello, World!")) zf.writestr("b.txt", b("b")) zf.writestr("1.txt", b("1")) zf.writestr("foo/bar/baz.txt", b("baz")) zf.writestr("foo/second.txt", b("hai")) zf.close() self.fs = zipfs.ZipFS(self.temp_filename, "r") def tearDown(self): self.fs.close() os.remove(self.temp_filename) def check(self, p): try: self.zipfile.getinfo(p) return True except: return False def test_reads(self): def read_contents(path): f = self.fs.open(path, 'rb') contents = f.read() return contents def check_contents(path, expected): self.assert_(read_contents(path) == expected) check_contents("a.txt", b("Hello, World!")) check_contents("1.txt", b("1")) check_contents("foo/bar/baz.txt", b("baz")) def test_getcontents(self): def read_contents(path): return self.fs.getcontents(path, 'rb') def check_contents(path, expected): self.assert_(read_contents(path) == expected) check_contents("a.txt", b("Hello, World!")) check_contents("1.txt", b("1")) check_contents("foo/bar/baz.txt", b("baz")) def test_is(self): self.assert_(self.fs.isfile('a.txt')) self.assert_(self.fs.isfile('1.txt')) self.assert_(self.fs.isfile('foo/bar/baz.txt')) self.assert_(self.fs.isdir('foo')) self.assert_(self.fs.isdir('foo/bar')) self.assert_(self.fs.exists('a.txt')) self.assert_(self.fs.exists('1.txt')) self.assert_(self.fs.exists('foo/bar/baz.txt')) self.assert_(self.fs.exists('foo')) self.assert_(self.fs.exists('foo/bar')) def test_listdir(self): def check_listing(path, expected): dir_list = self.fs.listdir(path) self.assert_(sorted(dir_list) == sorted(expected)) for item in dir_list: self.assert_(isinstance(item, unicode)) check_listing('/', ['a.txt', '1.txt', 'foo', 'b.txt']) check_listing('foo', ['second.txt', 'bar']) check_listing('foo/bar', ['baz.txt']) class TestWriteZipFS(unittest.TestCase): def setUp(self): self.temp_filename = "".join(random.choice("abcdefghijklmnopqrstuvwxyz") for _ in range(6))+".zip" self.temp_filename = os.path.join(tempfile.gettempdir(), self.temp_filename) zip_fs = zipfs.ZipFS(self.temp_filename, 'w') def makefile(filename, contents): if dirname(filename): zip_fs.makedir(dirname(filename), recursive=True, allow_recreate=True) f = zip_fs.open(filename, 'wb') f.write(contents) f.close() makefile("a.txt", b("Hello, World!")) makefile("b.txt", b("b")) makefile(u"\N{GREEK SMALL LETTER ALPHA}/\N{GREEK CAPITAL LETTER OMEGA}.txt", b("this is the alpha and the omega")) makefile("foo/bar/baz.txt", b("baz")) makefile("foo/second.txt", b("hai")) zip_fs.close() def tearDown(self): os.remove(self.temp_filename) def test_valid(self): zf = zipfile.ZipFile(self.temp_filename, "r") self.assert_(zf.testzip() is None) zf.close() def test_creation(self): zf = zipfile.ZipFile(self.temp_filename, "r") def check_contents(filename, contents): if PY3: zcontents = zf.read(filename) else: zcontents = zf.read(filename.encode("CP437")) self.assertEqual(contents, zcontents) check_contents("a.txt", b("Hello, World!")) check_contents("b.txt", b("b")) check_contents("foo/bar/baz.txt", b("baz")) check_contents("foo/second.txt", b("hai")) check_contents(u"\N{GREEK SMALL LETTER ALPHA}/\N{GREEK CAPITAL LETTER OMEGA}.txt", b("this is the alpha and the omega")) class TestAppendZipFS(TestWriteZipFS): def setUp(self): self.temp_filename = "".join(random.choice("abcdefghijklmnopqrstuvwxyz") for _ in range(6))+".zip" self.temp_filename = os.path.join(tempfile.gettempdir(), self.temp_filename) zip_fs = zipfs.ZipFS(self.temp_filename, 'w') def makefile(filename, contents): if dirname(filename): zip_fs.makedir(dirname(filename), recursive=True, allow_recreate=True) f = zip_fs.open(filename, 'wb') f.write(contents) f.close() makefile("a.txt", b("Hello, World!")) makefile("b.txt", b("b")) zip_fs.close() zip_fs = zipfs.ZipFS(self.temp_filename, 'a') makefile("foo/bar/baz.txt", b("baz")) makefile(u"\N{GREEK SMALL LETTER ALPHA}/\N{GREEK CAPITAL LETTER OMEGA}.txt", b("this is the alpha and the omega")) makefile("foo/second.txt", b("hai")) zip_fs.close() class TestZipFSErrors(unittest.TestCase): def setUp(self): self.workdir = tempfile.mkdtemp() def tearDown(self): shutil.rmtree(self.workdir) def test_bogus_zipfile(self): badzip = os.path.join(self.workdir,"bad.zip") f = open(badzip,"wb") f.write(b("I'm not really a zipfile")) f.close() self.assertRaises(zipfs.ZipOpenError,zipfs.ZipFS,badzip) def test_missing_zipfile(self): missingzip = os.path.join(self.workdir,"missing.zip") self.assertRaises(zipfs.ZipNotFoundError,zipfs.ZipFS,missingzip) fs-0.5.4/fs/tests/test_multifs.py0000664000175000017500000000530512512525115016742 0ustar willwill00000000000000from fs.multifs import MultiFS from fs.memoryfs import MemoryFS import unittest from six import b class TestMultiFS(unittest.TestCase): def test_auto_close(self): """Test MultiFS auto close is working""" multi_fs = MultiFS() m1 = MemoryFS() m2 = MemoryFS() multi_fs.addfs('m1', m1) multi_fs.addfs('m2', m2) self.assert_(not m1.closed) self.assert_(not m2.closed) multi_fs.close() self.assert_(m1.closed) self.assert_(m2.closed) def test_no_auto_close(self): """Test MultiFS auto close can be disables""" multi_fs = MultiFS(auto_close=False) m1 = MemoryFS() m2 = MemoryFS() multi_fs.addfs('m1', m1) multi_fs.addfs('m2', m2) self.assert_(not m1.closed) self.assert_(not m2.closed) multi_fs.close() self.assert_(not m1.closed) self.assert_(not m2.closed) def test_priority(self): """Test priority order is working""" m1 = MemoryFS() m2 = MemoryFS() m3 = MemoryFS() m1.setcontents("name", b("m1")) m2.setcontents("name", b("m2")) m3.setcontents("name", b("m3")) multi_fs = MultiFS(auto_close=False) multi_fs.addfs("m1", m1) multi_fs.addfs("m2", m2) multi_fs.addfs("m3", m3) self.assert_(multi_fs.getcontents("name") == b("m3")) m1 = MemoryFS() m2 = MemoryFS() m3 = MemoryFS() m1.setcontents("name", b("m1")) m2.setcontents("name", b("m2")) m3.setcontents("name", b("m3")) multi_fs = MultiFS(auto_close=False) multi_fs.addfs("m1", m1) multi_fs.addfs("m2", m2, priority=10) multi_fs.addfs("m3", m3) self.assert_(multi_fs.getcontents("name") == b("m2")) m1 = MemoryFS() m2 = MemoryFS() m3 = MemoryFS() m1.setcontents("name", b("m1")) m2.setcontents("name", b("m2")) m3.setcontents("name", b("m3")) multi_fs = MultiFS(auto_close=False) multi_fs.addfs("m1", m1) multi_fs.addfs("m2", m2, priority=10) multi_fs.addfs("m3", m3, priority=10) self.assert_(multi_fs.getcontents("name") == b("m3")) m1 = MemoryFS() m2 = MemoryFS() m3 = MemoryFS() m1.setcontents("name", b("m1")) m2.setcontents("name", b("m2")) m3.setcontents("name", b("m3")) multi_fs = MultiFS(auto_close=False) multi_fs.addfs("m1", m1, priority=11) multi_fs.addfs("m2", m2, priority=10) multi_fs.addfs("m3", m3, priority=10) self.assert_(multi_fs.getcontents("name") == b("m1")) fs-0.5.4/fs/tests/test_archivefs.py0000664000175000017500000001467312512525115017241 0ustar willwill00000000000000""" fs.tests.test_archivefs: testcases for the ArchiveFS class """ import unittest import os import random import zipfile import tempfile import shutil import fs.tests from fs.path import * try: from fs.contrib import archivefs except ImportError: libarchive_available = False else: libarchive_available = True from six import PY3, b class TestReadArchiveFS(unittest.TestCase): __test__ = libarchive_available def setUp(self): self.temp_filename = "".join(random.choice("abcdefghijklmnopqrstuvwxyz") for _ in range(6))+".zip" self.temp_filename = os.path.join(tempfile.gettempdir(), self.temp_filename) self.zf = zipfile.ZipFile(self.temp_filename, "w") zf = self.zf zf.writestr("a.txt", b("Hello, World!")) zf.writestr("b.txt", b("b")) zf.writestr("1.txt", b("1")) zf.writestr("foo/bar/baz.txt", b("baz")) zf.writestr("foo/second.txt", b("hai")) zf.close() self.fs = archivefs.ArchiveFS(self.temp_filename, "r") def tearDown(self): self.fs.close() os.remove(self.temp_filename) def check(self, p): try: self.zipfile.getinfo(p) return True except: return False def test_reads(self): def read_contents(path): f = self.fs.open(path) contents = f.read() return contents def check_contents(path, expected): self.assert_(read_contents(path)==expected) check_contents("a.txt", b("Hello, World!")) check_contents("1.txt", b("1")) check_contents("foo/bar/baz.txt", b("baz")) def test_getcontents(self): def read_contents(path): return self.fs.getcontents(path) def check_contents(path, expected): self.assert_(read_contents(path)==expected) check_contents("a.txt", b("Hello, World!")) check_contents("1.txt", b("1")) check_contents("foo/bar/baz.txt", b("baz")) def test_is(self): self.assert_(self.fs.isfile('a.txt')) self.assert_(self.fs.isfile('1.txt')) self.assert_(self.fs.isfile('foo/bar/baz.txt')) self.assert_(self.fs.isdir('foo')) self.assert_(self.fs.isdir('foo/bar')) self.assert_(self.fs.exists('a.txt')) self.assert_(self.fs.exists('1.txt')) self.assert_(self.fs.exists('foo/bar/baz.txt')) self.assert_(self.fs.exists('foo')) self.assert_(self.fs.exists('foo/bar')) def test_listdir(self): def check_listing(path, expected): dir_list = self.fs.listdir(path) self.assert_(sorted(dir_list) == sorted(expected)) for item in dir_list: self.assert_(isinstance(item,unicode)) check_listing('/', ['a.txt', '1.txt', 'foo', 'b.txt']) check_listing('foo', ['second.txt', 'bar']) check_listing('foo/bar', ['baz.txt']) class TestWriteArchiveFS(unittest.TestCase): __test__ = libarchive_available def setUp(self): self.temp_filename = "".join(random.choice("abcdefghijklmnopqrstuvwxyz") for _ in range(6))+".zip" self.temp_filename = os.path.join(tempfile.gettempdir(), self.temp_filename) archive_fs = archivefs.ArchiveFS(self.temp_filename, format='zip', mode='w') def makefile(filename, contents): if dirname(filename): archive_fs.makedir(dirname(filename), recursive=True, allow_recreate=True) f = archive_fs.open(filename, 'wb') f.write(contents) f.close() makefile("a.txt", b("Hello, World!")) makefile("b.txt", b("b")) makefile(u"\N{GREEK SMALL LETTER ALPHA}/\N{GREEK CAPITAL LETTER OMEGA}.txt", b("this is the alpha and the omega")) makefile("foo/bar/baz.txt", b("baz")) makefile("foo/second.txt", b("hai")) archive_fs.close() def tearDown(self): os.remove(self.temp_filename) def test_valid(self): zf = zipfile.ZipFile(self.temp_filename, "r") self.assert_(zf.testzip() is None) zf.close() def test_creation(self): zf = zipfile.ZipFile(self.temp_filename, "r") def check_contents(filename, contents): if PY3: zcontents = zf.read(filename) else: zcontents = zf.read(filename.encode(archivefs.ENCODING)) self.assertEqual(contents, zcontents) check_contents("a.txt", b("Hello, World!")) check_contents("b.txt", b("b")) check_contents("foo/bar/baz.txt", b("baz")) check_contents("foo/second.txt", b("hai")) check_contents(u"\N{GREEK SMALL LETTER ALPHA}/\N{GREEK CAPITAL LETTER OMEGA}.txt", b("this is the alpha and the omega")) #~ class TestAppendArchiveFS(TestWriteArchiveFS): #~ __test__ = libarchive_available #~ def setUp(self): #~ self.temp_filename = "".join(random.choice("abcdefghijklmnopqrstuvwxyz") for _ in range(6))+".zip" #~ self.temp_filename = os.path.join(tempfile.gettempdir(), self.temp_filename) #~ zip_fs = zipfs.ZipFS(self.temp_filename, 'w') #~ def makefile(filename, contents): #~ if dirname(filename): #~ zip_fs.makedir(dirname(filename), recursive=True, allow_recreate=True) #~ f = zip_fs.open(filename, 'wb') #~ f.write(contents) #~ f.close() #~ makefile("a.txt", b("Hello, World!")) #~ makefile("b.txt", b("b")) #~ zip_fs.close() #~ zip_fs = zipfs.ZipFS(self.temp_filename, 'a') #~ makefile("foo/bar/baz.txt", b("baz")) #~ makefile(u"\N{GREEK SMALL LETTER ALPHA}/\N{GREEK CAPITAL LETTER OMEGA}.txt", b("this is the alpha and the omega")) #~ makefile("foo/second.txt", b("hai")) #~ zip_fs.close() #~ class TestArchiveFSErrors(unittest.TestCase): #~ __test__ = libarchive_available #~ def setUp(self): #~ self.workdir = tempfile.mkdtemp() #~ def tearDown(self): #~ shutil.rmtree(self.workdir) #~ def test_bogus_zipfile(self): #~ badzip = os.path.join(self.workdir,"bad.zip") #~ f = open(badzip,"wb") #~ f.write(b("I'm not really a zipfile")) #~ f.close() #~ self.assertRaises(zipfs.ZipOpenError,zipfs.ZipFS,badzip) #~ def test_missing_zipfile(self): #~ missingzip = os.path.join(self.workdir,"missing.zip") #~ self.assertRaises(zipfs.ZipNotFoundError,zipfs.ZipFS,missingzip) if __name__ == '__main__': unittest.main() fs-0.5.4/fs/tests/test_mountfs.py0000664000175000017500000000570412512525115016755 0ustar willwill00000000000000from fs.mountfs import MountFS from fs.memoryfs import MemoryFS import unittest class TestMountFS(unittest.TestCase): def test_auto_close(self): """Test MountFS auto close is working""" multi_fs = MountFS() m1 = MemoryFS() m2 = MemoryFS() multi_fs.mount('/m1', m1) multi_fs.mount('/m2', m2) self.assert_(not m1.closed) self.assert_(not m2.closed) multi_fs.close() self.assert_(m1.closed) self.assert_(m2.closed) def test_no_auto_close(self): """Test MountFS auto close can be disabled""" multi_fs = MountFS(auto_close=False) m1 = MemoryFS() m2 = MemoryFS() multi_fs.mount('/m1', m1) multi_fs.mount('/m2', m2) self.assert_(not m1.closed) self.assert_(not m2.closed) multi_fs.close() self.assert_(not m1.closed) self.assert_(not m2.closed) def test_mountfile(self): """Test mounting a file""" quote = b"""If you wish to make an apple pie from scratch, you must first invent the universe.""" mem_fs = MemoryFS() mem_fs.makedir('foo') mem_fs.setcontents('foo/bar.txt', quote) foo_dir = mem_fs.opendir('foo') mount_fs = MountFS() mount_fs.mountfile('bar.txt', foo_dir.open, foo_dir.getinfo) self.assert_(mount_fs.isdir('/')) self.assert_(mount_fs.isdir('./')) self.assert_(mount_fs.isdir('')) # Check we can see the mounted file in the dir list self.assertEqual(mount_fs.listdir(), ["bar.txt"]) self.assert_(not mount_fs.exists('nobodyhere.txt')) self.assert_(mount_fs.exists('bar.txt')) self.assert_(mount_fs.isfile('bar.txt')) self.assert_(not mount_fs.isdir('bar.txt')) # Check open and getinfo callables self.assertEqual(mount_fs.getcontents('bar.txt'), quote) self.assertEqual(mount_fs.getsize('bar.txt'), len(quote)) # Check changes are written back mem_fs.setcontents('foo/bar.txt', 'baz') self.assertEqual(mount_fs.getcontents('bar.txt'), b'baz') self.assertEqual(mount_fs.getsize('bar.txt'), len('baz')) # Check changes are written to the original fs self.assertEqual(mem_fs.getcontents('foo/bar.txt'), b'baz') self.assertEqual(mem_fs.getsize('foo/bar.txt'), len('baz')) # Check unmount self.assert_(mount_fs.unmount("bar.txt")) self.assertEqual(mount_fs.listdir(), []) self.assert_(not mount_fs.exists('bar.txt')) # Check unount a second time is a null op, and returns False self.assertFalse(mount_fs.unmount("bar.txt")) def test_empty(self): """Test MountFS with nothing mounted.""" mount_fs = MountFS() self.assertEqual(mount_fs.getinfo(''), {}) self.assertEqual(mount_fs.getxattr('', 'yo'), None) self.assertEqual(mount_fs.listdir(), []) self.assertEqual(list(mount_fs.ilistdir()), []) fs-0.5.4/fs/tests/test_watch.py0000664000175000017500000001742212512525115016370 0ustar willwill00000000000000""" fs.tests.test_watch: testcases for change watcher support """ import os import sys import time import gc import pickle import unittest from fs.path import * from fs.errors import * from fs.watch import * from fs.tests import FSTestCases try: from fs.osfs import watch_inotify except ImportError: watch_inotify = None if sys.platform == "win32": try: from fs.osfs import watch_win32 except ImportError: watch_win32 = None else: watch_win32 = None import logging logging.getLogger('pyinotify').setLevel(logging.ERROR) import six from six import PY3, b class WatcherTestCases: """Testcases for filesystems providing change watcher support. This class should be used as a mixin to the unittest.TestCase class for filesystems that provide change watcher support. """ def setupWatchers(self): self._captured_events = [] self.watchfs.add_watcher(self._captured_events.append) def clearCapturedEvents(self): del self._captured_events[:] def waitForEvents(self): if isinstance(self.watchfs,PollingWatchableFS): self.watchfs._poll_cond.acquire() self.watchfs._poll_cond.wait() self.watchfs._poll_cond.wait() self.watchfs._poll_cond.release() else: time.sleep(2) def assertEventOccurred(self,cls,path=None,event_list=None,**attrs): if not self.checkEventOccurred(cls,path,event_list,**attrs): args = (cls.__name__,path,attrs) assert False, "Event did not occur: %s(%s,%s)" % args def checkEventOccurred(self,cls,path=None,event_list=None,**attrs): if event_list is None: event_list = self._captured_events self.waitForEvents() for event in event_list: if isinstance(event,cls): if path is None or event.path == path: for (k,v) in attrs.iteritems(): if getattr(event,k) != v: break else: # all attrs match - found it! return True return False def test_watch_makedir(self): self.setupWatchers() self.fs.makedir("test1") self.assertEventOccurred(CREATED,"/test1") def test_watch_makedir_with_two_watchers(self): self.setupWatchers() events2 = [] self.watchfs.add_watcher(events2.append) self.fs.makedir("test1") self.assertEventOccurred(CREATED,"/test1") self.assertEventOccurred(CREATED,"/test1",event_list=events2) def test_watch_readfile(self): self.setupWatchers() self.fs.setcontents("hello", b("hello world")) self.assertEventOccurred(CREATED,"/hello") self.clearCapturedEvents() old_atime = self.fs.getinfo("hello").get("accessed_time") self.assertEquals(self.fs.getcontents("hello"), b("hello world")) if not isinstance(self.watchfs,PollingWatchableFS): # Help it along by updting the atime. # TODO: why is this necessary? if self.fs.hassyspath("hello"): syspath = self.fs.getsyspath("hello") mtime = os.stat(syspath).st_mtime atime = int(time.time()) os.utime(self.fs.getsyspath("hello"),(atime,mtime)) self.assertEventOccurred(ACCESSED,"/hello") elif old_atime is not None: # Some filesystems don't update atime synchronously, or only # update it if it's too old, or don't update it at all! # Try to force the issue, wait for it to change, but eventually # give up and bail out. for i in xrange(10): if self.fs.getinfo("hello").get("accessed_time") != old_atime: if not self.checkEventOccurred(MODIFIED,"/hello"): self.assertEventOccurred(ACCESSED,"/hello") break time.sleep(0.2) if self.fs.hassyspath("hello"): syspath = self.fs.getsyspath("hello") mtime = os.stat(syspath).st_mtime atime = int(time.time()) os.utime(self.fs.getsyspath("hello"),(atime,mtime)) def test_watch_writefile(self): self.setupWatchers() self.fs.setcontents("hello", b("hello world")) self.assertEventOccurred(CREATED,"/hello") self.clearCapturedEvents() self.fs.setcontents("hello", b("hello again world")) self.assertEventOccurred(MODIFIED,"/hello") def test_watch_single_file(self): self.fs.setcontents("hello", b("hello world")) events = [] self.watchfs.add_watcher(events.append,"/hello",(MODIFIED,)) self.fs.setcontents("hello", b("hello again world")) self.fs.remove("hello") self.waitForEvents() for evt in events: assert isinstance(evt,MODIFIED) self.assertEquals(evt.path,"/hello") def test_watch_single_file_remove(self): self.fs.makedir("testing") self.fs.setcontents("testing/hello", b("hello world")) events = [] self.watchfs.add_watcher(events.append,"/testing/hello",(REMOVED,)) self.fs.setcontents("testing/hello", b("hello again world")) self.waitForEvents() self.fs.remove("testing/hello") self.waitForEvents() self.assertEquals(len(events),1) assert isinstance(events[0],REMOVED) self.assertEquals(events[0].path,"/testing/hello") def test_watch_iter_changes(self): changes = iter_changes(self.watchfs) self.fs.makedir("test1") self.fs.setcontents("test1/hello", b("hello world")) self.waitForEvents() self.fs.removedir("test1",force=True) self.waitForEvents() self.watchfs.close() # Locate the CREATED(test1) event event = changes.next(timeout=1) while not isinstance(event,CREATED) or event.path != "/test1": event = changes.next(timeout=1) # Locate the CREATED(test1/hello) event event = changes.next(timeout=1) while not isinstance(event,CREATED) or event.path != "/test1/hello": event = changes.next(timeout=1) # Locate the REMOVED(test1) event event = changes.next(timeout=1) while not isinstance(event,REMOVED) or event.path != "/test1": event = changes.next(timeout=1) # Locate the CLOSED event event = changes.next(timeout=1) while not isinstance(event,CLOSED): event = changes.next(timeout=1) # That should be the last event in the list self.assertRaises(StopIteration,getattr(changes, "next"),timeout=1) changes.close() from fs import tempfs, osfs class TestWatchers_TempFS(unittest.TestCase,FSTestCases,WatcherTestCases): def setUp(self): self.fs = tempfs.TempFS() watchfs = osfs.OSFS(self.fs.root_path) self.watchfs = ensure_watchable(watchfs,poll_interval=0.1) if watch_inotify is not None: self.assertEquals(watchfs,self.watchfs) if watch_win32 is not None: self.assertEquals(watchfs,self.watchfs) def tearDown(self): self.watchfs.close() self.fs.close() def check(self, p): return self.fs.exists(p) from fs import memoryfs class TestWatchers_MemoryFS(unittest.TestCase,FSTestCases,WatcherTestCases): def setUp(self): self.fs = self.watchfs = WatchableFS(memoryfs.MemoryFS()) def tearDown(self): self.watchfs.close() self.fs.close() def check(self, p): return self.fs.exists(p) class TestWatchers_MemoryFS_polling(TestWatchers_MemoryFS): def setUp(self): self.fs = memoryfs.MemoryFS() self.watchfs = ensure_watchable(self.fs,poll_interval=0.1) fs-0.5.4/fs/tests/__init__.py0000664000175000017500000013325212621462466015774 0ustar willwill00000000000000#!/usr/bin/env python """ fs.tests: testcases for the fs module """ from __future__ import with_statement # Send any output from the logging module to stdout, so it will # be captured by nose and reported appropriately import sys import logging logging.basicConfig(level=logging.ERROR, stream=sys.stdout) from fs.base import * from fs.path import * from fs.errors import * from fs.filelike import StringIO import datetime import unittest import os import os.path import pickle import random import copy import time try: import threading except ImportError: import dummy_threading as threading import six from six import PY3, b class FSTestCases(object): """Base suite of testcases for filesystem implementations. Any FS subclass should be capable of passing all of these tests. To apply the tests to your own FS implementation, simply use FSTestCase as a mixin for your own unittest.TestCase subclass and have the setUp method set self.fs to an instance of your FS implementation. NB. The Filesystem being tested must have a capacity of at least 3MB. This class is designed as a mixin so that it's not detected by test loading tools such as nose. """ def check(self, p): """Check that a file exists within self.fs""" return self.fs.exists(p) def test_invalid_chars(self): """Check paths validate ok""" # Will have to be overriden selectively for custom validepath methods self.assertEqual(self.fs.validatepath(''), None) self.assertEqual(self.fs.validatepath('.foo'), None) self.assertEqual(self.fs.validatepath('foo'), None) self.assertEqual(self.fs.validatepath('foo/bar'), None) self.assert_(self.fs.isvalidpath('foo/bar')) def test_tree(self): """Test tree print""" self.fs.makedir('foo') self.fs.createfile('foo/bar.txt') self.fs.tree() def test_meta(self): """Checks getmeta / hasmeta are functioning""" # getmeta / hasmeta are hard to test, since there is no way to validate # the implementation's response meta_names = ["read_only", "network", "unicode_paths"] stupid_meta = 'thismetashouldnotexist!"r$$%^&&*()_+' self.assertRaises(NoMetaError, self.fs.getmeta, stupid_meta) self.assertFalse(self.fs.hasmeta(stupid_meta)) self.assertEquals(None, self.fs.getmeta(stupid_meta, None)) self.assertEquals(3.14, self.fs.getmeta(stupid_meta, 3.14)) for meta_name in meta_names: try: meta = self.fs.getmeta(meta_name) self.assertTrue(self.fs.hasmeta(meta_name)) except NoMetaError: self.assertFalse(self.fs.hasmeta(meta_name)) def test_root_dir(self): self.assertTrue(self.fs.isdir("")) self.assertTrue(self.fs.isdir("/")) # These may be false (e.g. empty dict) but mustn't raise errors self.fs.getinfo("") self.assertTrue(self.fs.getinfo("/") is not None) def test_getsyspath(self): try: syspath = self.fs.getsyspath("/") except NoSysPathError: pass else: self.assertTrue(isinstance(syspath, unicode)) syspath = self.fs.getsyspath("/", allow_none=True) if syspath is not None: self.assertTrue(isinstance(syspath, unicode)) def test_debug(self): str(self.fs) repr(self.fs) self.assert_(hasattr(self.fs, 'desc')) def test_open_on_directory(self): self.fs.makedir("testdir") try: f = self.fs.open("testdir") except ResourceInvalidError: pass except Exception: raise ecls = sys.exc_info()[0] assert False, "%s raised instead of ResourceInvalidError" % (ecls,) else: f.close() assert False, "ResourceInvalidError was not raised" def test_writefile(self): self.assertRaises(ResourceNotFoundError, self.fs.open, "test1.txt") f = self.fs.open("test1.txt", "wb") f.write(b("testing")) f.close() self.assertTrue(self.check("test1.txt")) f = self.fs.open("test1.txt", "rb") self.assertEquals(f.read(), b("testing")) f.close() f = self.fs.open("test1.txt", "wb") f.write(b("test file overwrite")) f.close() self.assertTrue(self.check("test1.txt")) f = self.fs.open("test1.txt", "rb") self.assertEquals(f.read(), b("test file overwrite")) f.close() def test_createfile(self): test = b('now with content') self.fs.createfile("test.txt") self.assert_(self.fs.exists("test.txt")) self.assertEqual(self.fs.getcontents("test.txt", "rb"), b('')) self.fs.setcontents("test.txt", test) self.fs.createfile("test.txt") self.assertEqual(self.fs.getcontents("test.txt", "rb"), test) self.fs.createfile("test.txt", wipe=True) self.assertEqual(self.fs.getcontents("test.txt", "rb"), b('')) def test_readline(self): text = b"Hello\nWorld\n" self.fs.setcontents('a.txt', text) with self.fs.open('a.txt', 'rb') as f: line = f.readline() self.assertEqual(line, b"Hello\n") def test_setcontents(self): # setcontents() should accept both a string... self.fs.setcontents("hello", b("world")) self.assertEquals(self.fs.getcontents("hello", "rb"), b("world")) # ...and a file-like object self.fs.setcontents("hello", StringIO(b("to you, good sir!"))) self.assertEquals(self.fs.getcontents( "hello", "rb"), b("to you, good sir!")) # setcontents() should accept both a string... self.fs.setcontents("hello", b("world"), chunk_size=2) self.assertEquals(self.fs.getcontents("hello", "rb"), b("world")) # ...and a file-like object self.fs.setcontents("hello", StringIO( b("to you, good sir!")), chunk_size=2) self.assertEquals(self.fs.getcontents( "hello", "rb"), b("to you, good sir!")) self.fs.setcontents("hello", b("")) self.assertEquals(self.fs.getcontents("hello", "rb"), b("")) def test_setcontents_async(self): # setcontents() should accept both a string... self.fs.setcontents_async("hello", b("world")).wait() self.assertEquals(self.fs.getcontents("hello", "rb"), b("world")) # ...and a file-like object self.fs.setcontents_async("hello", StringIO( b("to you, good sir!"))).wait() self.assertEquals(self.fs.getcontents("hello"), b("to you, good sir!")) self.fs.setcontents_async("hello", b("world"), chunk_size=2).wait() self.assertEquals(self.fs.getcontents("hello", "rb"), b("world")) # ...and a file-like object self.fs.setcontents_async("hello", StringIO( b("to you, good sir!")), chunk_size=2).wait() self.assertEquals(self.fs.getcontents( "hello", "rb"), b("to you, good sir!")) def test_isdir_isfile(self): self.assertFalse(self.fs.exists("dir1")) self.assertFalse(self.fs.isdir("dir1")) self.assertFalse(self.fs.isfile("a.txt")) self.fs.setcontents("a.txt", b('')) self.assertFalse(self.fs.isdir("dir1")) self.assertTrue(self.fs.exists("a.txt")) self.assertTrue(self.fs.isfile("a.txt")) self.assertFalse(self.fs.exists("a.txt/thatsnotadir")) self.fs.makedir("dir1") self.assertTrue(self.fs.isdir("dir1")) self.assertTrue(self.fs.exists("dir1")) self.assertTrue(self.fs.exists("a.txt")) self.fs.remove("a.txt") self.assertFalse(self.fs.exists("a.txt")) def test_listdir(self): def check_unicode(items): for item in items: self.assertTrue(isinstance(item, unicode)) self.fs.setcontents(u"a", b('')) self.fs.setcontents("b", b('')) self.fs.setcontents("foo", b('')) self.fs.setcontents("bar", b('')) # Test listing of the root directory d1 = self.fs.listdir() self.assertEqual(len(d1), 4) self.assertEqual(sorted(d1), [u"a", u"b", u"bar", u"foo"]) check_unicode(d1) d1 = self.fs.listdir("") self.assertEqual(len(d1), 4) self.assertEqual(sorted(d1), [u"a", u"b", u"bar", u"foo"]) check_unicode(d1) d1 = self.fs.listdir("/") self.assertEqual(len(d1), 4) check_unicode(d1) # Test listing absolute paths d2 = self.fs.listdir(absolute=True) self.assertEqual(len(d2), 4) self.assertEqual(sorted(d2), [u"/a", u"/b", u"/bar", u"/foo"]) check_unicode(d2) # Create some deeper subdirectories, to make sure their # contents are not inadvertantly included self.fs.makedir("p/1/2/3", recursive=True) self.fs.setcontents("p/1/2/3/a", b('')) self.fs.setcontents("p/1/2/3/b", b('')) self.fs.setcontents("p/1/2/3/foo", b('')) self.fs.setcontents("p/1/2/3/bar", b('')) self.fs.makedir("q") # Test listing just files, just dirs, and wildcards dirs_only = self.fs.listdir(dirs_only=True) files_only = self.fs.listdir(files_only=True) contains_a = self.fs.listdir(wildcard="*a*") self.assertEqual(sorted(dirs_only), [u"p", u"q"]) self.assertEqual(sorted(files_only), [u"a", u"b", u"bar", u"foo"]) self.assertEqual(sorted(contains_a), [u"a", u"bar"]) check_unicode(dirs_only) check_unicode(files_only) check_unicode(contains_a) # Test listing a subdirectory d3 = self.fs.listdir("p/1/2/3") self.assertEqual(len(d3), 4) self.assertEqual(sorted(d3), [u"a", u"b", u"bar", u"foo"]) check_unicode(d3) # Test listing a subdirectory with absoliute and full paths d4 = self.fs.listdir("p/1/2/3", absolute=True) self.assertEqual(len(d4), 4) self.assertEqual(sorted(d4), [u"/p/1/2/3/a", u"/p/1/2/3/b", u"/p/1/2/3/bar", u"/p/1/2/3/foo"]) check_unicode(d4) d4 = self.fs.listdir("p/1/2/3", full=True) self.assertEqual(len(d4), 4) self.assertEqual(sorted(d4), [u"p/1/2/3/a", u"p/1/2/3/b", u"p/1/2/3/bar", u"p/1/2/3/foo"]) check_unicode(d4) # Test that appropriate errors are raised self.assertRaises(ResourceNotFoundError, self.fs.listdir, "zebra") self.assertRaises(ResourceInvalidError, self.fs.listdir, "foo") def test_listdirinfo(self): def check_unicode(items): for (nm, info) in items: self.assertTrue(isinstance(nm, unicode)) def check_equal(items, target): names = [nm for (nm, info) in items] self.assertEqual(sorted(names), sorted(target)) self.fs.setcontents(u"a", b('')) self.fs.setcontents("b", b('')) self.fs.setcontents("foo", b('')) self.fs.setcontents("bar", b('')) # Test listing of the root directory d1 = self.fs.listdirinfo() self.assertEqual(len(d1), 4) check_equal(d1, [u"a", u"b", u"bar", u"foo"]) check_unicode(d1) d1 = self.fs.listdirinfo("") self.assertEqual(len(d1), 4) check_equal(d1, [u"a", u"b", u"bar", u"foo"]) check_unicode(d1) d1 = self.fs.listdirinfo("/") self.assertEqual(len(d1), 4) check_equal(d1, [u"a", u"b", u"bar", u"foo"]) check_unicode(d1) # Test listing absolute paths d2 = self.fs.listdirinfo(absolute=True) self.assertEqual(len(d2), 4) check_equal(d2, [u"/a", u"/b", u"/bar", u"/foo"]) check_unicode(d2) # Create some deeper subdirectories, to make sure their # contents are not inadvertantly included self.fs.makedir("p/1/2/3", recursive=True) self.fs.setcontents("p/1/2/3/a", b('')) self.fs.setcontents("p/1/2/3/b", b('')) self.fs.setcontents("p/1/2/3/foo", b('')) self.fs.setcontents("p/1/2/3/bar", b('')) self.fs.makedir("q") # Test listing just files, just dirs, and wildcards dirs_only = self.fs.listdirinfo(dirs_only=True) files_only = self.fs.listdirinfo(files_only=True) contains_a = self.fs.listdirinfo(wildcard="*a*") check_equal(dirs_only, [u"p", u"q"]) check_equal(files_only, [u"a", u"b", u"bar", u"foo"]) check_equal(contains_a, [u"a", u"bar"]) check_unicode(dirs_only) check_unicode(files_only) check_unicode(contains_a) # Test listing a subdirectory d3 = self.fs.listdirinfo("p/1/2/3") self.assertEqual(len(d3), 4) check_equal(d3, [u"a", u"b", u"bar", u"foo"]) check_unicode(d3) # Test listing a subdirectory with absoliute and full paths d4 = self.fs.listdirinfo("p/1/2/3", absolute=True) self.assertEqual(len(d4), 4) check_equal(d4, [u"/p/1/2/3/a", u"/p/1/2/3/b", u"/p/1/2/3/bar", u"/p/1/2/3/foo"]) check_unicode(d4) d4 = self.fs.listdirinfo("p/1/2/3", full=True) self.assertEqual(len(d4), 4) check_equal(d4, [u"p/1/2/3/a", u"p/1/2/3/b", u"p/1/2/3/bar", u"p/1/2/3/foo"]) check_unicode(d4) # Test that appropriate errors are raised self.assertRaises(ResourceNotFoundError, self.fs.listdirinfo, "zebra") self.assertRaises(ResourceInvalidError, self.fs.listdirinfo, "foo") def test_walk(self): self.fs.setcontents('a.txt', b('hello')) self.fs.setcontents('b.txt', b('world')) self.fs.makeopendir('foo').setcontents('c', b('123')) sorted_walk = sorted([(d, sorted(fs)) for (d, fs) in self.fs.walk()]) self.assertEquals(sorted_walk, [("/", ["a.txt", "b.txt"]), ("/foo", ["c"])]) # When searching breadth-first, shallow entries come first found_a = False for _, files in self.fs.walk(search="breadth"): if "a.txt" in files: found_a = True if "c" in files: break assert found_a, "breadth search order was wrong" # When searching depth-first, deep entries come first found_c = False for _, files in self.fs.walk(search="depth"): if "c" in files: found_c = True if "a.txt" in files: break assert found_c, "depth search order was wrong: " + \ str(list(self.fs.walk(search="depth"))) def test_walk_wildcard(self): self.fs.setcontents('a.txt', b('hello')) self.fs.setcontents('b.txt', b('world')) self.fs.makeopendir('foo').setcontents('c', b('123')) self.fs.makeopendir('.svn').setcontents('ignored', b('')) for dir_path, paths in self.fs.walk(wildcard='*.txt'): for path in paths: self.assert_(path.endswith('.txt')) for dir_path, paths in self.fs.walk(wildcard=lambda fn: fn.endswith('.txt')): for path in paths: self.assert_(path.endswith('.txt')) def test_walk_dir_wildcard(self): self.fs.setcontents('a.txt', b('hello')) self.fs.setcontents('b.txt', b('world')) self.fs.makeopendir('foo').setcontents('c', b('123')) self.fs.makeopendir('.svn').setcontents('ignored', b('')) for dir_path, paths in self.fs.walk(dir_wildcard=lambda fn: not fn.endswith('.svn')): for path in paths: self.assert_('.svn' not in path) def test_walkfiles(self): self.fs.makeopendir('bar').setcontents('a.txt', b('123')) self.fs.makeopendir('foo').setcontents('b', b('123')) self.assertEquals(sorted( self.fs.walkfiles()), ["/bar/a.txt", "/foo/b"]) self.assertEquals(sorted(self.fs.walkfiles( dir_wildcard="*foo*")), ["/foo/b"]) self.assertEquals(sorted(self.fs.walkfiles( wildcard="*.txt")), ["/bar/a.txt"]) def test_walkdirs(self): self.fs.makeopendir('bar').setcontents('a.txt', b('123')) self.fs.makeopendir('foo').makeopendir( "baz").setcontents('b', b('123')) self.assertEquals(sorted(self.fs.walkdirs()), [ "/", "/bar", "/foo", "/foo/baz"]) self.assertEquals(sorted(self.fs.walkdirs( wildcard="*foo*")), ["/", "/foo", "/foo/baz"]) def test_unicode(self): alpha = u"\N{GREEK SMALL LETTER ALPHA}" beta = u"\N{GREEK SMALL LETTER BETA}" self.fs.makedir(alpha) self.fs.setcontents(alpha + "/a", b('')) self.fs.setcontents(alpha + "/" + beta, b('')) self.assertTrue(self.check(alpha)) self.assertEquals(sorted(self.fs.listdir(alpha)), ["a", beta]) def test_makedir(self): check = self.check self.fs.makedir("a") self.assertTrue(check("a")) self.assertRaises( ParentDirectoryMissingError, self.fs.makedir, "a/b/c") self.fs.makedir("a/b/c", recursive=True) self.assert_(check("a/b/c")) self.fs.makedir("foo/bar/baz", recursive=True) self.assert_(check("foo/bar/baz")) self.fs.makedir("a/b/child") self.assert_(check("a/b/child")) self.assertRaises(DestinationExistsError, self.fs.makedir, "/a/b") self.fs.makedir("/a/b", allow_recreate=True) self.fs.setcontents("/a/file", b('')) self.assertRaises(ResourceInvalidError, self.fs.makedir, "a/file") def test_remove(self): self.fs.setcontents("a.txt", b('')) self.assertTrue(self.check("a.txt")) self.fs.remove("a.txt") self.assertFalse(self.check("a.txt")) self.assertRaises(ResourceNotFoundError, self.fs.remove, "a.txt") self.fs.makedir("dir1") self.assertRaises(ResourceInvalidError, self.fs.remove, "dir1") self.fs.setcontents("/dir1/a.txt", b('')) self.assertTrue(self.check("dir1/a.txt")) self.fs.remove("dir1/a.txt") self.assertFalse(self.check("/dir1/a.txt")) def test_removedir(self): check = self.check self.fs.makedir("a") self.assert_(check("a")) self.fs.removedir("a") self.assertRaises(ResourceNotFoundError, self.fs.removedir, "a") self.assert_(not check("a")) self.fs.makedir("a/b/c/d", recursive=True) self.assertRaises(DirectoryNotEmptyError, self.fs.removedir, "a/b") self.fs.removedir("a/b/c/d") self.assert_(not check("a/b/c/d")) self.fs.removedir("a/b/c") self.assert_(not check("a/b/c")) self.fs.removedir("a/b") self.assert_(not check("a/b")) # Test recursive removal of empty parent dirs self.fs.makedir("foo/bar/baz", recursive=True) self.fs.removedir("foo/bar/baz", recursive=True) self.assert_(not check("foo/bar/baz")) self.assert_(not check("foo/bar")) self.assert_(not check("foo")) self.fs.makedir("foo/bar/baz", recursive=True) self.fs.setcontents("foo/file.txt", b("please don't delete me")) self.fs.removedir("foo/bar/baz", recursive=True) self.assert_(not check("foo/bar/baz")) self.assert_(not check("foo/bar")) self.assert_(check("foo/file.txt")) # Ensure that force=True works as expected self.fs.makedir("frollic/waggle", recursive=True) self.fs.setcontents("frollic/waddle.txt", b("waddlewaddlewaddle")) self.assertRaises(DirectoryNotEmptyError, self.fs.removedir, "frollic") self.assertRaises( ResourceInvalidError, self.fs.removedir, "frollic/waddle.txt") self.fs.removedir("frollic", force=True) self.assert_(not check("frollic")) # Test removing unicode dirs kappa = u"\N{GREEK CAPITAL LETTER KAPPA}" self.fs.makedir(kappa) self.assert_(self.fs.isdir(kappa)) self.fs.removedir(kappa) self.assertRaises(ResourceNotFoundError, self.fs.removedir, kappa) self.assert_(not self.fs.isdir(kappa)) self.fs.makedir(pathjoin("test", kappa), recursive=True) self.assert_(check(pathjoin("test", kappa))) self.fs.removedir("test", force=True) self.assert_(not check("test")) def test_rename(self): check = self.check # test renaming a file in the same directory self.fs.setcontents("foo.txt", b("Hello, World!")) self.assert_(check("foo.txt")) self.fs.rename("foo.txt", "bar.txt") self.assert_(check("bar.txt")) self.assert_(not check("foo.txt")) # test renaming a directory in the same directory self.fs.makedir("dir_a") self.fs.setcontents("dir_a/test.txt", b("testerific")) self.assert_(check("dir_a")) self.fs.rename("dir_a", "dir_b") self.assert_(check("dir_b")) self.assert_(check("dir_b/test.txt")) self.assert_(not check("dir_a/test.txt")) self.assert_(not check("dir_a")) # test renaming a file into a different directory self.fs.makedir("dir_a") self.fs.rename("dir_b/test.txt", "dir_a/test.txt") self.assert_(not check("dir_b/test.txt")) self.assert_(check("dir_a/test.txt")) # test renaming a file into a non-existent directory self.assertRaises(ParentDirectoryMissingError, self.fs.rename, "dir_a/test.txt", "nonexistent/test.txt") def test_info(self): test_str = b("Hello, World!") self.fs.setcontents("info.txt", test_str) info = self.fs.getinfo("info.txt") self.assertEqual(info['size'], len(test_str)) self.fs.desc("info.txt") self.assertRaises(ResourceNotFoundError, self.fs.getinfo, "notafile") self.assertRaises( ResourceNotFoundError, self.fs.getinfo, "info.txt/inval") def test_infokeys(self): test_str = b("Hello, World!") self.fs.setcontents("info.txt", test_str) info = self.fs.getinfo("info.txt") for k, v in info.iteritems(): self.assertEqual(self.fs.getinfokeys('info.txt', k), {k: v}) test_info = {} if 'modified_time' in info: test_info['modified_time'] = info['modified_time'] if 'size' in info: test_info['size'] = info['size'] self.assertEqual(self.fs.getinfokeys('info.txt', 'size', 'modified_time'), test_info) self.assertEqual(self.fs.getinfokeys('info.txt', 'thiscantpossiblyexistininfo'), {}) def test_getsize(self): test_str = b("*") * 23 self.fs.setcontents("info.txt", test_str) size = self.fs.getsize("info.txt") self.assertEqual(size, len(test_str)) def test_movefile(self): check = self.check contents = b( "If the implementation is hard to explain, it's a bad idea.") def makefile(path): self.fs.setcontents(path, contents) def checkcontents(path): check_contents = self.fs.getcontents(path, "rb") self.assertEqual(check_contents, contents) return contents == check_contents self.fs.makedir("foo/bar", recursive=True) makefile("foo/bar/a.txt") self.assert_(check("foo/bar/a.txt")) self.assert_(checkcontents("foo/bar/a.txt")) self.fs.move("foo/bar/a.txt", "foo/b.txt") self.assert_(not check("foo/bar/a.txt")) self.assert_(check("foo/b.txt")) self.assert_(checkcontents("foo/b.txt")) self.fs.move("foo/b.txt", "c.txt") self.assert_(not check("foo/b.txt")) self.assert_(check("/c.txt")) self.assert_(checkcontents("/c.txt")) makefile("foo/bar/a.txt") self.assertRaises( DestinationExistsError, self.fs.move, "foo/bar/a.txt", "/c.txt") self.assert_(check("foo/bar/a.txt")) self.assert_(check("/c.txt")) self.fs.move("foo/bar/a.txt", "/c.txt", overwrite=True) self.assert_(not check("foo/bar/a.txt")) self.assert_(check("/c.txt")) def test_movedir(self): check = self.check contents = b( "If the implementation is hard to explain, it's a bad idea.") def makefile(path): self.fs.setcontents(path, contents) self.assertRaises(ResourceNotFoundError, self.fs.movedir, "a", "b") self.fs.makedir("a") self.fs.makedir("b") makefile("a/1.txt") makefile("a/2.txt") makefile("a/3.txt") self.fs.makedir("a/foo/bar", recursive=True) makefile("a/foo/bar/baz.txt") self.fs.movedir("a", "copy of a") self.assert_(self.fs.isdir("copy of a")) self.assert_(check("copy of a/1.txt")) self.assert_(check("copy of a/2.txt")) self.assert_(check("copy of a/3.txt")) self.assert_(check("copy of a/foo/bar/baz.txt")) self.assert_(not check("a/1.txt")) self.assert_(not check("a/2.txt")) self.assert_(not check("a/3.txt")) self.assert_(not check("a/foo/bar/baz.txt")) self.assert_(not check("a/foo/bar")) self.assert_(not check("a/foo")) self.assert_(not check("a")) self.fs.makedir("a") self.assertRaises( DestinationExistsError, self.fs.movedir, "copy of a", "a") self.fs.movedir("copy of a", "a", overwrite=True) self.assert_(not check("copy of a")) self.assert_(check("a/1.txt")) self.assert_(check("a/2.txt")) self.assert_(check("a/3.txt")) self.assert_(check("a/foo/bar/baz.txt")) def test_cant_copy_from_os(self): sys_executable = os.path.abspath(os.path.realpath(sys.executable)) self.assertRaises(FSError, self.fs.copy, sys_executable, "py.exe") def test_copyfile(self): check = self.check contents = b( "If the implementation is hard to explain, it's a bad idea.") def makefile(path, contents=contents): self.fs.setcontents(path, contents) def checkcontents(path, contents=contents): check_contents = self.fs.getcontents(path, "rb") self.assertEqual(check_contents, contents) return contents == check_contents self.fs.makedir("foo/bar", recursive=True) makefile("foo/bar/a.txt") self.assert_(check("foo/bar/a.txt")) self.assert_(checkcontents("foo/bar/a.txt")) # import rpdb2; rpdb2.start_embedded_debugger('password'); self.fs.copy("foo/bar/a.txt", "foo/b.txt") self.assert_(check("foo/bar/a.txt")) self.assert_(check("foo/b.txt")) self.assert_(checkcontents("foo/bar/a.txt")) self.assert_(checkcontents("foo/b.txt")) self.fs.copy("foo/b.txt", "c.txt") self.assert_(check("foo/b.txt")) self.assert_(check("/c.txt")) self.assert_(checkcontents("/c.txt")) makefile("foo/bar/a.txt", b("different contents")) self.assert_(checkcontents("foo/bar/a.txt", b("different contents"))) self.assertRaises( DestinationExistsError, self.fs.copy, "foo/bar/a.txt", "/c.txt") self.assert_(checkcontents("/c.txt")) self.fs.copy("foo/bar/a.txt", "/c.txt", overwrite=True) self.assert_(checkcontents("foo/bar/a.txt", b("different contents"))) self.assert_(checkcontents("/c.txt", b("different contents"))) def test_copydir(self): check = self.check contents = b( "If the implementation is hard to explain, it's a bad idea.") def makefile(path): self.fs.setcontents(path, contents) def checkcontents(path): check_contents = self.fs.getcontents(path) self.assertEqual(check_contents, contents) return contents == check_contents self.fs.makedir("a") self.fs.makedir("b") makefile("a/1.txt") makefile("a/2.txt") makefile("a/3.txt") self.fs.makedir("a/foo/bar", recursive=True) makefile("a/foo/bar/baz.txt") self.fs.copydir("a", "copy of a") self.assert_(check("copy of a/1.txt")) self.assert_(check("copy of a/2.txt")) self.assert_(check("copy of a/3.txt")) self.assert_(check("copy of a/foo/bar/baz.txt")) checkcontents("copy of a/1.txt") self.assert_(check("a/1.txt")) self.assert_(check("a/2.txt")) self.assert_(check("a/3.txt")) self.assert_(check("a/foo/bar/baz.txt")) checkcontents("a/1.txt") self.assertRaises(DestinationExistsError, self.fs.copydir, "a", "b") self.fs.copydir("a", "b", overwrite=True) self.assert_(check("b/1.txt")) self.assert_(check("b/2.txt")) self.assert_(check("b/3.txt")) self.assert_(check("b/foo/bar/baz.txt")) checkcontents("b/1.txt") def test_copydir_with_dotfile(self): check = self.check contents = b( "If the implementation is hard to explain, it's a bad idea.") def makefile(path): self.fs.setcontents(path, contents) self.fs.makedir("a") makefile("a/1.txt") makefile("a/2.txt") makefile("a/.hidden.txt") self.fs.copydir("a", "copy of a") self.assert_(check("copy of a/1.txt")) self.assert_(check("copy of a/2.txt")) self.assert_(check("copy of a/.hidden.txt")) self.assert_(check("a/1.txt")) self.assert_(check("a/2.txt")) self.assert_(check("a/.hidden.txt")) def test_readwriteappendseek(self): def checkcontents(path, check_contents): read_contents = self.fs.getcontents(path, "rb") self.assertEqual(read_contents, check_contents) return read_contents == check_contents test_strings = [b("Beautiful is better than ugly."), b("Explicit is better than implicit."), b("Simple is better than complex.")] all_strings = b("").join(test_strings) self.assertRaises(ResourceNotFoundError, self.fs.open, "a.txt", "r") self.assert_(not self.fs.exists("a.txt")) f1 = self.fs.open("a.txt", "wb") pos = 0 for s in test_strings: f1.write(s) pos += len(s) self.assertEqual(pos, f1.tell()) f1.close() self.assert_(self.fs.exists("a.txt")) self.assert_(checkcontents("a.txt", all_strings)) f2 = self.fs.open("b.txt", "wb") f2.write(test_strings[0]) f2.close() self.assert_(checkcontents("b.txt", test_strings[0])) f3 = self.fs.open("b.txt", "ab") # On win32, tell() gives zero until you actually write to the file # self.assertEquals(f3.tell(),len(test_strings[0])) f3.write(test_strings[1]) self.assertEquals(f3.tell(), len(test_strings[0])+len(test_strings[1])) f3.write(test_strings[2]) self.assertEquals(f3.tell(), len(all_strings)) f3.close() self.assert_(checkcontents("b.txt", all_strings)) f4 = self.fs.open("b.txt", "wb") f4.write(test_strings[2]) f4.close() self.assert_(checkcontents("b.txt", test_strings[2])) f5 = self.fs.open("c.txt", "wb") for s in test_strings: f5.write(s+b("\n")) f5.close() f6 = self.fs.open("c.txt", "rb") for s, t in zip(f6, test_strings): self.assertEqual(s, t+b("\n")) f6.close() f7 = self.fs.open("c.txt", "rb") f7.seek(13) word = f7.read(6) self.assertEqual(word, b("better")) f7.seek(1, os.SEEK_CUR) word = f7.read(4) self.assertEqual(word, b("than")) f7.seek(-9, os.SEEK_END) word = f7.read(7) self.assertEqual(word, b("complex")) f7.close() self.assertEqual(self.fs.getcontents("a.txt", "rb"), all_strings) def test_truncate(self): def checkcontents(path, check_contents): read_contents = self.fs.getcontents(path, "rb") self.assertEqual(read_contents, check_contents) return read_contents == check_contents self.fs.setcontents("hello", b("world")) checkcontents("hello", b("world")) self.fs.setcontents("hello", b("hi")) checkcontents("hello", b("hi")) self.fs.setcontents("hello", b("1234567890")) checkcontents("hello", b("1234567890")) with self.fs.open("hello", "rb+") as f: f.truncate(7) checkcontents("hello", b("1234567")) with self.fs.open("hello", "rb+") as f: f.seek(5) f.truncate() checkcontents("hello", b("12345")) def test_truncate_to_larger_size(self): with self.fs.open("hello", "wb") as f: f.truncate(30) self.assertEquals(self.fs.getsize("hello"), 30) # Some file systems (FTPFS) don't support both reading and writing if self.fs.getmeta('file.read_and_write', True): with self.fs.open("hello", "rb+") as f: f.seek(25) f.write(b("123456")) with self.fs.open("hello", "rb") as f: f.seek(25) self.assertEquals(f.read(), b("123456")) def test_write_past_end_of_file(self): if self.fs.getmeta('file.read_and_write', True): with self.fs.open("write_at_end", "wb") as f: f.seek(25) f.write(b("EOF")) with self.fs.open("write_at_end", "rb") as f: self.assertEquals(f.read(), b("\x00")*25 + b("EOF")) def test_with_statement(self): # This is a little tricky since 'with' is actually new syntax. # We use eval() to make this method safe for old python versions. import sys if sys.version_info[0] >= 2 and sys.version_info[1] >= 5: # A successful 'with' statement contents = "testing the with statement" code = "from __future__ import with_statement\n" code += "with self.fs.open('f.txt','wb-') as testfile:\n" code += " testfile.write(contents)\n" code += "self.assertEquals(self.fs.getcontents('f.txt', 'rb'),contents)" code = compile(code, "", 'exec') eval(code) # A 'with' statement raising an error contents = "testing the with statement" code = "from __future__ import with_statement\n" code += "with self.fs.open('f.txt','wb-') as testfile:\n" code += " testfile.write(contents)\n" code += " raise ValueError\n" code = compile(code, "", 'exec') self.assertRaises(ValueError, eval, code, globals(), locals()) self.assertEquals(self.fs.getcontents('f.txt', 'rb'), contents) def test_pickling(self): if self.fs.getmeta('pickle_contents', True): self.fs.setcontents("test1", b("hello world")) fs2 = pickle.loads(pickle.dumps(self.fs)) self.assert_(fs2.isfile("test1")) fs3 = pickle.loads(pickle.dumps(self.fs, -1)) self.assert_(fs3.isfile("test1")) else: # Just make sure it doesn't throw an exception fs2 = pickle.loads(pickle.dumps(self.fs)) def test_big_file(self): """Test handling of a big file (1MB)""" chunk_size = 1024 * 256 num_chunks = 4 def chunk_stream(): """Generate predictable-but-randomy binary content.""" r = random.Random(0) randint = r.randint int2byte = six.int2byte for _i in xrange(num_chunks): c = b("").join(int2byte(randint( 0, 255)) for _j in xrange(chunk_size//8)) yield c * 8 f = self.fs.open("bigfile", "wb") try: for chunk in chunk_stream(): f.write(chunk) finally: f.close() chunks = chunk_stream() f = self.fs.open("bigfile", "rb") try: try: while True: if chunks.next() != f.read(chunk_size): assert False, "bigfile was corrupted" except StopIteration: if f.read() != b(""): assert False, "bigfile was corrupted" finally: f.close() def test_settimes(self): def cmp_datetimes(d1, d2): """Test datetime objects are the same to within the timestamp accuracy""" dts1 = time.mktime(d1.timetuple()) dts2 = time.mktime(d2.timetuple()) return int(dts1) == int(dts2) d1 = datetime.datetime(2010, 6, 20, 11, 0, 9, 987699) d2 = datetime.datetime(2010, 7, 5, 11, 0, 9, 500000) self.fs.setcontents('/dates.txt', b('check dates')) # If the implementation supports settimes, check that the times # can be set and then retrieved try: self.fs.settimes('/dates.txt', d1, d2) except UnsupportedError: pass else: info = self.fs.getinfo('/dates.txt') self.assertTrue(cmp_datetimes(d1, info['accessed_time'])) self.assertTrue(cmp_datetimes(d2, info['modified_time'])) def test_removeroot(self): self.assertRaises(RemoveRootError, self.fs.removedir, "/") def test_zero_read(self): """Test read(0) returns empty string""" self.fs.setcontents('foo.txt', b('Hello, World')) with self.fs.open('foo.txt', 'rb') as f: self.assert_(len(f.read(0)) == 0) with self.fs.open('foo.txt', 'rt') as f: self.assert_(len(f.read(0)) == 0) # May be disabled - see end of file class ThreadingTestCases(object): """Testcases for thread-safety of FS implementations.""" # These are either too slow to be worth repeating, # or cannot possibly break cross-thread. _dont_retest = ("test_pickling", "test_multiple_overwrite",) __lock = threading.RLock() def _yield(self): # time.sleep(0.001) # Yields without a delay time.sleep(0) def _lock(self): self.__lock.acquire() def _unlock(self): self.__lock.release() def _makeThread(self, func, errors): def runThread(): try: func() except Exception: errors.append(sys.exc_info()) thread = threading.Thread(target=runThread) thread.daemon = True return thread def _runThreads(self, *funcs): check_interval = sys.getcheckinterval() sys.setcheckinterval(1) try: errors = [] threads = [self._makeThread(f, errors) for f in funcs] for t in threads: t.start() for t in threads: t.join() for (c, e, t) in errors: raise e, None, t finally: sys.setcheckinterval(check_interval) def test_setcontents_threaded(self): def setcontents(name, contents): f = self.fs.open(name, "wb") self._yield() try: f.write(contents) self._yield() finally: f.close() def thread1(): c = b("thread1 was 'ere") setcontents("thread1.txt", c) self.assertEquals(self.fs.getcontents("thread1.txt", 'rb'), c) def thread2(): c = b("thread2 was 'ere") setcontents("thread2.txt", c) self.assertEquals(self.fs.getcontents("thread2.txt", 'rb'), c) self._runThreads(thread1, thread2) def test_setcontents_threaded_samefile(self): def setcontents(name, contents): f = self.fs.open(name, "wb") self._yield() try: f.write(contents) self._yield() finally: f.close() def thread1(): c = b("thread1 was 'ere") setcontents("threads.txt", c) self._yield() self.assertEquals(self.fs.listdir("/"), ["threads.txt"]) def thread2(): c = b("thread2 was 'ere") setcontents("threads.txt", c) self._yield() self.assertEquals(self.fs.listdir("/"), ["threads.txt"]) def thread3(): c = b("thread3 was 'ere") setcontents("threads.txt", c) self._yield() self.assertEquals(self.fs.listdir("/"), ["threads.txt"]) try: self._runThreads(thread1, thread2, thread3) except ResourceLockedError: # that's ok, some implementations don't support concurrent writes pass def test_cases_in_separate_dirs(self): class TestCases_in_subdir(self.__class__, unittest.TestCase): """Run all testcases against a subdir of self.fs""" def __init__(this, subdir): super(TestCases_in_subdir, this).__init__("test_listdir") this.subdir = subdir for meth in dir(this): if not meth.startswith("test_"): continue if meth in self._dont_retest: continue if not hasattr(FSTestCases, meth): continue if self.fs.exists(subdir): self.fs.removedir(subdir, force=True) self.assertFalse(self.fs.isdir(subdir)) self.assertTrue(self.fs.isdir("/")) self.fs.makedir(subdir) self._yield() getattr(this, meth)() @property def fs(this): return self.fs.opendir(this.subdir) def check(this, p): return self.check(pathjoin(this.subdir, relpath(p))) def thread1(): TestCases_in_subdir("thread1") def thread2(): TestCases_in_subdir("thread2") def thread3(): TestCases_in_subdir("thread3") self._runThreads(thread1, thread2, thread3) def test_makedir_winner(self): errors = [] def makedir(): try: self.fs.makedir("testdir") except DestinationExistsError, e: errors.append(e) def makedir_noerror(): try: self.fs.makedir("testdir", allow_recreate=True) except DestinationExistsError, e: errors.append(e) def removedir(): try: self.fs.removedir("testdir") except (ResourceNotFoundError, ResourceLockedError), e: errors.append(e) # One thread should succeed, one should error self._runThreads(makedir, makedir) self.assertEquals(len(errors), 1) self.fs.removedir("testdir") # One thread should succeed, two should error errors = [] self._runThreads(makedir, makedir, makedir) if len(errors) != 2: raise AssertionError(errors) self.fs.removedir("testdir") # All threads should succeed errors = [] self._runThreads(makedir_noerror, makedir_noerror, makedir_noerror) self.assertEquals(len(errors), 0) self.assertTrue(self.fs.isdir("testdir")) self.fs.removedir("testdir") # makedir() can beat removedir() and vice-versa errors = [] self._runThreads(makedir, removedir) if self.fs.isdir("testdir"): self.assertEquals(len(errors), 1) self.assertFalse(isinstance(errors[0], DestinationExistsError)) self.fs.removedir("testdir") else: self.assertEquals(len(errors), 0) def test_concurrent_copydir(self): self.fs.makedir("a") self.fs.makedir("a/b") self.fs.setcontents("a/hello.txt", b("hello world")) self.fs.setcontents("a/guido.txt", b("is a space alien")) self.fs.setcontents("a/b/parrot.txt", b("pining for the fiords")) def copydir(): self._yield() self.fs.copydir("a", "copy of a") def copydir_overwrite(): self._yield() self.fs.copydir("a", "copy of a", overwrite=True) # This should error out since we're not overwriting self.assertRaises( DestinationExistsError, self._runThreads, copydir, copydir) self.assert_(self.fs.isdir('a')) self.assert_(self.fs.isdir('a')) copydir_overwrite() self.assert_(self.fs.isdir('a')) # This should run to completion and give a valid state, unless # files get locked when written to. try: self._runThreads(copydir_overwrite, copydir_overwrite) except ResourceLockedError: pass self.assertTrue(self.fs.isdir("copy of a")) self.assertTrue(self.fs.isdir("copy of a/b")) self.assertEqual(self.fs.getcontents( "copy of a/b/parrot.txt", 'rb'), b("pining for the fiords")) self.assertEqual(self.fs.getcontents( "copy of a/hello.txt", 'rb'), b("hello world")) self.assertEqual(self.fs.getcontents( "copy of a/guido.txt", 'rb'), b("is a space alien")) def test_multiple_overwrite(self): contents = [b("contents one"), b( "contents the second"), b("number three")] def thread1(): for i in xrange(30): for c in contents: self.fs.setcontents("thread1.txt", c) self.assertEquals(self.fs.getsize("thread1.txt"), len(c)) self.assertEquals(self.fs.getcontents( "thread1.txt", 'rb'), c) def thread2(): for i in xrange(30): for c in contents: self.fs.setcontents("thread2.txt", c) self.assertEquals(self.fs.getsize("thread2.txt"), len(c)) self.assertEquals(self.fs.getcontents( "thread2.txt", 'rb'), c) self._runThreads(thread1, thread2) # Uncomment to temporarily disable threading tests # class ThreadingTestCases(object): # _dont_retest = () fs-0.5.4/fs/tests/test_s3fs.py0000664000175000017500000000264512512525115016141 0ustar willwill00000000000000""" fs.tests.test_s3fs: testcases for the S3FS module These tests are set up to be skipped by default, since they're very slow, require a valid AWS account, and cost money. You'll have to set the '__test__' attribute the True on te TestS3FS class to get them running. """ import unittest from fs.tests import FSTestCases, ThreadingTestCases from fs.path import * from six import PY3 try: from fs import s3fs except ImportError: raise unittest.SkipTest("s3fs wasn't importable") class TestS3FS(unittest.TestCase,FSTestCases,ThreadingTestCases): # Disable the tests by default __test__ = False bucket = "test-s3fs.rfk.id.au" def setUp(self): self.fs = s3fs.S3FS(self.bucket) for k in self.fs._s3bukt.list(): self.fs._s3bukt.delete_key(k) def tearDown(self): self.fs.close() def test_concurrent_copydir(self): # makedir() on S3FS is currently not atomic pass def test_makedir_winner(self): # makedir() on S3FS is currently not atomic pass def test_multiple_overwrite(self): # S3's eventual-consistency seems to be breaking this test pass class TestS3FS_prefix(TestS3FS): def setUp(self): self.fs = s3fs.S3FS(self.bucket,"/unittest/files") for k in self.fs._s3bukt.list(): self.fs._s3bukt.delete_key(k) def tearDown(self): self.fs.close() fs-0.5.4/fs/rpcfs.py0000664000175000017500000002607512512525115014202 0ustar willwill00000000000000""" fs.rpcfs ======== This module provides the class 'RPCFS' to access a remote FS object over XML-RPC. You probably want to use this in conjunction with the 'RPCFSServer' class from the :mod:`fs.expose.xmlrpc` module. """ import xmlrpclib import socket import base64 from fs.base import * from fs.errors import * from fs.path import * from fs import iotools from fs.filelike import StringIO import six from six import PY3, b def re_raise_faults(func): """Decorator to re-raise XML-RPC faults as proper exceptions.""" def wrapper(*args, **kwds): try: return func(*args, **kwds) except (xmlrpclib.Fault), f: #raise # Make sure it's in a form we can handle print f.faultString bits = f.faultString.split(" ") if bits[0] not in [":") cls = bits[0] msg = ">:".join(bits[1:]) cls = cls.strip('\'') print "-" + cls cls = _object_by_name(cls) # Re-raise using the remainder of the fault code as message if cls: if issubclass(cls, FSError): raise cls('', msg=msg) else: raise cls(msg) raise f except socket.error, e: raise RemoteConnectionError(str(e), details=e) return wrapper def _object_by_name(name, root=None): """Look up an object by dotted-name notation.""" bits = name.split(".") if root is None: try: obj = globals()[bits[0]] except KeyError: try: obj = __builtins__[bits[0]] except KeyError: obj = __import__(bits[0], globals()) else: obj = getattr(root, bits[0]) if len(bits) > 1: return _object_by_name(".".join(bits[1:]), obj) else: return obj class ReRaiseFaults: """XML-RPC proxy wrapper that re-raises Faults as proper Exceptions.""" def __init__(self, obj): self._obj = obj def __getattr__(self, attr): val = getattr(self._obj, attr) if callable(val): val = re_raise_faults(val) self.__dict__[attr] = val return val class RPCFS(FS): """Access a filesystem exposed via XML-RPC. This class provides the client-side logic for accessing a remote FS object, and is dual to the RPCFSServer class defined in fs.expose.xmlrpc. Example:: fs = RPCFS("http://my.server.com/filesystem/location/") """ _meta = {'thread_safe' : True, 'virtual': False, 'network' : True, } def __init__(self, uri, transport=None): """Constructor for RPCFS objects. The only required argument is the URI of the server to connect to. This will be passed to the underlying XML-RPC server proxy object, along with the 'transport' argument if it is provided. :param uri: address of the server """ super(RPCFS, self).__init__(thread_synchronize=True) self.uri = uri self._transport = transport self.proxy = self._make_proxy() self.isdir('/') @synchronize def _make_proxy(self): kwds = dict(allow_none=True, use_datetime=True) if self._transport is not None: proxy = xmlrpclib.ServerProxy(self.uri, self._transport, **kwds) else: proxy = xmlrpclib.ServerProxy(self.uri, **kwds) return ReRaiseFaults(proxy) def __str__(self): return '' % (self.uri,) def __repr__(self): return '' % (self.uri,) @synchronize def __getstate__(self): state = super(RPCFS, self).__getstate__() try: del state['proxy'] except KeyError: pass return state def __setstate__(self, state): super(RPCFS, self).__setstate__(state) self.proxy = self._make_proxy() def encode_path(self, path): """Encode a filesystem path for sending over the wire. Unfortunately XMLRPC only supports ASCII strings, so this method must return something that can be represented in ASCII. The default is base64-encoded UTF8. """ return six.text_type(base64.b64encode(path.encode("utf8")), 'ascii') def decode_path(self, path): """Decode paths arriving over the wire.""" return six.text_type(base64.b64decode(path.encode('ascii')), 'utf8') @synchronize def getmeta(self, meta_name, default=NoDefaultMeta): if default is NoDefaultMeta: meta = self.proxy.getmeta(meta_name) else: meta = self.proxy.getmeta_default(meta_name, default) if isinstance(meta, basestring): # To allow transport of meta with invalid xml chars (like null) meta = self.encode_path(meta) return meta @synchronize def hasmeta(self, meta_name): return self.proxy.hasmeta(meta_name) @synchronize @iotools.filelike_to_stream def open(self, path, mode='r', buffering=-1, encoding=None, errors=None, newline=None, line_buffering=False, **kwargs): # TODO: chunked transport of large files epath = self.encode_path(path) if "w" in mode: self.proxy.set_contents(epath, xmlrpclib.Binary(b(""))) if "r" in mode or "a" in mode or "+" in mode: try: data = self.proxy.get_contents(epath, "rb").data except IOError: if "w" not in mode and "a" not in mode: raise ResourceNotFoundError(path) if not self.isdir(dirname(path)): raise ParentDirectoryMissingError(path) self.proxy.set_contents(path, xmlrpclib.Binary(b(""))) else: data = b("") f = StringIO(data) if "a" not in mode: f.seek(0, 0) else: f.seek(0, 2) oldflush = f.flush oldclose = f.close oldtruncate = f.truncate def newflush(): self._lock.acquire() try: oldflush() self.proxy.set_contents(epath, xmlrpclib.Binary(f.getvalue())) finally: self._lock.release() def newclose(): self._lock.acquire() try: f.flush() oldclose() finally: self._lock.release() def newtruncate(size=None): self._lock.acquire() try: oldtruncate(size) f.flush() finally: self._lock.release() f.flush = newflush f.close = newclose f.truncate = newtruncate return f @synchronize def exists(self, path): path = self.encode_path(path) return self.proxy.exists(path) @synchronize def isdir(self, path): path = self.encode_path(path) return self.proxy.isdir(path) @synchronize def isfile(self, path): path = self.encode_path(path) return self.proxy.isfile(path) @synchronize def listdir(self, path="./", wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False): enc_path = self.encode_path(path) if not callable(wildcard): entries = self.proxy.listdir(enc_path, wildcard, full, absolute, dirs_only, files_only) entries = [self.decode_path(e) for e in entries] else: entries = self.proxy.listdir(enc_path, None, False, False, dirs_only, files_only) entries = [self.decode_path(e) for e in entries] entries = [e for e in entries if wildcard(e)] if full: entries = [relpath(pathjoin(path, e)) for e in entries] elif absolute: entries = [abspath(pathjoin(path, e)) for e in entries] return entries @synchronize def makedir(self, path, recursive=False, allow_recreate=False): path = self.encode_path(path) return self.proxy.makedir(path, recursive, allow_recreate) @synchronize def remove(self, path): path = self.encode_path(path) return self.proxy.remove(path) @synchronize def removedir(self, path, recursive=False, force=False): path = self.encode_path(path) return self.proxy.removedir(path, recursive, force) @synchronize def rename(self, src, dst): src = self.encode_path(src) dst = self.encode_path(dst) return self.proxy.rename(src, dst) @synchronize def settimes(self, path, accessed_time, modified_time): path = self.encode_path(path) return self.proxy.settimes(path, accessed_time, modified_time) @synchronize def getinfo(self, path): path = self.encode_path(path) info = self.proxy.getinfo(path) return info @synchronize def desc(self, path): path = self.encode_path(path) return self.proxy.desc(path) @synchronize def getxattr(self, path, attr, default=None): path = self.encode_path(path) attr = self.encode_path(attr) return self.fs.getxattr(path, attr, default) @synchronize def setxattr(self, path, attr, value): path = self.encode_path(path) attr = self.encode_path(attr) return self.fs.setxattr(path, attr, value) @synchronize def delxattr(self, path, attr): path = self.encode_path(path) attr = self.encode_path(attr) return self.fs.delxattr(path, attr) @synchronize def listxattrs(self, path): path = self.encode_path(path) return [self.decode_path(a) for a in self.fs.listxattrs(path)] @synchronize def copy(self, src, dst, overwrite=False, chunk_size=16384): src = self.encode_path(src) dst = self.encode_path(dst) return self.proxy.copy(src, dst, overwrite, chunk_size) @synchronize def move(self, src, dst, overwrite=False, chunk_size=16384): src = self.encode_path(src) dst = self.encode_path(dst) return self.proxy.move(src, dst, overwrite, chunk_size) @synchronize def movedir(self, src, dst, overwrite=False, ignore_errors=False, chunk_size=16384): src = self.encode_path(src) dst = self.encode_path(dst) return self.proxy.movedir(src, dst, overwrite, ignore_errors, chunk_size) @synchronize def copydir(self, src, dst, overwrite=False, ignore_errors=False, chunk_size=16384): src = self.encode_path(src) dst = self.encode_path(dst) return self.proxy.copydir(src, dst, overwrite, ignore_errors, chunk_size) fs-0.5.4/fs/__init__.py0000664000175000017500000000336512621617161014625 0ustar willwill00000000000000""" fs: a filesystem abstraction. This module provides an abstract base class 'FS' that defines a consistent interface to different kinds of filesystem, along with a range of concrete implementations of this interface such as: OSFS: access the local filesystem, through the 'os' module TempFS: a temporary filesystem that's automatically cleared on exit MemoryFS: a filesystem that exists only in memory ZipFS: access a zipfile like a filesystem SFTPFS: access files on a SFTP server S3FS: access files stored in Amazon S3 """ __version__ = "0.5.4" __author__ = "Will McGugan (will@willmcgugan.com)" # provide these by default so people can use 'fs.path.basename' etc. from fs import errors from fs import path _thread_synchronize_default = True def set_thread_synchronize_default(sync): """Sets the default thread synchronisation flag. FS objects are made thread-safe through the use of a per-FS threading Lock object. Since this can introduce an small overhead it can be disabled with this function if the code is single-threaded. :param sync: Set whether to use thread synchronisation for new FS objects """ global _thread_synchronization_default _thread_synchronization_default = sync # Store some identifiers in the fs namespace import os SEEK_CUR = os.SEEK_CUR SEEK_END = os.SEEK_END SEEK_SET = os.SEEK_SET # Allow clean use of logging throughout the lib import logging as _logging class _NullHandler(_logging.Handler): def emit(self,record): pass _logging.getLogger("fs").addHandler(_NullHandler()) def getLogger(name): """Get a logger object for use within the pyfilesystem library.""" assert name.startswith("fs.") return _logging.getLogger(name) fs-0.5.4/fs/path.py0000664000175000017500000004031712512525115014014 0ustar willwill00000000000000""" fs.path ======= Useful functions for FS path manipulation. This is broadly similar to the standard ``os.path`` module but works with paths in the canonical format expected by all FS objects (that is, separated by forward slashes and with an optional leading slash). """ import re import os _requires_normalization = re.compile(r'(^|/)\.\.?($|/)|//').search def normpath(path): """Normalizes a path to be in the format expected by FS objects. This function removes trailing slashes, collapses duplicate slashes, and generally tries very hard to return a new path in the canonical FS format. If the path is invalid, ValueError will be raised. :param path: path to normalize :returns: a valid FS path >>> normpath("/foo//bar/frob/../baz") '/foo/bar/baz' >>> normpath("foo/../../bar") Traceback (most recent call last) ... BackReferenceError: Too many backrefs in 'foo/../../bar' """ if path in ('', '/'): return path # An early out if there is no need to normalize this path if not _requires_normalization(path): return path.rstrip('/') prefix = u'/' if path.startswith('/') else u'' components = [] append = components.append special = ('..', '.', '').__contains__ try: for component in path.split('/'): if special(component): if component == '..': components.pop() else: append(component) except IndexError: # Imported here because errors imports this module (path), # causing a circular import. from fs.errors import BackReferenceError raise BackReferenceError('Too many backrefs in \'%s\'' % path) return prefix + u'/'.join(components) if os.sep != '/': def ospath(path): """Replace path separators in an OS path if required""" return path.replace(os.sep, '/') else: def ospath(path): """Replace path separators in an OS path if required""" return path def iteratepath(path, numsplits=None): """Iterate over the individual components of a path. :param path: Path to iterate over :numsplits: Maximum number of splits """ path = relpath(normpath(path)) if not path: return [] if numsplits == None: return path.split('/') else: return path.split('/', numsplits) def recursepath(path, reverse=False): """Returns intermediate paths from the root to the given path :param reverse: reverses the order of the paths >>> recursepath('a/b/c') ['/', u'/a', u'/a/b', u'/a/b/c'] """ if path in ('', '/'): return [u'/'] path = abspath(normpath(path)) + '/' paths = [u'/'] find = path.find append = paths.append pos = 1 len_path = len(path) while pos < len_path: pos = find('/', pos) append(path[:pos]) pos += 1 if reverse: return paths[::-1] return paths def isabs(path): """Return True if path is an absolute path.""" return path.startswith('/') def abspath(path): """Convert the given path to an absolute path. Since FS objects have no concept of a 'current directory' this simply adds a leading '/' character if the path doesn't already have one. """ if not path.startswith('/'): return u'/' + path return path def relpath(path): """Convert the given path to a relative path. This is the inverse of abspath(), stripping a leading '/' from the path if it is present. :param path: Path to adjust >>> relpath('/a/b') 'a/b' """ return path.lstrip('/') def pathjoin(*paths): """Joins any number of paths together, returning a new path string. :param paths: Paths to join are given in positional arguments >>> pathjoin('foo', 'bar', 'baz') 'foo/bar/baz' >>> pathjoin('foo/bar', '../baz') 'foo/baz' >>> pathjoin('foo/bar', '/baz') '/baz' """ absolute = False relpaths = [] for p in paths: if p: if p[0] == '/': del relpaths[:] absolute = True relpaths.append(p) path = normpath(u"/".join(relpaths)) if absolute: path = abspath(path) return path def pathcombine(path1, path2): """Joins two paths together. This is faster than `pathjoin`, but only works when the second path is relative, and there are no backreferences in either path. >>> pathcombine("foo/bar", "baz") 'foo/bar/baz' """ if not path1: return path2.lstrip() return "%s/%s" % (path1.rstrip('/'), path2.lstrip('/')) def join(*paths): """Joins any number of paths together, returning a new path string. This is a simple alias for the ``pathjoin`` function, allowing it to be used as ``fs.path.join`` in direct correspondence with ``os.path.join``. :param paths: Paths to join are given in positional arguments """ return pathjoin(*paths) def pathsplit(path): """Splits a path into (head, tail) pair. This function splits a path into a pair (head, tail) where 'tail' is the last pathname component and 'head' is all preceding components. :param path: Path to split >>> pathsplit("foo/bar") ('foo', 'bar') >>> pathsplit("foo/bar/baz") ('foo/bar', 'baz') >>> pathsplit("/foo/bar/baz") ('/foo/bar', 'baz') """ if '/' not in path: return ('', path) split = path.rsplit('/', 1) return (split[0] or '/', split[1]) def split(path): """Splits a path into (head, tail) pair. This is a simple alias for the ``pathsplit`` function, allowing it to be used as ``fs.path.split`` in direct correspondence with ``os.path.split``. :param path: Path to split """ return pathsplit(path) def splitext(path): """Splits the extension from the path, and returns the path (up to the last '.' and the extension). :param path: A path to split >>> splitext('baz.txt') ('baz', 'txt') >>> splitext('foo/bar/baz.txt') ('foo/bar/baz', 'txt') """ parent_path, pathname = pathsplit(path) if '.' not in pathname: return path, '' pathname, ext = pathname.rsplit('.', 1) path = pathjoin(parent_path, pathname) return path, '.' + ext def isdotfile(path): """Detects if a path references a dot file, i.e. a resource who's name starts with a '.' :param path: Path to check >>> isdotfile('.baz') True >>> isdotfile('foo/bar/.baz') True >>> isdotfile('foo/bar.baz') False """ return basename(path).startswith('.') def dirname(path): """Returns the parent directory of a path. This is always equivalent to the 'head' component of the value returned by pathsplit(path). :param path: A FS path >>> dirname('foo/bar/baz') 'foo/bar' >>> dirname('/foo/bar') '/foo' >>> dirname('/foo') '/' """ return pathsplit(path)[0] def basename(path): """Returns the basename of the resource referenced by a path. This is always equivalent to the 'tail' component of the value returned by pathsplit(path). :param path: A FS path >>> basename('foo/bar/baz') 'baz' >>> basename('foo/bar') 'bar' >>> basename('foo/bar/') '' """ return pathsplit(path)[1] def issamedir(path1, path2): """Return true if two paths reference a resource in the same directory. :param path1: An FS path :param path2: An FS path >>> issamedir("foo/bar/baz.txt", "foo/bar/spam.txt") True >>> issamedir("foo/bar/baz/txt", "spam/eggs/spam.txt") False """ return dirname(normpath(path1)) == dirname(normpath(path2)) def isbase(path1, path2): p1 = forcedir(abspath(path1)) p2 = forcedir(abspath(path2)) return p1 == p2 or p1.startswith(p2) def isprefix(path1, path2): """Return true is path1 is a prefix of path2. :param path1: An FS path :param path2: An FS path >>> isprefix("foo/bar", "foo/bar/spam.txt") True >>> isprefix("foo/bar/", "foo/bar") True >>> isprefix("foo/barry", "foo/baz/bar") False >>> isprefix("foo/bar/baz/", "foo/baz/bar") False """ bits1 = path1.split("/") bits2 = path2.split("/") while bits1 and bits1[-1] == "": bits1.pop() if len(bits1) > len(bits2): return False for (bit1, bit2) in zip(bits1, bits2): if bit1 != bit2: return False return True def forcedir(path): """Ensure the path ends with a trailing forward slash :param path: An FS path >>> forcedir("foo/bar") 'foo/bar/' >>> forcedir("foo/bar/") 'foo/bar/' """ if not path.endswith('/'): return path + '/' return path def frombase(path1, path2): if not isprefix(path1, path2): raise ValueError("path1 must be a prefix of path2") return path2[len(path1):] def relativefrom(base, path): """Return a path relative from a given base path, i.e. insert backrefs as appropriate to reach the path from the base. :param base_path: Path to a directory :param path: Path you wish to make relative >>> relativefrom("foo/bar", "baz/index.html") '../../baz/index.html' """ base = list(iteratepath(base)) path = list(iteratepath(path)) common = 0 for a, b in zip(base, path): if a != b: break common += 1 return u'/'.join([u'..'] * (len(base) - common) + path[common:]) class PathMap(object): """Dict-like object with paths for keys. A PathMap is like a dictionary where the keys are all FS paths. It has two main advantages over a standard dictionary. First, keys are normalized automatically:: >>> pm = PathMap() >>> pm["hello/world"] = 42 >>> print pm["/hello/there/../world"] 42 Second, various dictionary operations (e.g. listing or clearing values) can be efficiently performed on a subset of keys sharing some common prefix:: # list all values in the map pm.values() # list all values for paths starting with "/foo/bar" pm.values("/foo/bar") Under the hood, a PathMap is a trie-like structure where each level is indexed by path name component. This allows lookups to be performed in O(number of path components) while permitting efficient prefix-based operations. """ def __init__(self): self._map = {} def __getitem__(self, path): """Get the value stored under the given path.""" m = self._map for name in iteratepath(path): try: m = m[name] except KeyError: raise KeyError(path) try: return m[""] except KeyError: raise KeyError(path) def __contains__(self, path): """Check whether the given path has a value stored in the map.""" try: self[path] except KeyError: return False else: return True def __setitem__(self, path, value): """Set the value stored under the given path.""" m = self._map for name in iteratepath(path): try: m = m[name] except KeyError: m = m.setdefault(name, {}) m[""] = value def __delitem__(self, path): """Delete the value stored under the given path.""" ms = [[self._map, None]] for name in iteratepath(path): try: ms.append([ms[-1][0][name], None]) except KeyError: raise KeyError(path) else: ms[-2][1] = name try: del ms[-1][0][""] except KeyError: raise KeyError(path) else: while len(ms) > 1 and not ms[-1][0]: del ms[-1] del ms[-1][0][ms[-1][1]] def get(self, path, default=None): """Get the value stored under the given path, or the given default.""" try: return self[path] except KeyError: return default def pop(self, path, default=None): """Pop the value stored under the given path, or the given default.""" ms = [[self._map, None]] for name in iteratepath(path): try: ms.append([ms[-1][0][name], None]) except KeyError: return default else: ms[-2][1] = name try: val = ms[-1][0].pop("") except KeyError: val = default else: while len(ms) > 1 and not ms[-1][0]: del ms[-1] del ms[-1][0][ms[-1][1]] return val def setdefault(self, path, value): m = self._map for name in iteratepath(path): try: m = m[name] except KeyError: m = m.setdefault(name, {}) return m.setdefault("", value) def clear(self, root="/"): """Clear all entries beginning with the given root path.""" m = self._map for name in iteratepath(root): try: m = m[name] except KeyError: return m.clear() def iterkeys(self, root="/", m=None): """Iterate over all keys beginning with the given root path.""" if m is None: m = self._map for name in iteratepath(root): try: m = m[name] except KeyError: return for (nm, subm) in m.iteritems(): if not nm: yield abspath(root) else: k = pathcombine(root, nm) for subk in self.iterkeys(k, subm): yield subk def __iter__(self): return self.iterkeys() def keys(self,root="/"): return list(self.iterkeys(root)) def itervalues(self, root="/", m=None): """Iterate over all values whose keys begin with the given root path.""" root = normpath(root) if m is None: m = self._map for name in iteratepath(root): try: m = m[name] except KeyError: return for (nm, subm) in m.iteritems(): if not nm: yield subm else: k = pathcombine(root, nm) for subv in self.itervalues(k, subm): yield subv def values(self, root="/"): return list(self.itervalues(root)) def iteritems(self, root="/", m=None): """Iterate over all (key,value) pairs beginning with the given root.""" root = normpath(root) if m is None: m = self._map for name in iteratepath(root): try: m = m[name] except KeyError: return for (nm, subm) in m.iteritems(): if not nm: yield (abspath(normpath(root)), subm) else: k = pathcombine(root, nm) for (subk, subv) in self.iteritems(k, subm): yield (subk, subv) def items(self, root="/"): return list(self.iteritems(root)) def iternames(self, root="/"): """Iterate over all names beneath the given root path. This is basically the equivalent of listdir() for a PathMap - it yields the next level of name components beneath the given path. """ m = self._map for name in iteratepath(root): try: m = m[name] except KeyError: return for (nm, subm) in m.iteritems(): if nm and subm: yield nm def names(self, root="/"): return list(self.iternames(root)) _wild_chars = frozenset('*?[]!{}') def iswildcard(path): """Check if a path ends with a wildcard >>> is_wildcard('foo/bar/baz.*') True >>> is_wildcard('foo/bar') False """ assert path is not None return not _wild_chars.isdisjoint(path) if __name__ == "__main__": print recursepath('a/b/c') print relativefrom('/', '/foo') print relativefrom('/foo/bar', '/foo/baz') print relativefrom('/foo/bar/baz', '/foo/egg') print relativefrom('/foo/bar/baz/egg', '/foo/egg') fs-0.5.4/fs/opener.py0000664000175000017500000006527612621464031014363 0ustar willwill00000000000000""" fs.opener ========= Open filesystems via a URI. There are occasions when you want to specify a filesystem from the command line or in a config file. This module enables that functionality, and can return an FS object given a filesystem specification in a URI-like syntax (inspired by the syntax of http://commons.apache.org/vfs/filesystems.html). The `OpenerRegistry` class maps the protocol (file, ftp etc.) on to an Opener object, which returns an appropriate filesystem object and path. You can create a custom opener registry that opens just the filesystems you require, or use the opener registry defined here (also called `opener`) that can open any supported filesystem. The `parse` method of an `OpenerRegsitry` object returns a tuple of an FS object a path. Here's an example of how to use the default opener registry:: >>> from fs.opener import opener >>> opener.parse('ftp://ftp.mozilla.org/pub') (, u'pub') You can use use the `opendir` method, which just returns an FS object. In the example above, `opendir` will return a FS object for the directory `pub`:: >>> opener.opendir('ftp://ftp.mozilla.org/pub') /pub> If you are just interested in a single file, use the `open` method of a registry which returns a file-like object, and has the same signature as FS objects and the `open` builtin:: >>> opener.open('ftp://ftp.mozilla.org/pub/README') The `opendir` and `open` methods can also be imported from the top-level of this module for sake of convenience. To avoid shadowing the builtin `open` method, they are named `fsopendir` and `fsopen`. Here's how you might import them:: from fs.opener import fsopendir, fsopen """ __all__ = ['OpenerError', 'NoOpenerError', 'OpenerRegistry', 'opener', 'fsopen', 'fsopendir', 'OpenerRegistry', 'Opener', 'OSFSOpener', 'ZipOpener', 'RPCOpener', 'FTPOpener', 'SFTPOpener', 'MemOpener', 'DebugOpener', 'TempOpener', 'S3Opener', 'TahoeOpener', 'DavOpener', 'HTTPOpener'] from fs.path import pathsplit, join, iswildcard, normpath from fs.osfs import OSFS from fs.filelike import FileWrapper from os import getcwd import os.path import re from urlparse import urlparse class OpenerError(Exception): """The base exception thrown by openers""" pass class NoOpenerError(OpenerError): """Thrown when there is no opener for the given protocol""" pass def _expand_syspath(path): if path is None: return path if path.startswith('\\\\?\\'): path = path[4:] path = os.path.expanduser(os.path.expandvars(path)) path = os.path.normpath(os.path.abspath(path)) return path def _parse_credentials(url): scheme = None if '://' in url: scheme, url = url.split('://', 1) username = None password = None if '@' in url: credentials, url = url.split('@', 1) if ':' in credentials: username, password = credentials.split(':', 1) else: username = credentials if scheme is not None: url = '%s://%s' % (scheme, url) return username, password, url def _parse_name(fs_name): if '#' in fs_name: fs_name, fs_name_params = fs_name.split('#', 1) return fs_name, fs_name_params else: return fs_name, None def _split_url_path(url): if '://' not in url: url = 'http://' + url scheme, netloc, path, _params, _query, _fragment = urlparse(url) url = '%s://%s' % (scheme, netloc) return url, path def _FSClosingFile(fs, file_object, mode): original_close = file_object.close def close(): try: fs.close() except: pass return original_close() file_object.close = close return file_object class OpenerRegistry(object): """An opener registry that stores a number of opener objects used to parse FS URIs""" re_fs_url = re.compile(r''' ^ (.*?) :\/\/ (?: (?:(.*?)@(.*?)) |(.*?) ) (?: !(.*?)$ )*$ ''', re.VERBOSE) def __init__(self, openers=[]): self.registry = {} self.openers = {} self.default_opener = 'osfs' for opener in openers: self.add(opener) @classmethod def split_segments(self, fs_url): match = self.re_fs_url.match(fs_url) return match def get_opener(self, name): """Retrieve an opener for the given protocol :param name: name of the opener to open :raises NoOpenerError: if no opener has been registered of that name """ if name not in self.registry: raise NoOpenerError("No opener for %s" % name) index = self.registry[name] return self.openers[index] def add(self, opener): """Adds an opener to the registry :param opener: a class derived from fs.opener.Opener """ index = len(self.openers) self.openers[index] = opener for name in opener.names: self.registry[name] = index def parse(self, fs_url, default_fs_name=None, writeable=False, create_dir=False, cache_hint=True): """Parses a FS url and returns an fs object a path within that FS object (if indicated in the path). A tuple of (, ) is returned. :param fs_url: an FS url :param default_fs_name: the default FS to use if none is indicated (defaults is OSFS) :param writeable: if True, a writeable FS will be returned :param create_dir: if True, then the directory in the FS will be created """ orig_url = fs_url match = self.split_segments(fs_url) if match: fs_name, credentials, url1, url2, path = match.groups() if credentials: fs_url = '%s@%s' % (credentials, url1) else: fs_url = url2 path = path or '' fs_url = fs_url or '' if ':' in fs_name: fs_name, sub_protocol = fs_name.split(':', 1) fs_url = '%s://%s' % (sub_protocol, fs_url) if '!' in path: paths = path.split('!') path = paths.pop() fs_url = '%s!%s' % (fs_url, '!'.join(paths)) fs_name = fs_name or self.default_opener else: fs_name = default_fs_name or self.default_opener fs_url = _expand_syspath(fs_url) path = '' fs_name, fs_name_params = _parse_name(fs_name) opener = self.get_opener(fs_name) if fs_url is None: raise OpenerError("Unable to parse '%s'" % orig_url) fs, fs_path = opener.get_fs(self, fs_name, fs_name_params, fs_url, writeable, create_dir) fs.cache_hint(cache_hint) if fs_path and iswildcard(fs_path): pathname, resourcename = pathsplit(fs_path or '') if pathname: fs = fs.opendir(pathname) return fs, resourcename fs_path = join(fs_path, path) if create_dir and fs_path: if not fs.getmeta('read_only', False): fs.makedir(fs_path, allow_recreate=True) pathname, resourcename = pathsplit(fs_path or '') if pathname and resourcename: fs = fs.opendir(pathname) fs_path = resourcename return fs, fs_path or '' def open(self, fs_url, mode='r', **kwargs): """Opens a file from a given FS url If you intend to do a lot of file manipulation, it would likely be more efficient to do it directly through the an FS instance (from `parse` or `opendir`). This method is fine for one-offs though. :param fs_url: a FS URL, e.g. ftp://ftp.mozilla.org/README :param mode: mode to open file file :rtype: a file """ writeable = 'w' in mode or 'a' in mode or '+' in mode fs, path = self.parse(fs_url, writeable=writeable) file_object = fs.open(path, mode) file_object = _FSClosingFile(fs, file_object, mode) #file_object.fs = fs return file_object def getcontents(self, fs_url, mode='rb', encoding=None, errors=None, newline=None): """Gets the contents from a given FS url (if it references a file) :param fs_url: a FS URL e.g. ftp://ftp.mozilla.org/README """ fs, path = self.parse(fs_url) return fs.getcontents(path, mode, encoding=encoding, errors=errors, newline=newline) def opendir(self, fs_url, writeable=True, create_dir=False): """Opens an FS object from an FS URL :param fs_url: an FS URL e.g. ftp://ftp.mozilla.org :param writeable: set to True (the default) if the FS must be writeable :param create_dir: create the directory references by the FS URL, if it doesn't already exist """ fs, path = self.parse(fs_url, writeable=writeable, create_dir=create_dir) if path and '://' not in fs_url: # A shortcut to return an OSFS rather than a SubFS for os paths return OSFS(fs_url) if path: fs = fs.opendir(path) return fs class Opener(object): """The base class for openers Opener follow a very simple protocol. To create an opener, derive a class from `Opener` and define a classmethod called `get_fs`, which should have the following signature:: @classmethod def get_fs(cls, registry, fs_name, fs_name_params, fs_path, writeable, create_dir): The parameters of `get_fs` are as follows: * `fs_name` the name of the opener, as extracted from the protocol part of the url, * `fs_name_params` reserved for future use * `fs_path` the path part of the url * `writeable` if True, then `get_fs` must return an FS that can be written to * `create_dir` if True then `get_fs` should attempt to silently create the directory references in path In addition to `get_fs` an opener class should contain two class attributes: names and desc. `names` is a list of protocols that list opener will opener. `desc` is an English description of the individual opener syntax. """ pass class OSFSOpener(Opener): names = ['osfs', 'file'] desc = """OS filesystem opener, works with any valid system path. This is the default opener and will be used if you don't indicate which opener to use. examples: * file://relative/foo/bar/baz.txt (opens a relative file) * file:///home/user (opens a directory from a absolute path) * osfs://~/ (open the user's home directory) * foo/bar.baz (file:// is the default opener)""" @classmethod def get_fs(cls, registry, fs_name, fs_name_params, fs_path, writeable, create_dir): from fs.osfs import OSFS path = os.path.normpath(fs_path) if create_dir and not os.path.exists(path): from fs.osfs import _os_makedirs _os_makedirs(path) dirname, resourcename = os.path.split(fs_path) osfs = OSFS(dirname) return osfs, resourcename class ZipOpener(Opener): names = ['zip', 'zip64'] desc = """Opens zip files. Use zip64 for > 2 gigabyte zip files, if you have a 64 bit processor. examples: * zip://myzip.zip (open a local zip file) * zip://myzip.zip!foo/bar/insidezip.txt (reference a file insize myzip.zip) * zip:ftp://ftp.example.org/myzip.zip (open a zip file stored on a ftp server)""" @classmethod def get_fs(cls, registry, fs_name, fs_name_params, fs_path, writeable, create_dir): zip_fs, zip_path = registry.parse(fs_path) if zip_path is None: raise OpenerError('File required for zip opener') if zip_fs.exists(zip_path): if writeable: open_mode = 'r+b' else: open_mode = 'rb' else: open_mode = 'w+' if zip_fs.hassyspath(zip_path): zip_file = zip_fs.getsyspath(zip_path) else: zip_file = zip_fs.open(zip_path, mode=open_mode) _username, _password, fs_path = _parse_credentials(fs_path) from fs.zipfs import ZipFS if zip_file is None: zip_file = fs_path mode = 'r' if writeable: mode = 'a' allow_zip_64 = fs_name.endswith('64') zipfs = ZipFS(zip_file, mode=mode, allow_zip_64=allow_zip_64) return zipfs, None class RPCOpener(Opener): names = ['rpc'] desc = """An opener for filesystems server over RPC (see the fsserve command). examples: rpc://127.0.0.1:8000 (opens a RPC server running on local host, port 80) rpc://www.example.org (opens an RPC server on www.example.org, default port 80)""" @classmethod def get_fs(cls, registry, fs_name, fs_name_params, fs_path, writeable, create_dir): from fs.rpcfs import RPCFS _username, _password, fs_path = _parse_credentials(fs_path) if '://' not in fs_path: fs_path = 'http://' + fs_path scheme, netloc, path, _params, _query, _fragment = urlparse(fs_path) rpcfs = RPCFS('%s://%s' % (scheme, netloc)) if create_dir and path: rpcfs.makedir(path, recursive=True, allow_recreate=True) return rpcfs, path or None class FTPOpener(Opener): names = ['ftp'] desc = """An opener for FTP (File Transfer Protocl) server examples: * ftp://ftp.mozilla.org (opens the root of ftp.mozilla.org) * ftp://ftp.example.org/foo/bar (opens /foo/bar on ftp.mozilla.org)""" @classmethod def get_fs(cls, registry, fs_name, fs_name_params, fs_path, writeable, create_dir): from fs.ftpfs import FTPFS username, password, fs_path = _parse_credentials(fs_path) scheme, _netloc, _path, _params, _query, _fragment = urlparse(fs_path) if not scheme: fs_path = 'ftp://' + fs_path scheme, netloc, path, _params, _query, _fragment = urlparse(fs_path) dirpath, resourcepath = pathsplit(path) url = netloc ftpfs = FTPFS(url, user=username or '', passwd=password or '', follow_symlinks=(fs_name_params == "symlinks")) ftpfs.cache_hint(True) if create_dir and path: ftpfs.makedir(path, recursive=True, allow_recreate=True) if dirpath: ftpfs = ftpfs.opendir(dirpath) if not resourcepath: return ftpfs, None else: return ftpfs, resourcepath class SFTPOpener(Opener): names = ['sftp'] desc = """An opener for SFTP (Secure File Transfer Protocol) servers examples: * sftp://username:password@example.org (opens sftp server example.org with username and password * sftp://example.org (opens example.org with public key authentication)""" @classmethod def get_fs(cls, registry, fs_name, fs_name_params, fs_path, writeable, create_dir): username, password, fs_path = _parse_credentials(fs_path) from fs.sftpfs import SFTPFS credentials = {} if username is not None: credentials['username'] = username if password is not None: credentials['password'] = password if '/' in fs_path: addr, fs_path = fs_path.split('/', 1) else: addr = fs_path fs_path = '/' fs_path, resourcename = pathsplit(fs_path) host = addr port = None if ':' in host: addr, port = host.rsplit(':', 1) try: port = int(port) except ValueError: pass else: host = (addr, port) if create_dir: sftpfs = SFTPFS(host, root_path='/', **credentials) if not sftpfs._transport.is_authenticated(): sftpfs.close() raise OpenerError('SFTP requires authentication') sftpfs = sftpfs.makeopendir(fs_path) return sftpfs, None sftpfs = SFTPFS(host, root_path=fs_path, **credentials) if not sftpfs._transport.is_authenticated(): sftpfs.close() raise OpenerError('SFTP requires authentication') return sftpfs, resourcename class MemOpener(Opener): names = ['mem', 'ram'] desc = """Creates an in-memory filesystem (very fast but contents will disappear on exit). Useful for creating a fast temporary filesystem for serving or mounting with fsserve or fsmount. NB: If you user fscp or fsmv to copy/move files here, you are effectively deleting them! examples: * mem:// (opens a new memory filesystem) * mem://foo/bar (opens a new memory filesystem with subdirectory /foo/bar) """ @classmethod def get_fs(cls, registry, fs_name, fs_name_params, fs_path, writeable, create_dir): from fs.memoryfs import MemoryFS memfs = MemoryFS() if create_dir: memfs = memfs.makeopendir(fs_path) return memfs, None class DebugOpener(Opener): names = ['debug'] desc = """For developers -- adds debugging information to output. example: * debug:ftp://ftp.mozilla.org (displays details of calls made to a ftp filesystem)""" @classmethod def get_fs(cls, registry, fs_name, fs_name_params, fs_path, writeable, create_dir): from fs.wrapfs.debugfs import DebugFS if fs_path: fs, _path = registry.parse(fs_path, writeable=writeable, create_dir=create_dir) return DebugFS(fs, verbose=False), None if fs_name_params == 'ram': from fs.memoryfs import MemoryFS return DebugFS(MemoryFS(), identifier=fs_name_params, verbose=False), None else: from fs.tempfs import TempFS return DebugFS(TempFS(), identifier=fs_name_params, verbose=False), None class TempOpener(Opener): names = ['temp'] desc = """Creates a temporary filesystem that is erased on exit. Probably only useful for mounting or serving. NB: If you use fscp or fsmv to copy/move files here, you are effectively deleting them! example: * temp://""" @classmethod def get_fs(cls, registry, fs_name, fs_name_params, fs_path, writeable, create_dir): from fs.tempfs import TempFS from fs.wrapfs.lazyfs import LazyFS fs = LazyFS((TempFS,(),{"identifier":fs_name_params})) return fs, fs_path class S3Opener(Opener): names = ['s3'] desc = """Opens a filesystem stored on Amazon S3 storage The environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY should be set""" @classmethod def get_fs(cls, registry, fs_name, fs_name_params, fs_path, writeable, create_dir): from fs.s3fs import S3FS username, password, bucket = _parse_credentials(fs_path) path = '' if '/' in bucket: bucket, path = fs_path.split('/', 1) fs = S3FS(bucket, aws_access_key=username or None, aws_secret_key=password or None) if path: dirpath, resourcepath = pathsplit(path) if dirpath: fs = fs.opendir(dirpath) path = resourcepath return fs, path class TahoeOpener(Opener): names = ['tahoe'] desc = """Opens a Tahoe-LAFS filesystem example: * tahoe://http://pubgrid.tahoe-lafs.org/uri/URI:DIR2:h5bkxelehowscijdb [...]""" @classmethod def get_fs(cls, registry, fs_name, fs_name_params, fs_path, writeable, create_dir): from fs.contrib.tahoelafs import TahoeLAFS if '/uri/' not in fs_path: raise OpenerError("""Tahoe-LAFS url should be in the form /uri/""") url, dircap = fs_path.split('/uri/') path = '' if '/' in dircap: dircap, path = dircap.split('/', 1) fs = TahoeLAFS(dircap, webapi=url) if '/' in path: dirname, _resourcename = pathsplit(path) if create_dir: fs = fs.makeopendir(dirname) else: fs = fs.opendir(dirname) path = '' return fs, path class DavOpener(Opener): names = ['dav'] desc = """Opens a WebDAV server example: * dav://example.org/dav""" @classmethod def get_fs(cls, registry, fs_name, fs_name_params, fs_path, writeable, create_dir): from fs.contrib.davfs import DAVFS url = fs_path if '://' not in url: url = 'http://' + url scheme, url = url.split('://', 1) username, password, url = _parse_credentials(url) credentials = None if username or password: credentials = {} if username: credentials['username'] = username if password: credentials['password'] = password url = '%s://%s' % (scheme, url) fs = DAVFS(url, credentials=credentials) return fs, '' class HTTPOpener(Opener): names = ['http', 'https'] desc = """HTTP file opener. HTTP only supports reading files, and not much else. example: * http://www.example.org/index.html""" @classmethod def get_fs(cls, registry, fs_name, fs_name_params, fs_path, writeable, create_dir): from fs.httpfs import HTTPFS if '/' in fs_path: dirname, resourcename = fs_path.rsplit('/', 1) else: dirname = fs_path resourcename = '' fs = HTTPFS('http://' + dirname) return fs, resourcename class UserDataOpener(Opener): names = ['appuserdata', 'appuser'] desc = """Opens a filesystem for a per-user application directory. The 'domain' should be in the form :. (the author name and version are optional). example: * appuserdata://myapplication * appuserdata://examplesoft:myapplication * appuserdata://anotherapp.1.1 * appuserdata://examplesoft:anotherapp.1.3""" FSClass = 'UserDataFS' @classmethod def get_fs(cls, registry, fs_name, fs_name_params, fs_path, writeable, create_dir): import fs.appdirfs fs_class = getattr(fs.appdirfs, cls.FSClass) if ':' in fs_path: appauthor, appname = fs_path.split(':', 1) else: appauthor = None appname = fs_path if '/' in appname: appname, path = appname.split('/', 1) else: path = '' if '.' in appname: appname, appversion = appname.split('.', 1) else: appversion = None fs = fs_class(appname, appauthor=appauthor, version=appversion, create=create_dir) if '/' in path: subdir, path = path.rsplit('/', 1) if create_dir: fs = fs.makeopendir(subdir, recursive=True) else: fs = fs.opendir(subdir) return fs, path class SiteDataOpener(UserDataOpener): names = ['appsitedata', 'appsite'] desc = """Opens a filesystem for an application site data directory. The 'domain' should be in the form :. (the author name and version are optional). example: * appsitedata://myapplication * appsitedata://examplesoft:myapplication * appsitedata://anotherapp.1.1 * appsitedata://examplesoft:anotherapp.1.3""" FSClass = 'SiteDataFS' class UserCacheOpener(UserDataOpener): names = ['appusercache', 'appcache'] desc = """Opens a filesystem for an per-user application cache directory. The 'domain' should be in the form :. (the author name and version are optional). example: * appusercache://myapplication * appusercache://examplesoft:myapplication * appusercache://anotherapp.1.1 * appusercache://examplesoft:anotherapp.1.3""" FSClass = 'UserCacheFS' class UserLogOpener(UserDataOpener): names = ['appuserlog', 'applog'] desc = """Opens a filesystem for an application site data directory. The 'domain' should be in the form :. (the author name and version are optional). example: * appuserlog://myapplication * appuserlog://examplesoft:myapplication * appuserlog://anotherapp.1.1 * appuserlog://examplesoft:anotherapp.1.3""" FSClass = 'UserLogFS' class MountOpener(Opener): names = ['mount'] desc = """Mounts other filesystems on a 'virtual' filesystem The path portion of the FS URL should be a path to an ini file, where the keys are the mount point, and the values are FS URLs to mount. The following is an example of such an ini file: [fs] resources=appuser://myapp/resources foo=~/foo foo/bar=mem:// [fs2] bar=~/bar example: * mount://fs.ini * mount://fs.ini!resources * mount://fs.ini:fs2""" @classmethod def get_fs(cls, registry, fs_name, fs_name_params, fs_path, writeable, create_dir): from fs.mountfs import MountFS from ConfigParser import ConfigParser cfg = ConfigParser() if '#' in fs_path: path, section = fs_path.split('#', 1) else: path = fs_path section = 'fs' cfg.readfp(registry.open(path)) mount_fs = MountFS() for mount_point, mount_path in cfg.items(section): mount_fs.mount(mount_point, registry.opendir(mount_path, create_dir=create_dir)) return mount_fs, '' class MultiOpener(Opener): names = ['multi'] desc = """Combines other filesystems in to a single filesystem. The path portion of the FS URL should be a path to an ini file, where the keys are the mount point, and the values are FS URLs to mount. The following is an example of such an ini file: [templates] dir1=templates/foo dir2=templates/bar example: * multi://fs.ini""" @classmethod def get_fs(cls, registry, fs_name, fs_name_params, fs_path, writeable, create_dir): from fs.multifs import MultiFS from ConfigParser import ConfigParser cfg = ConfigParser() if '#' in fs_path: path, section = fs_path.split('#', 1) else: path = fs_path section = 'fs' cfg.readfp(registry.open(path)) multi_fs = MultiFS() for name, fs_url in cfg.items(section): multi_fs.addfs(name, registry.opendir(fs_url, create_dir=create_dir)) return multi_fs, '' opener = OpenerRegistry([OSFSOpener, ZipOpener, RPCOpener, FTPOpener, SFTPOpener, MemOpener, DebugOpener, TempOpener, S3Opener, TahoeOpener, DavOpener, HTTPOpener, UserDataOpener, SiteDataOpener, UserCacheOpener, UserLogOpener, MountOpener, MultiOpener ]) fsopen = opener.open fsopendir = opener.opendir fs-0.5.4/LICENSE.txt0000664000175000017500000000304012512525115013711 0ustar willwill00000000000000Copyright (c) 2009-2015, Will McGugan and contributors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of PyFilesystem nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. fs-0.5.4/README.txt0000664000175000017500000000542612621464031013576 0ustar willwill00000000000000PyFilesystem ============ PyFilesystem is an abstraction layer for *filesystems*. In the same way that Python's file-like objects provide a common way of accessing files, PyFilesystem provides a common way of accessing entire filesystems. You can write platform-independent code to work with local files, that also works with any of the supported filesystems (zip, ftp, S3 etc.). Pyfilesystem works with Linux, Windows and Mac. Suported Filesystems --------------------- Here are a few of the filesystems that can be accessed with Pyfilesystem: * **DavFS** access files & directories on a WebDAV server * **FTPFS** access files & directories on an FTP server * **MemoryFS** access files & directories stored in memory (non-permanent but very fast) * **MountFS** creates a virtual directory structure built from other filesystems * **MultiFS** a virtual filesystem that combines a list of filesystems into one, and checks them in order when opening files * **OSFS** the native filesystem * **SFTPFS** access files & directores stored on a Secure FTP server * **S3FS** access files & directories stored on Amazon S3 storage * **TahoeLAFS** access files & directories stored on a Tahoe distributed filesystem * **ZipFS** access files and directories contained in a zip file Example ------- The following snippet prints the total number of bytes contained in all your Python files in `C:/projects` (including sub-directories):: from fs.osfs import OSFS projects_fs = OSFS('C:/projects') print sum(projects_fs.getsize(path) for path in projects_fs.walkfiles(wildcard="*.py")) That is, assuming you are on Windows and have a directory called 'projects' in your C drive. If you are on Linux / Mac, you might replace the second line with something like:: projects_fs = OSFS('~/projects') If you later want to display the total size of Python files stored in a zip file, you could make the following change to the first two lines:: from fs.zipfs import ZipFS projects_fs = ZipFS('source.zip') In fact, you could use any of the supported filesystems above, and the code would continue to work as before. An alternative to explicitly importing the filesystem class you want, is to use an FS opener which opens a filesystem from a URL-like syntax:: from fs.opener import fsopendir projects_fs = fsopendir('C:/projects') You could change ``C:/projects`` to ``zip://source.zip`` to open the zip file, or even ``ftp://ftp.example.org/code/projects/`` to sum up the bytes of Python stored on an ftp server. Screencast ---------- This is from an early version of PyFilesystem, but still relevant http://vimeo.com/12680842 Discussion Group ---------------- http://groups.google.com/group/pyfilesystem-discussion Further Information ------------------- http://www.willmcgugan.com/tag/fs/ fs-0.5.4/setup.cfg0000644000000000000000000000007312621617365013720 0ustar rootroot00000000000000[egg_info] tag_build = tag_date = 0 tag_svn_revision = 0 fs-0.5.4/CHANGES.txt0000664000175000017500000001154612573252400013712 0ustar willwill00000000000000 0.3: * New FS implementations: * FTPFS: access a plain old FTP server * S3FS: access remote files stored in Amazon S3 * RPCFS: access remote files using a simple XML-RPC protocol * SFTPFS: access remote files on a SFTP server * WrapFS: filesystem that wraps an FS object and transparently modifies its contents (think encryption, compression, ...) * LazyFS: lazily instantiate an FS object the first time it is used * ReadOnlyFS: a WrapFS that makes an fs read-only * Ability to expose FS objects to the outside world: * expose.fuse: expose an FS object using FUSE * expose.xmlrpc: expose an FS object a simple XML-RPC protocol * expose.sftp: expose an FS object SFTP * expose.django_storage: convert FS object to Django Storage object * Extended attribute support (getxattr/setxattr/delxattr/listxattrs) * Change watching support (add_watcher/del_watcher) * Insist on unicode paths throughout: * output paths are always unicode * bytestring input paths are decoded as early as possible * Renamed "fs.helpers" to "fs.path", and renamed the contained functions to match those offered by os.path * fs.remote: utilities for implementing FS classes that interface with a remote filesystem * fs.errors: updated exception hierarchy, with support for converting to/from standard OSError instances * Added cache_hint method to base.py * Added settimes method to base implementation * New implementation of print_fs, accessible through tree method on base class 0.4: * New FS implementations (under fs.contrib): * BigFS: read contents of a BIG file (C&C game file format) * DAVFS: access remote files stored on a WebDAV server * TahoeLAFS: access files stored in a Tahoe-LAFS grid * New fs.expose implementations: * dokan: mount an FS object as a drive using Dokan (win32-only) * importhook: import modules from files in an FS object * Modified listdir and walk methods to accept callables as well as strings for wildcards. * Added listdirinfo method, which yields both the entry names and the corresponding info dicts in a single operation. * Made SubFS a subclass of WrapFS, and moved it into its own module at fs.wrapfs.subfs. * Path-handling fixes for OSFS on win32: * Work properly when pointing to the root of a drive. * Better handling of remote UNC paths. * Add ability to switch off use of long UNC paths. * OSFSWatchMixin improvements: * watch_inotify: allow more than one watcher on a single path. * watch_win32: don't create immortal reference cycles. * watch_win32: report errors if the filesystem does't support ReadDirectoryChangesW. * MountFS: added support for mounting at the root directory, and for mounting over an existing mount. * Added 'getpathurl' and 'haspathurl' methods. * Added utils.isdir(fs,path,info) and utils.isfile(fs,path,info); these can often determine whether a path is a file or directory by inspecting the info dict and avoid an additional query to the filesystem. * Added utility module 'fs.filelike' with some helpers for building and manipulating file-like objects. * Added getmeta and hasmeta methods * Separated behaviour of setcontents and createfile * Added a getmmap to base * Added command line scripts fsls, fstree, fscat, fscp, fsmv * Added command line scripts fsmkdir, fsmount * Made SFTP automatically pick up keys if no other authentication is available * Optimized listdir and listdirinfo in SFTPFS * Made memoryfs work with threads * Added copyfile_non_atomic and movefile_non_atomic for improved performance of multi-threaded copies * Added a concept of a writeable FS to MultiFS * Added ilistdir() and ilistdirinfo() methods, which are generator-based variants of listdir() and listdirinfo(). * Removed obsolete module fs.objectree; use fs.path.PathMap instead. * Added setcontents_async method to base * Added `appdirfs` module to abstract per-user application directories 0.5: * Ported to Python 3.X * Added a DeleteRootError to exceptions thrown when trying to delete '/' * Added a remove_all function to utils * Added sqlitefs to fs.contrib, contributed by Nitin Bhide * Added archivefs to fs.contrib, contributed by btimby * Added some polish to fstree command and unicode box lines rather than ascii art 0.5.1: * Fixed a hang bug in readline * Added copydir_progress to fs.utils 0.5.2: * Added utils.open_atomic_write 0.5.3: * Implemented scandir in listdir if available * Fix for issue where local.getpreferredencoding returns empty string fs-0.5.4/AUTHORS0000664000175000017500000000010212512525115013132 0ustar willwill00000000000000 Will McGugan (will@willmcgugan.com) Ryan Kelly (ryan@rfk.id.au) fs-0.5.4/setup.py0000664000175000017500000000350312621617201013603 0ustar willwill00000000000000#!/usr/bin/env python from setuptools import setup import sys PY3 = sys.version_info >= (3,) VERSION = "0.5.4" COMMANDS = ['fscat', 'fsinfo', 'fsls', 'fsmv', 'fscp', 'fsrm', 'fsserve', 'fstree', 'fsmkdir', 'fsmount'] CONSOLE_SCRIPTS = ['{0} = fs.commands.{0}:run'.format(command) for command in COMMANDS] classifiers = [ "Development Status :: 5 - Production/Stable", 'Intended Audience :: Developers', 'License :: OSI Approved :: BSD License', 'Operating System :: OS Independent', 'Programming Language :: Python', 'Programming Language :: Python :: 2.6', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Topic :: System :: Filesystems', ] with open('README.txt', 'r') as f: long_desc = f.read() extra = {} if PY3: extra["use_2to3"] = True setup(install_requires=['setuptools', 'six'], name='fs', version=VERSION, description="Filesystem abstraction layer", long_description=long_desc, license="BSD", author="Will McGugan", author_email="will@willmcgugan.com", url="http://pypi.python.org/pypi/fs/", platforms=['any'], packages=['fs', 'fs.expose', 'fs.expose.dokan', 'fs.expose.fuse', 'fs.expose.wsgi', 'fs.tests', 'fs.wrapfs', 'fs.osfs', 'fs.contrib', 'fs.contrib.bigfs', 'fs.contrib.davfs', 'fs.contrib.tahoelafs', 'fs.commands'], package_data={'fs': ['tests/data/*.txt']}, entry_points={"console_scripts": CONSOLE_SCRIPTS}, classifiers=classifiers, **extra )