43*>BsQj]v$J`q-Q^[1Fsi.5auKLh'E.' ٗy ٓ.:!,Uu )}?y m]]z ן܄ǿo{x} zK4&q̡3H<B@cO?*.pD.pmuڋUVNf oBa^ApJ?ѓ%1}22XDtm=ǖj~t]4AC%Mds_N1^?gQ+Q<{<M%hIQ^Y-5ӺӷISn D GO?p :ձT4s%u%PU Y6-( (.UYdl&敷ڴӛ+ϮrH#!b+)h3h5O mTCcT_Ӱlo/ yٛ%Ygߢ|}oEqOo
߿O?L#=L~g,?'g؟.)p=?7^y|*Z6m:v@uzt/0$K.HIZ2=e9l~SD[qgd1S* [2!Zʭ<ϩO(vv>"D?CyMEQ9VgOTi>'JyuauP6#WnNx F~ JTo-UʪW/Q%j#*ӕ&aGֲ9O$zTc$Zr[.;B7 ?
֛?3G>\:{F9}I@J8%Wے8.&1L$s i>z V1tcU673B [c*1|} l mZ[afmr `l|8 ڑFn)v~aٜU4VH |?Guc-4r8@!KWReR
91)(i q}8-fq-WF4tv\O;oMhk4GrtHt;O\ c2C&w8_<%u?0?1R7y?PV* 4y iw zyy i*CZU=AL#R(X
/R J
B qm)x}iGѳЉ(n zdž`spmw/SI*ֻU!6 IDAT*L;uYM`V)y;vw ( c`qMq _zO< ̳bY9 {#5c9D Dգ
/Gɑ~/>(~r$k(RH,(چ =?N 'n:gB A 4n*T p\O0y^}݇#A#rMNIۜYR(z.uѱh&w6Ce{,^=0c2fiT!99aWXx|ȤY}t9H-Vm 3yo*QŵꡘR7{ڼR ^c
d K97@ Wp865T te\*<<+n`rEiS
Dy'(z .aEYQZ,ÒzqA_NҨK)}HG&XK!LZUq8 Gz8idj=q\n;Z<kqٱFZ iZ .*@E
YԞ tBʠ+y1 bdekٰy_Q~wkϷ7BF!t1TVg?^Y%x `vЙ@ K
G~'"pUxkڒ AoB?ir=)81AS_d:*\2B
n
]QsP?CBh* <^GB#EߛbEg8jycEZI=sd~q۽nA;MIJby6sǾ/j9WoQu<=ۜir1<2$Ti%G sc4ǫ|8{ _
`+&Y`5鍬<<4uxyA`. g...t/p:dCYHdw'BGw -rEP!ꞌ}ت>Ӯ<螉ۗX"~$k
$7LM',u=`X]pb.ʔ%p gsnIFi%v0ITUIjҭCK
@ZTdvWX>#E}
<πu
1T#e<_" ]@9udR3=ʼ?g Bq qjCuR3l002Lگ<{ ޅMYzrri;q,)JMKj Ϟ_/x(!cH @}hϹHd2
ՋZڕ'2fE=]{o~[xi՞LEv"U#~ LP3fV!%
H Y4+T݈} ލ;^]U}DD6Dm5cQz{Zl ߣ g= R>g]"~{oTU]aGXG>RGyݲv<$%>$0t/ Tb8H7ϙrH$]!r3pJ)=-bݘHF
e 'Fi3pQX{Sbz,rabXȧ3?/+N뤴f Uy?{,@}Gg?=FxÕ}
q'0Or U}L_"%({bB| ѐȲ
n.]"=ɱ}HpFYs+-d?xcpsˠEB@] *Kb@y wp}}-Մ~*aY$YV^1~g=mI8]b:B0R
]"2[*u?.҂'V#/Q[F}TUʳ{p͗ `Hϟ˓
"`4əo!&^]1n輴@:NԚlG^-[Q0y̏bILkU:ǴJ4 >V`w&5ʜߋBg `ЧP R7j̗ؼwf I% y`r§x{,ڴ)2VEy;酊X=IӾ[H ΄93KZj5`^`Qah-Ү#k-fMp+ǰWȥ ? {_W H:eHb\u000&I
Յ$j R9ixP%(;i@+O%=yo~[#;\&d"2l#FsKU9<\.AAtbm-#/QnT6 ZjbÑMʷ.O
FjAmF5e
$}aq﹈ǯazrlaІ jx0ekɅfZI&y3nj)
4wHp s'Ϟ<
(B&EG^a٧1DyIV2 kg֒M*j#PɒsY$Tn5hh 3Z8mlXJ}$YoS:g oO?<8 j7Sm- 86+G$e {>_Sc /r~8IҺU Xڿ
Kbo!luOa oKcpUEބ|` qsgi+@!Oݲ _7X8"Ὧb7ƍ^k8ysPwYqP
H܌KE5[J,4Tj{,V7+D`sR(dщak q+(hʵ6*<[C_5V@k"zӎ/ޭQr
4 lCL~ j?#tΧE bܛxgpL}W].#7d k F SϥtG[G^jXVX?^3-8lM D;`}7- xkԚT v!N3Nr!awtIJodQfk/9P O%P@I {^kZ%
ңA[[pMǫܞWϓ ojKILwEJtuNݤjjL85 1ZPb"(ٳg|E/.UEYчRf@ȭ}Md ~֪&r=jqʳ\6T gS*p~'|_e? 甍"b<7#jwg f.Ct|֭~vdlRKjRvE1G-t> rg8썎 cy){T
CD Sbj`1"p>2PNiWnr ﱸs\[>Mvr%4Yiz`n LԒ" m5JZ|=lE$>ܿwK_>\sg @W=J[Nj]5 )2(鲑@ V9aUliU.ȹ?s4R haJ>k<9x **n'&WVJc蘬}ȢJ?y.V:#
eY].T<#-=9ٛ'<[$ yKyݛQ#o&گ 7}_2./a~n:m.-/QFxރr*D*bЇzؒX(ۃpZa%eA}$E'|_CG<; ;t'AW%#(U7Ey j1d&%ТvL람eds%|U7kIXvb<(hH!JԽ8e*5=m3=SB fU2
"ZE! | 4M2.` F6E5:w QuO8Ҙ*пM ( $Rd4@ ȍ02$ 8 w~{1}4ׇK鎡EpDY27°/"BT ORy6uV8hILXCz`+d h%#e5xȦ^7km6%`& FfDU'v ͦJbLq < p<[ L 8s8 qݢ5F6tXiE ns0[yok=ZMV;]owpy/MKc%te( 18a|Fm`{Yk?Koٻ_2Dc*Q 6)EͶo|c&]{z]97=c=a}:!_ 9QeZ1 Ɉ2 }k7UoN+Mĺ[b?v Po:eFI#`GA rPR}}{;/aI_[ [isoђk8|Ivϻ.Fa!zH:YU'"xSί
wq0 f0TH(8S7
ơN'DPr "|6nحLm&V}-U c
֪ s=UI']M% 7ϟ]e$*Gܠk栁 hW^S<"_ OL6VnKwT|x̓
ϒfmvbYCm7-|OzB
ZrgMiVTKh
6Hq oXi= ۔BV:]|KceKH Qrl(̒NbҐBq3Ef:r#$?3P]V?
uaTT@c+}[$#ޫ=IdlE1Sԍ*t]}ul F-$-a[4K|xqBDZ= iE(m
7 Ή
Lo~o G }Es뱯4!x^N? n \恻U"b i+ _:a ɮ!u4vz*Յ92,SJ'܍%Xg'
mLgaf=i*]aW [W-ᾦC |e زqd,iFDxv _7D0"r.*O|]6"F09nZԥRlLy|Zla+H"cvMD*76l
[JԢujلO
VNl)Eu,-5J Om#s2ɕёZ!iMOZ BtUPq֪rw1_yQzB:7GMJ䁰<,DK'o YnG\4%\ϼajc7ɥz:5s _~x" gk 'Gxx(cDip%Scxt(^Z!MCUF3E /Duk\j#f?í,;+c6vuz=nz9 ydέKQH' Me"Y h\N SX a읲:́M%*HM5JMz2ġ&֓퓥 7#?9}<5:B"h;aN$}Vnm[gVZpʢ_nwlnV٦H} C >fYPx|k_
yY+I6MJ9J#F87K&:sC|U2s>jQjдmȔ)5fTڣ*2AM)hs~BDӤ3fR3]MfQ>gA"AH&,", c"eر@b8$xb+IǞ=]Ţޱ͕F~ tWW=i ?S77gD5poJ4>Nާƚ B*p
V.oLJ\w}{q k+t%x+GQFV*h0{c/`4|+۬/܂Ujd=~r 82$ fZi:5xwޘ iaYA
~wed]W
E՚C6GsXt#0F
˳8M]wp`Yj\?'O>*1F6FьT)ZAT &ܮ M$P A|}Q&>;γ4aޣ]y ˲
_]9sIBF|PۻbѶo}3yi Y ū//!5E8Mq/݉%f[['?T |ٗ^N4*K!cUQu
^śR*ԝkߜ0$sEQ頉T\9gbۥ*Nd1eVvŃê%7=eJu)ԥ& gC %:Y.bKC` ASpwal3ښ V>3ܿ_~gy
V]XiUֳ
kOep
zTnnCt* V-i&K~(s٪V"](p%qJ~u?̅肏bzB 7+,N3cyB D.p rـ4FW
Yhc2Ft_O
VpwAcL
N~A[؈H_BCQ%H.x{ `@tQvb&sB\
۞-ԁyϹk9TxkԲ ! f'Gvq9+PQ+d\PA~ '%}9|KTgQ+BO-6YBw&QXC $Sڎ@4ZME تMuׂcjb YgW/ Ў:{
% R>C͉;, C\]3 x}xx
+A,&&#aŶ z {KčMo",K . ahJc oH%*ߛ2x<9 ܁hv<-h
!}DfD$D@0J+Cu `G
#\b`FJ ,
WM;?iA&^,xR؟
=У I!i ^3h\lQ iIg j1W 2RdI]o!Cdž"p0k߰Wc@bMDp^rUIR6n[E&IGz%vs*oN(Jޔ:Y5_[?u^kGm]A$\h4iCc\
TQfN+g~TΪF@MKCUX]dB5(jzvY$Q]eHF`Omxz7V5މ^cG-H")Ew)f}2 >nM3,ݧ~_}ۧ1M"A }q"d% j^IHD^_4]6f ] Hb%ۛfi0#T4'k LؠʂD*7"de(F q6~LfltJIGWY6Mm)#M
R \$!<0roΦi*ov}]1.{<yu vhjÐϲlʗ}txm___Ë/Z6.9 y,
!A*˷F>ONYhl6"xgpײ jJB{s=,2,59ٻYϣM>Q.s5zZ,>;\t
zb`0j{6)ړ:TA !AJkU E? ?} )\]@TAjH}ٳr>Z"fjp_+˸9B WLR'!,DDg+xj
}t :;20+AΕ.\] ŧ7՜DߓfMdݽ "M/lM\hfs6n)1:Jht
ҋ5`jY
V
b܌ՆŶ!ʻ(Ք5HE`Ͷ AZ]G<|$fsqP. y_fiѨ3E_ S.* C@~ϿTQLx5kItyfeV$ .U-Aj9ӏx}{$XUVS1NO/:
.S /gG cۯ95z%4X 3#,4&kӭʞqţX Ұ t/ _^ًA.^dA"ؠU.MA-pFýRZ f^:u]_{Cu
`x ;77ZXDo afr 5 LJ*bBZ] rSK2`r7oB6cHlrzT5KӍ# c7n]"E|>_O!#:76(+3,D 9\u|#jÈCR|ՠN˲xd*庮90gn\[
#Ϸ , NWpssSú
-
^Ͽxn~0-LRlxGh-ĹU !jhUWZig^|).Wi3hO^}ut>MG'x`9{|!% !_bGFVn O=JNir>t FE!n*O) nٯ<3O.ǥO9M^?6'ge*Ol 3 0BX,!'@F]gCyԃ_ [ٚKt l}]Cy ӏ
kRj=!"jrw*@r_ Xd)%m@ <}kºZV9/WW ȭ^^oRA" )mB8 k͜ N =6Rc%e/)9F/5A3tI6V "J.ih~]Vڜu2cd3-
wZr |>F|a퇙&m$3В\2@ sJ@Razai8: !7
9҂p>t* sz JD ¶$ñ6l=Ҏ`hܬ,K=x-.g#1&<=On-o"q}t
D!)P)∔ WÃ{P[y?ɉ68ZVo:' W/?RQrxvS6%%kV "8]'o?,
[(A AA.5= J* Cra[R` S0q
_*tjɥ.VtiaF@>3[M̨!UKes39,Ү o iLHQ#-&FPXj%Wy4L0%&˲Q ɫ9*+ X_ \ZL$dHX>ΔR I^6Ul+ nn^ޅ @ i(t 1b033s 9 >]{-Vo{L\
# WM۪6y ыEv,=ȏp8Tlٶ m
FRʂX{˾k_-"30A_CaZمkf
K>T F[Uɖ˽[ON^էlP F)Eᗖփ+5xCMv4 ->_`DKDUqHg G? Z *[K?o '0/vzvRY3,M!JM奬 m2xflUFFyY s>" Z t9 qN'_ooQѕ r p"p>e08 `AޟU7Y
TT)=FQ,Lk%6o 1Ã}ᇿ?@>6
3]4
I?/mx' ! >SA K -0bgϞRf w!DN^vpb:S^sm
OYc2Q56̽ d-mF2Đ1y.aRNsJ ^j9?@_ |-\N
P ZN;H"P?U)% eL^zLQ
4 ) . V<>.
~$q ,휧U^QQZ-|
R|Bc!e
H5PY"CRYc !VJ
k,s%Nqy.7D^U{9 }jݬߦMqpD? 0o OQ@8B c ~ڴnk#
jӚ v#k98i:uamXvr܇ug%a N?٫ln>#8Ϧ 6W[씤ڈ:.^#ZxAބzxGܸ<xv?b?FkhH:̧'Qо 3?8teV{wׁ /~paAUih< EOa;[OqiP >{$Z9<amc|--}x>~o?_z{))
ж8RJYfJv2UF'va<> f}P@
HK=GunŖ1a3e0F ghDImbT3Kƃ.~UI{I5xy@JzB
T)lx 4zR7% 1iJ{8 W!aӼǸѪ څGW&jqmצ\g"9xviͬ'PEQ%xW"s/e"܈ es.0DA{v^p 8<' IENDB` rdata-0.11.2/docs/_static/switcher.json 0000664 0000000 0000000 00000000427 14571337522 0020003 0 ustar 00root root 0000000 0000000 [
{
"name": "dev",
"version": "dev",
"url": "https://rdata.readthedocs.io/en/latest/"
},
{
"name": "0.10.0 (stable)",
"version": "stable",
"url": "https://rdata.readthedocs.io/en/stable/",
"preferred": true
}
] rdata-0.11.2/docs/_templates/ 0000775 0000000 0000000 00000000000 14571337522 0015764 5 ustar 00root root 0000000 0000000 rdata-0.11.2/docs/_templates/autosummary/ 0000775 0000000 0000000 00000000000 14571337522 0020352 5 ustar 00root root 0000000 0000000 rdata-0.11.2/docs/_templates/autosummary/base.rst 0000664 0000000 0000000 00000000150 14571337522 0022012 0 ustar 00root root 0000000 0000000 {{ objname | escape | underline}}
.. currentmodule:: {{ module }}
.. auto{{ objtype }}:: {{ objname }} rdata-0.11.2/docs/_templates/autosummary/class.rst 0000664 0000000 0000000 00000000753 14571337522 0022216 0 ustar 00root root 0000000 0000000 {{ objname | escape | underline}}
.. currentmodule:: {{ module }}
.. autoclass:: {{ objname }}
{% block methods %}
{% if methods %}
.. rubric:: {{ _('Methods') }}
.. autosummary::
{% for item in methods %}
{% if item != "__init__" %}
~{{ name }}.{{ item }}
{% endif %}
{%- endfor %}
{% endif %}
{% for item in methods %}
{% if item != "__init__" %}
.. automethod:: {{ item }}
{% endif %}
{%- endfor %}
{% endblock %} rdata-0.11.2/docs/apilist.rst 0000664 0000000 0000000 00000003515 14571337522 0016032 0 ustar 00root root 0000000 0000000 API List
========
List of functions and structures
--------------------------------
A complete list of all functions and structures provided by rdata.
Convenience functions
^^^^^^^^^^^^^^^^^^^^^
Functions that read and transform a `.rds` or `.rda` file, performing parsing and conversion with
one line of code.
.. autosummary::
:toctree: modules
rdata.read_rds
rdata.read_rda
Parse :code:`.rda` format
^^^^^^^^^^^^^^^^^^^^^^^^^
Functions for parsing data in the :code:`.rda` format. These functions return a structure representing
the contents of the file, without transforming it to more appropriate Python objects. Thus, if a different
way of converting R objects to Python objects is needed, it can be done from this structure.
.. autosummary::
:toctree: modules
rdata.parser.parse_file
rdata.parser.parse_data
Conversion of the R objects
^^^^^^^^^^^^^^^^^^^^^^^^^^^
These objects and functions convert the parsed R objects to appropriate Python objects. The Python object
corresponding to a R object is chosen to preserve most original properties, but it could change in the
future, if a more fitting Python object is found.
.. autosummary::
:toctree: modules
rdata.conversion.Converter
rdata.conversion.SimpleConverter
rdata.conversion.convert
rdata.conversion.DEFAULT_CLASS_MAP
Auxiliary structures
^^^^^^^^^^^^^^^^^^^^
These classes are used to represent R objects which have no clear analog in Python, so that the information
therein can be retrieved.
.. autosummary::
:toctree: modules
rdata.conversion.RBuiltin
rdata.conversion.RBytecode
rdata.conversion.RFunction
rdata.conversion.REnvironment
rdata.conversion.RExpression
rdata.conversion.RExternalPointer
rdata.conversion.RLanguage
rdata.conversion.SrcFile
rdata.conversion.SrcFileCopy
rdata.conversion.SrcRef
rdata-0.11.2/docs/conf.py 0000664 0000000 0000000 00000016313 14571337522 0015132 0 ustar 00root root 0000000 0000000 """Configuration of the Sphinx documentation."""
# rdata documentation build configuration file, created by
# sphinx-quickstart on Tue Aug 7 12:49:32 2018.
#
# This file is execfile()d with the current directory set to its
# containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
import importlib.metadata
import os
import sys
import textwrap
import rdata
# General information about the project.
project = "rdata"
author = "Carlos Ramos Carreño"
copyright = "2018, Carlos Ramos Carreño" # noqa: A001
github_url = "https://github.com/vnmabus/rdata"
rtd_version = os.environ.get("READTHEDOCS_VERSION")
rtd_version_type = os.environ.get("READTHEDOCS_VERSION_TYPE")
switcher_version = rtd_version
if switcher_version == "latest":
switcher_version = "dev"
elif rtd_version_type not in {"branch", "tag"}:
switcher_version = rdata.__version__
rtd_branch = os.environ.get(" READTHEDOCS_GIT_IDENTIFIER", "develop")
language = "en"
try:
release = importlib.metadata.version("rdata")
except importlib.metadata.PackageNotFoundError:
print( # noqa: T201
f"To build the documentation, The distribution information of\n"
f"{project} has to be available. Either install the package\n"
f"into your development environment or run 'setup.py develop'\n"
f"to setup the metadata. A virtualenv is recommended!\n",
)
sys.exit(1)
version = ".".join(release.split(".")[:2])
# -- General configuration ------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
"myst_parser",
"sphinx_codeautolink",
"sphinx.ext.autodoc",
"sphinx.ext.autosummary",
"sphinx.ext.intersphinx",
"sphinx.ext.mathjax",
"sphinx.ext.napoleon",
"sphinx.ext.todo",
"sphinx.ext.viewcode",
"sphinx_gallery.gen_gallery",
"jupyterlite_sphinx", # Move after sphinx gallery
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ["_templates"]
# The suffix(es) of source filenames.
source_suffix = ".rst"
# The master toctree document.
master_doc = "index"
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This patterns also effect to html_static_path and html_extra_path
exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = "sphinx"
add_module_names = False
# -- Options for HTML output ----------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = "pydata_sphinx_theme"
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#
html_theme_options = {
"use_edit_page_button": True,
"github_url": github_url,
"switcher": {
"json_url": (
"https://rdata.readthedocs.io/en/latest/_static/switcher.json"
),
"version_match": switcher_version,
},
"show_version_warning_banner": True,
"navbar_start": ["navbar-logo", "version-switcher"],
"icon_links": [
{
"name": "PyPI",
"url": "https://pypi.org/project/rdata",
"icon": "https://avatars.githubusercontent.com/u/2964877",
"type": "url",
},
{
"name": "Anaconda",
"url": "https://anaconda.org/conda-forge/rdata",
"icon": "https://avatars.githubusercontent.com/u/3571983",
"type": "url",
},
],
"logo": {
"text": "🗃 rdata",
},
}
html_context = {
"github_user": "vnmabus",
"github_repo": "rdata",
"github_version": "develop",
"doc_path": "docs",
}
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ["_static"]
# -- Options for LaTeX output ---------------------------------------------
latex_engine = "lualatex"
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, "rdata.tex",
"rdata Documentation", author, "manual"),
]
# -- Options for manual page output ---------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
(master_doc, "rdata", "rdata Documentation", [author], 1),
]
# -- Options for Texinfo output -------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(
master_doc,
"rdata",
"rdata documentation",
author,
"rdata",
"Read R datasets from Python.",
"Miscellaneous",
),
]
# -- Options for Epub output ----------------------------------------------
# Bibliographic Dublin Core info.
epub_title = project
epub_author = author
epub_publisher = author
epub_copyright = copyright
# A list of files that should not be packed into the epub file.
epub_exclude_files = ["search.html"]
# -- Options for "sphinx.ext.autodoc.typehints" --
autodoc_preserve_defaults = True
autodoc_typehints = "description"
# -- Options for "sphinx.ext.autosummary" --
autosummary_generate = True
# -- Options for "sphinx.ext.intersphinx" --
intersphinx_mapping = {
"igraph": ("https://python.igraph.org/en/stable/api", None),
"matplotlib": ("https://matplotlib.org/stable", None),
"numpy": ("https://numpy.org/doc/stable", None),
"pandas": ("https://pandas.pydata.org/pandas-docs/stable", None),
"python": (f"https://docs.python.org/{sys.version_info.major}", None),
"scipy": ("https://docs.scipy.org/doc/scipy", None),
"sklearn": ("https://scikit-learn.org/stable", None),
"xarray": ("http://xarray.pydata.org/en/stable/", None),
}
# -- Options for "sphinx.ext.todo" --
todo_include_todos = True
# -- Options for "sphinx_gallery.gen_gallery" --
sphinx_gallery_conf = {
"examples_dirs": ["../examples"],
"gallery_dirs": ["auto_examples"],
"reference_url": {
"rdata": None,
},
"doc_module": "rdata",
"jupyterlite": {
"use_jupyter_lab": True,
},
"first_notebook_cell": textwrap.dedent("""\
%pip install lzma
%pip install rdata
%pip install ipywidgets
import pyodide_http
pyodide_http.patch_all()
"""),
}
rdata-0.11.2/docs/contributors.md 0000664 0000000 0000000 00000000043 14571337522 0016703 0 ustar 00root root 0000000 0000000 ```{include} ../CONTRIBUTORS.md
``` rdata-0.11.2/docs/conversions.rst 0000664 0000000 0000000 00000010533 14571337522 0016733 0 ustar 00root root 0000000 0000000 Default conversions
===================
This page list the default conversions applied to R objects to convert them to
Python objects.
Basic types
-----------
The conversion of basic types is performed directly by the
:class:`~rdata.conversion.Converter` used.
Thus, changing the conversion for basic types currently requires creating a
custom :class:`~rdata.conversion.Converter` class.
The default :class:`~rdata.conversion.SimpleConverter` realizes the following
conversions:
================== ================================================================================================
R object type Python conversion
================== ================================================================================================
builtin function :class:`rdata.conversion.RBuiltin`.
bytecode :class:`rdata.conversion.RBytecode`.
char (internal) :class:`str` or :class:`bytes` (depending on the encoding flags).
closure :class:`rdata.conversion.RFunction`.
complex :class:`numpy.ndarray` with 128-bits complex dtype.
:class:`numpy.ma.MaskedArray` with 128-bits complex dtype if it contains NA values.
:class:`xarray.DataArray` if it contains labeled dimensions.
environment :class:`rdata.conversion.REnvironment`.
There are three special cases: the empty, base and global environments, which are
all empty by default. The base and global environments may be supplied to the
converter.
expression :class:`rdata.conversion.RExpression`.
external pointer :class:`rdata.conversion.RExternalPointer`.
integer :class:`numpy.ndarray` with 32-bits integer dtype.
:class:`numpy.ma.MaskedArray` with 32-bits integer dtype if it contains NA values.
:class:`xarray.DataArray` if it contains labeled dimensions.
language :class:`rdata.conversion.RLanguage`.
list :class:`list` (if untagged).
:class:`dict` (if tagged). Empty lists are considered tagged.
logical (boolean) :class:`numpy.ndarray` with boolean dtype.
:class:`numpy.ma.MaskedArray` with boolean dtype if it contains NA values.
:class:`xarray.DataArray` if it contains labeled dimensions.
missing argument :data:`NotImplemented`.
NULL :data:`None`.
real :class:`numpy.ndarray` with 64-bits floating point dtype.
:class:`numpy.ma.MaskedArray` with 64-bits floating point dtype if it contains NA values.
:class:`xarray.DataArray` if it contains labeled dimensions.
reference The referenced value, that is, an object already converted.
S4 object :class:`types.SimpleNamespace`.
special function :class:`rdata.conversion.RBuiltin`.
string :class:`numpy.ndarray` with suitable fixed-length string dtype.
symbol :class:`str`.
vector :class:`list` (if untagged).
:class:`dict` (if tagged). Empty lists are considered tagged.
================== ================================================================================================
Custom classes
--------------
In addition, objects containing a `"class"` attribute are passed to a "constructor function", if one is available.
A dictionary of constructor functions can be supplied to the converter, where the key of each element corresponds
to the class name.
When the `"class"` attribute contains several class names, these are tried in order.
The default constructor dictionary allows to convert the following R classes:
================== ================================================================================================
R class Python conversion
================== ================================================================================================
data.frame :class:`pandas.DataFrame`.
factor :class:`pandas.Categorical`.
ordered :class:`pandas.Categorical` (with ordered categories).
srcfile :class:`rdata.conversion.SrcFile`.
srcfilecopy :class:`rdata.conversion.SrcFileCopy`.
srcref :class:`rdata.conversion.SrcRef`.
ts :class:`pandas.Series`.
================== ================================================================================================
rdata-0.11.2/docs/index.rst 0000664 0000000 0000000 00000004654 14571337522 0015501 0 ustar 00root root 0000000 0000000 rdata version |version|
=======================
|build-status| |docs| |coverage| |pypi| |zenodo|
The package rdata offers a lightweight way to import R datasets/objects stored
in the ".rda" and ".rds" formats into Python.
Its main advantages are:
- It is a pure Python implementation, with no dependencies on the R language or
related libraries.
Thus, it can be used anywhere where Python is supported, including the web
using `Pyodide `_.
- It attempt to support all R objects that can be meaningfully translated.
As opposed to other solutions, you are no limited to import dataframes or
data with a particular structure.
- It allows users to easily customize the conversion of R classes to Python
ones.
Does your data use custom R classes?
Worry no longer, as it is possible to define custom conversions to the Python
classes of your choosing.
- It has a permissive license (MIT). As opposed to other packages that depend
on R libraries and thus need to adhere to the GPL license, you can use rdata
as a dependency on MIT, BSD or even closed source projects.
.. toctree::
:maxdepth: 4
:hidden:
:caption: Contents:
installation
simpleusage
apilist
auto_examples/index
Try online!
conversions
contributors
The package rdata is developed `on Github `_.
Please report `issues `_ there
as well.
.. |build-status| image:: https://github.com/vnmabus/rdata/actions/workflows/main.yml/badge.svg?branch=master
:alt: build status
:scale: 100%
:target: https://github.com/vnmabus/rdata/actions/workflows/main.yml
.. |docs| image:: https://readthedocs.org/projects/rdata/badge/?version=latest
:alt: Documentation Status
:scale: 100%
:target: https://rdata.readthedocs.io/en/latest/?badge=latest
.. |coverage| image:: http://codecov.io/github/vnmabus/rdata/coverage.svg?branch=develop
:alt: Coverage Status
:scale: 100%
:target: https://codecov.io/gh/vnmabus/rdata/branch/develop
.. |pypi| image:: https://badge.fury.io/py/rdata.svg
:alt: Pypi version
:scale: 100%
:target: https://pypi.python.org/pypi/rdata/
.. |zenodo| image:: https://zenodo.org/badge/DOI/10.5281/zenodo.6382237.svg
:alt: Zenodo DOI
:scale: 100%
:target: https://doi.org/10.5281/zenodo.6382237
rdata-0.11.2/docs/installation.rst 0000664 0000000 0000000 00000000701 14571337522 0017060 0 ustar 00root root 0000000 0000000 Installation
============
rdata is on PyPi and can be installed using :code:`pip`:
.. code::
pip install rdata
It is also available for :code:`conda` using the :code:`conda-forge` channel:
.. code::
conda install -c conda-forge rdata
Installing the develop version
------------------------------
The current version from the develop branch can be installed as
.. code::
pip install git+https://github.com/vnmabus/rdata.git@develop rdata-0.11.2/docs/make.bat 0000664 0000000 0000000 00000001451 14571337522 0015235 0 ustar 00root root 0000000 0000000 @ECHO OFF
pushd %~dp0
REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=.
set BUILDDIR=_build
set SPHINXPROJ=rdata
if "%1" == "" goto help
%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.http://sphinx-doc.org/
exit /b 1
)
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
goto end
:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
:end
popd
rdata-0.11.2/docs/simpleusage.rst 0000664 0000000 0000000 00000006521 14571337522 0016703 0 ustar 00root root 0000000 0000000 Simple usage
============
Read a R dataset
----------------
The common way of reading an R dataset is the following one:
.. code:: python
import rdata
converted = rdata.read_rda(rdata.TESTDATA_PATH / "test_vector.rda")
converted
which results in
.. code::
{'test_vector': array([1., 2., 3.])}
Under the hood, this is equivalent to the following code:
.. code:: python
import rdata
parsed = rdata.parser.parse_file(rdata.TESTDATA_PATH / "test_vector.rda")
converted = rdata.conversion.convert(parsed)
converted
This consists on two steps:
#. First, the file is parsed using the function
:func:`rdata.parser.parse_file`. This provides a literal description of the
file contents as a hierarchy of Python objects representing the basic R
objects. This step is unambiguous and always the same.
#. Then, each object must be converted to an appropriate Python object. In this
step there are several choices on which Python type is the most appropriate
as the conversion for a given R object. Thus, we provide a default
:func:`rdata.conversion.convert` routine, which tries to select Python
objects that preserve most information of the original R object. For custom
R classes, it is also possible to specify conversion routines to Python
objects.
Convert custom R classes
------------------------
The basic :func:`~rdata.conversion.convert` routine only constructs a
:class:`~rdata.conversion.SimpleConverter` object and calls its
:meth:`~rdata.conversion.SimpleConverter.convert` method. All arguments of
:func:`~rdata.conversion.convert` are directly passed to the
:class:`~rdata.conversion.SimpleConverter` initialization method.
It is possible, although not trivial, to make a custom
:class:`~rdata.conversion.Converter` object to change the way in which the
basic R objects are transformed to Python objects. However, a more common
situation is that one does not want to change how basic R objects are
converted, but instead wants to provide conversions for specific R classes.
This can be done by passing a dictionary to the
:class:`~rdata.conversion.SimpleConverter` initialization method, containing
as keys the names of R classes and as values, callables that convert a
R object of that class to a Python object. By default, the dictionary used
is :data:`~rdata.conversion.DEFAULT_CLASS_MAP`, which can convert
commonly used R classes such as
`data.frame `_
and `factor `_.
As an example, here is how we would implement a conversion routine for the
factor class to :class:`bytes` objects, instead of the default conversion to
Pandas :class:`~pandas.Categorical` objects:
.. code:: python
import rdata
def factor_constructor(obj, attrs):
values = [bytes(attrs['levels'][i - 1], 'utf8')
if i >= 0 else None for i in obj]
return values
new_dict = {
**rdata.conversion.DEFAULT_CLASS_MAP,
"factor": factor_constructor
}
converted = rdata.read_rda(
rdata.TESTDATA_PATH / "test_dataframe.rda",
constructor_dict=new_dict,
)
converted
which has the following result:
.. code::
{'test_dataframe': class value
1 b'a' 1
2 b'b' 2
3 b'b' 3} rdata-0.11.2/examples/ 0000775 0000000 0000000 00000000000 14571337522 0014515 5 ustar 00root root 0000000 0000000 rdata-0.11.2/examples/README.txt 0000664 0000000 0000000 00000000072 14571337522 0016212 0 ustar 00root root 0000000 0000000 Examples
========
Examples of the package functionality.
rdata-0.11.2/examples/__init__.py 0000664 0000000 0000000 00000000036 14571337522 0016625 0 ustar 00root root 0000000 0000000 """Documentation examples."""
rdata-0.11.2/examples/plot_cran.py 0000664 0000000 0000000 00000012426 14571337522 0017055 0 ustar 00root root 0000000 0000000 """
Loading a RDA file with custom types from CRAN
==============================================
A more advanced example showing how to read a dataset in the RDATA format from
the CRAN repository of R packages that include custom R types.
"""
# %%
# We will show how to load the graph of the classical
# `seven bridges of Königsberg problem
# `_ from the
# R package igraphdata.
#
# .. warning::
# This is for illustration purposes only. If you plan to use the same
# dataset repeatedly it is better to download it, or to use a package that
# caches it, such as
# `scikit-datasets `_.
#
# We will make use of the function
# :external+python:func:`urllib.request.urlopen` to load the url, as well as
# the package rdata.
# The package is a tar file so we need also to import the
# :external+python:mod:`tarfile` module.
# We will use the package `igraph `_ for
# constructing the graph in Python.
# Finally, we will import some plotting routines from Matplotlib.
import tarfile
from urllib.request import urlopen
import igraph
import igraph.drawing
import matplotlib.pyplot as plt
from matplotlib.colors import to_hex
import rdata
# %%
# The following URL contains the link to download the package from CRAN.
pkg_url = (
"https://cran.r-project.org/src/contrib/Archive/"
"igraphdata/igraphdata_1.0.0.tar.gz"
)
# %%
# The dataset is contained in the "data" folder, as it is common for
# R packages.
# The file is named Koenisberg and it is in the RDATA format
# (.rda extension).
data_path = "igraphdata/data/Koenigsberg.rda"
# %%
# We proceed to open the package using
# :external+python:func:`~urllib.request.urlopen`
# and :external+python:mod:`tarfile`.
with urlopen(pkg_url) as package:
with tarfile.open(fileobj=package, mode="r|gz") as package_tar:
for member in package_tar:
if member.name == data_path:
dataset = package_tar.extractfile(member)
assert dataset
with dataset:
parsed = rdata.parser.parse_file(dataset)
break
# %%
# We could try to convert this dataset to Python objects.
converted = rdata.conversion.convert(parsed)
print(converted)
# %%
# From this representation, we can see that .rda files contain a mapping
# of variable names to objects, and not just one object as .rds files.
# In this case there is just one variable called "Koenigsberg", as the
# dataset itself, but that is not necessarily always the case.
# %%
# We can also see that there is no default conversion for the "igraph"
# class, representing a graph.
# Thus, the converted object is a list of the underlying vectors used
# by this type.
# %%
# It is however possible to define our own conversion routines for R classes
# using the package rdata.
# For that purpose we need to create a "constructor" function, that accepts
# as arguments the underlying object to convert and its attributes, and
# returns the converted object.
# %%
# In this example, the object will be received as a list, corresponding to
# the `igraph_t structure defined by the igraph package
# `_.
# We will convert it to a :external+igraph:class:`~igraph.Graph` object from
# the `Python version of the igraph package
# `_.
# The attrs dict is empty and will not be used.
def graph_constructor(obj, attrs):
"""Construct graph object from R representation."""
n_vertices = int(obj[0][0])
is_directed = obj[1]
edge_from = obj[2].astype(int)
edge_to = obj[3].astype(int)
# output_edge_index = obj[4]
# input_edge_index = obj[5]
# output_vertex_edge_index = obj[6]
# input_vertex_edge_index = obj[7]
graph_attrs = obj[8][1]
vertex_attrs = obj[8][2]
edge_attrs = obj[8][3]
return igraph.Graph(
n=n_vertices,
directed=is_directed,
edges=list(zip(edge_from, edge_to)),
graph_attrs=graph_attrs,
vertex_attrs=vertex_attrs,
edge_attrs=edge_attrs,
)
# %%
# We create a dict with all the constructors that we want to apply.
# In this case, we include first the default constructors (which
# provide transformations for common R classes) and our newly created
# constructor.
# The key used for the dictionary entries should be the name of the
# corresponding R class.
constructor_dict = {
**rdata.conversion.DEFAULT_CLASS_MAP,
"igraph": graph_constructor,
}
# %%
# We can now call the :func:`rdata.conversion.convert` functtion, supplying
# the dictionary of constructors to use.
converted = rdata.conversion.convert(parsed, constructor_dict=constructor_dict)
# %%
# Finally, we check the constructed graph by plotting it using the
# :external+igraph:func:`igraph.drawing.plot` function.
fig, axes = plt.subplots()
plt.subplots_adjust(left=0, right=1, bottom=0, top=1)
igraph.drawing.plot(
converted["Koenigsberg"],
target=axes,
vertex_label=converted["Koenigsberg"].vs["name"],
vertex_label_size=8,
vertex_size=120,
vertex_color=to_hex("tab:blue"),
edge_label=converted["Koenigsberg"].es["name"],
edge_label_size=8,
)
plt.show()
rdata-0.11.2/examples/plot_example.py 0000664 0000000 0000000 00000001141 14571337522 0017555 0 ustar 00root root 0000000 0000000 # fmt: off
"""
R data loading
==============
Use the file uploader to convert files to Python.
"""
# sphinx_gallery_thumbnail_path = '_static/R_logo.svg'
from ipywidgets import FileUpload, interact
import rdata
@interact(files=FileUpload(accept="*.rd*", multiple=True))
def convert_from_file(files):
"""Open a rds or rdata file and display its contents as Python objects."""
for f in files:
parsed = rdata.parser.parse_data(f.content)
converted = rdata.conversion.convert(parsed)
for key, value in converted.items():
print(f"{key}:")
print(value)
rdata-0.11.2/examples/plot_zenodo.py 0000664 0000000 0000000 00000004607 14571337522 0017432 0 ustar 00root root 0000000 0000000 """
Loading a RDS file from a URL
=============================
A simple example showing how to read a dataset in the RDS format from a URL.
"""
# sphinx_gallery_thumbnail_path = '_static/download.png'
# %%
# If the data to read is accesible at a particular URL, we can open it as a
# file using the function :external+python:func:`urllib.request.urlopen`.
# Thus, we need to import that function as well as the rdata package.
from urllib.request import urlopen
import rdata
# %%
# For this example we will use a dataset hosted at
# `Zenodo `_.
# This is a small dataset containing information about some fungal pathogens.
#
dataset_url = (
"https://zenodo.org/records/7425539/files/core_fungal_pathogens.rds"
)
# %%
# The object resulting from calling
# :external+python:func:`~urllib.request.urlopen` can be then
# passed to :func:`~rdata.parser.parse_file` as if it were a normal file.
with urlopen(dataset_url) as dataset:
parsed = rdata.parser.parse_file(dataset)
# %%
# RDS files do not have a special magic number that identifies them.
# Thus, when reading a RDS file, rdata has to suppose that the file is a valid
# RDS file, and warns about that.
# We can omit this warning by passing manually the extension of the file
# instead.
with urlopen(dataset_url) as dataset:
parsed = rdata.parser.parse_file(dataset, extension=".rds")
# %%
# This parsed object contains a lossless representation of the internal data
# contained in the file.
# This data mimics the internal format used in R, and is thus not directly
# usable.
# However, we can retrieve some information about the file that will be lost
# after the conversion to a Python object, such as the version of the format
# employed or the encoding used for the strings.
print(parsed.versions.format)
print(parsed.extra.encoding)
# %%
# In order to convert it to Python objects we need to use the function
# :func:`rdata.conversion.convert`.
converted = rdata.conversion.convert(parsed)
# %%
# RDS files contain just one R object.
# In this particular case, it is a R dataframe object, that will be converted
# to a Pandas dataframe by default.
converted
# %%
# As usually we just want to parse and convert a given dataset, the convenience
# functions :func:`rdata.read_rds` and :func:`rdata.read_rda` can be used with
# that purpose.
with urlopen(dataset_url) as dataset:
data = rdata.read_rds(dataset)
data
rdata-0.11.2/pyproject.toml 0000664 0000000 0000000 00000006677 14571337522 0015633 0 ustar 00root root 0000000 0000000 [project]
name = "rdata"
description = "Read R datasets from Python."
readme = "README.rst"
requires-python = ">=3.9"
license = {file = "LICENSE"}
keywords = [
"rdata",
"r",
"dataset",
]
authors = [
{name = "Carlos Ramos Carreño", email = "vnmabus@gmail.com"},
]
maintainers = [
{name = "Carlos Ramos Carreño", email = "vnmabus@gmail.com"},
]
classifiers = [
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
"Intended Audience :: Science/Research",
"License :: OSI Approved :: MIT License",
"Natural Language :: English",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Topic :: File Formats",
"Topic :: Scientific/Engineering :: Mathematics",
"Topic :: Software Development :: Libraries :: Python Modules",
"Typing :: Typed",
]
dynamic = ["version"]
dependencies = [
"numpy",
"xarray",
"pandas",
"typing_extensions>4.4",
]
[project.optional-dependencies]
docs = [
"igraph",
"ipywidgets",
"jupyterlite-sphinx",
"jupyterlite-pyodide-kernel",
"matplotlib",
"myst-parser",
"pydata-sphinx-theme",
"sphinx>=3.1",
"sphinx-codeautolink",
"sphinx-gallery",
]
typing = [
"matplotlib>=3.8",
"mypy",
"pandas-stubs",
]
test = [
"pytest",
"pytest-cov",
"numpy>=1.14",
]
[project.urls]
homepage = "https://github.com/vnmabus/rdata"
documentation = "https://rdata.readthedocs.io"
repository = "https://github.com/vnmabus/rdata"
[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"
[tool.isort]
multi_line_output = 3
include_trailing_comma = true
use_parentheses = true
combine_as_imports = true
skip_glob = "**/plot_*.py plot_*.py"
[tool.mypy]
strict = true
strict_equality = true
implicit_reexport = true
[[tool.mypy.overrides]]
module = [
"igraph.*",
"ipywidgets.*",
]
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = "examples.*"
disallow_untyped_defs = false
[tool.pytest.ini_options]
addopts = "--doctest-modules --doctest-glob='*.rst'"
doctest_optionflags = "NORMALIZE_WHITESPACE ELLIPSIS"
norecursedirs = ".* build dist *.egg venv .svn _build docs/auto_examples examples asv_benchmarks"
[tool.ruff.lint]
select = [
"ALL",
]
ignore = [
"ANN101", # self does not need to be typed
"D212", # incompatible with D213, which is our preferred style for multiline docstrings
"Q003", # do not change quotation marks to avoid escaping
"PLC0414", # allow explicit re-exports
"S101", # assert is allowed
"TID252", # relative imports allowed
]
[tool.ruff.lint.per-file-ignores]
"plot_*.py" = [
"ANN", # no type hints in examples
"ARG001", # Some unused args are needed
"B018", # single object expressions are not useless in examples (they display the object)
"D205", # examples do not have a blank line in docstring
"D415", # first line in examples does not end with period
"ERA001", # Commented code may be useful for the reader
"S310", # URLs in examples have been validated
"T201", # print allowed in examples
]
"plot_cran.py" = [
"SIM117", # multiple with necessary for now
]
[tool.ruff.lint.isort]
combine-as-imports = true
[tool.ruff.lint.pydocstyle]
convention = "google"
[tool.ruff.lint.pylint]
max-args = 7
[tool.setuptools.packages.find]
include = ["rdata*"]
[tool.setuptools.dynamic]
version = {attr = "rdata.__version__"} rdata-0.11.2/rdata/ 0000775 0000000 0000000 00000000000 14571337522 0013772 5 ustar 00root root 0000000 0000000 rdata-0.11.2/rdata/__init__.py 0000664 0000000 0000000 00000001065 14571337522 0016105 0 ustar 00root root 0000000 0000000 """rdata: Read R datasets from Python."""
from __future__ import annotations
from importlib.resources import files
from typing import TYPE_CHECKING, Final
from . import conversion as conversion, parser as parser, testing as testing
from ._read import read_rda as read_rda, read_rds as read_rds
if TYPE_CHECKING:
from .parser._parser import Traversable
def _get_test_data_path() -> Traversable:
return files(__name__) / "tests" / "data"
TESTDATA_PATH: Final[Traversable] = _get_test_data_path()
"""
Path of the test data.
"""
__version__ = "0.11.2"
rdata-0.11.2/rdata/_read.py 0000664 0000000 0000000 00000016073 14571337522 0015425 0 ustar 00root root 0000000 0000000 """Functions to perform parsing and conversion in one step."""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
from .conversion._conversion import DEFAULT_CLASS_MAP, ConstructorDict, convert
from .parser._parser import (
DEFAULT_ALTREP_MAP,
AcceptableFile,
AltRepConstructorMap,
Traversable,
parse_file,
)
if TYPE_CHECKING:
import os
from collections.abc import MutableMapping
def read_rdata( # noqa: PLR0913
file_or_path: AcceptableFile | os.PathLike[Any] | Traversable | str,
*,
expand_altrep: bool = True,
altrep_constructor_dict: AltRepConstructorMap = DEFAULT_ALTREP_MAP,
extension: str | None = None,
constructor_dict: ConstructorDict = DEFAULT_CLASS_MAP,
default_encoding: str | None = None,
force_default_encoding: bool = False,
global_environment: MutableMapping[str, Any] | None = None,
base_environment: MutableMapping[str, Any] | None = None,
) -> Any: # noqa: ANN401
parsed = parse_file(
file_or_path=file_or_path,
expand_altrep=expand_altrep,
altrep_constructor_dict=altrep_constructor_dict,
extension=extension,
)
return convert(
parsed,
constructor_dict=constructor_dict,
default_encoding=default_encoding,
force_default_encoding=force_default_encoding,
global_environment=global_environment,
base_environment=base_environment,
)
def read_rds( # noqa: PLR0913
file_or_path: AcceptableFile | os.PathLike[Any] | Traversable | str,
*,
expand_altrep: bool = True,
altrep_constructor_dict: AltRepConstructorMap = DEFAULT_ALTREP_MAP,
constructor_dict: ConstructorDict = DEFAULT_CLASS_MAP,
default_encoding: str | None = None,
force_default_encoding: bool = False,
global_environment: MutableMapping[str, Any] | None = None,
base_environment: MutableMapping[str, Any] | None = None,
) -> Any: # noqa: ANN401
"""
Read an RDS file, containing an R object.
This is a convenience function that wraps :func:`rdata.parser.parse_file`
and :func:`rdata.parser.convert`, as it is the common use case.
Args:
file_or_path: File in the RDS format.
expand_altrep: Whether to translate ALTREPs to normal objects.
altrep_constructor_dict: Dictionary mapping each ALTREP to
its constructor.
constructor_dict: Dictionary mapping names of R classes to constructor
functions with the following prototype:
.. code-block :: python
def constructor(obj, attrs):
...
This dictionary can be used to support custom R classes. By
default, the dictionary used is
:data:`~rdata.conversion._conversion.DEFAULT_CLASS_MAP`
which has support for several common classes.
default_encoding: Default encoding used for strings with unknown
encoding. If `None`, the one stored in the file will be used, or
ASCII as a fallback.
force_default_encoding:
Use the default encoding even if the strings specify other
encoding.
global_environment: Global environment to use. By default is an empty
environment.
base_environment: Base environment to use. By default is an empty
environment.
Returns:
Contents of the file converted to a Python object.
See Also:
:func:`read_rda`: Similar function that parses a RDA or RDATA file.
Examples:
Parse one of the included examples, containing a dataframe
>>> import rdata
>>>
>>> data = rdata.read_rds(
... rdata.TESTDATA_PATH / "test_dataframe.rds"
... )
>>> data
class value
1 a 1
2 b 2
3 b 3
"""
return read_rdata(
file_or_path=file_or_path,
expand_altrep=expand_altrep,
altrep_constructor_dict=altrep_constructor_dict,
extension=".rds",
constructor_dict=constructor_dict,
default_encoding=default_encoding,
force_default_encoding=force_default_encoding,
global_environment=global_environment,
base_environment=base_environment,
)
def read_rda( # noqa: PLR0913
file_or_path: AcceptableFile | os.PathLike[Any] | Traversable | str,
*,
expand_altrep: bool = True,
altrep_constructor_dict: AltRepConstructorMap = DEFAULT_ALTREP_MAP,
constructor_dict: ConstructorDict = DEFAULT_CLASS_MAP,
default_encoding: str | None = None,
force_default_encoding: bool = False,
global_environment: MutableMapping[str, Any] | None = None,
base_environment: MutableMapping[str, Any] | None = None,
) -> dict[str, Any]:
"""
Read an RDA or RDATA file, containing an R object.
This is a convenience function that wraps :func:`rdata.parser.parse_file`
and :func:`rdata.parser.convert`, as it is the common use case.
Args:
file_or_path: File in the RDA format.
expand_altrep: Whether to translate ALTREPs to normal objects.
altrep_constructor_dict: Dictionary mapping each ALTREP to
its constructor.
constructor_dict: Dictionary mapping names of R classes to constructor
functions with the following prototype:
.. code-block :: python
def constructor(obj, attrs):
...
This dictionary can be used to support custom R classes. By
default, the dictionary used is
:data:`~rdata.conversion._conversion.DEFAULT_CLASS_MAP`
which has support for several common classes.
default_encoding: Default encoding used for strings with unknown
encoding. If `None`, the one stored in the file will be used, or
ASCII as a fallback.
force_default_encoding:
Use the default encoding even if the strings specify other
encoding.
global_environment: Global environment to use. By default is an empty
environment.
base_environment: Base environment to use. By default is an empty
environment.
Returns:
Contents of the file converted to a Python object.
See Also:
:func:`read_rds`: Similar function that parses a RDS file.
Examples:
Parse one of the included examples, containing a dataframe
>>> import rdata
>>>
>>> data = rdata.read_rda(
... rdata.TESTDATA_PATH / "test_dataframe.rda"
... )
>>> data
{'test_dataframe': class value
1 a 1
2 b 2
3 b 3}
"""
return read_rdata( # type: ignore[no-any-return]
file_or_path=file_or_path,
expand_altrep=expand_altrep,
altrep_constructor_dict=altrep_constructor_dict,
extension=".rda",
constructor_dict=constructor_dict,
default_encoding=default_encoding,
force_default_encoding=force_default_encoding,
global_environment=global_environment,
base_environment=base_environment,
)
rdata-0.11.2/rdata/conversion/ 0000775 0000000 0000000 00000000000 14571337522 0016157 5 ustar 00root root 0000000 0000000 rdata-0.11.2/rdata/conversion/__init__.py 0000664 0000000 0000000 00000001546 14571337522 0020276 0 ustar 00root root 0000000 0000000 """Utilities for converting R objects to Python ones."""
from ._conversion import (
DEFAULT_CLASS_MAP as DEFAULT_CLASS_MAP,
Converter as Converter,
RBuiltin as RBuiltin,
RBytecode as RBytecode,
REnvironment as REnvironment,
RExpression as RExpression,
RExternalPointer as RExternalPointer,
RFunction as RFunction,
RLanguage as RLanguage,
SimpleConverter as SimpleConverter,
SrcFile as SrcFile,
SrcFileCopy as SrcFileCopy,
SrcRef as SrcRef,
convert as convert,
convert_array as convert_array,
convert_attrs as convert_attrs,
convert_char as convert_char,
convert_list as convert_list,
convert_symbol as convert_symbol,
convert_vector as convert_vector,
dataframe_constructor as dataframe_constructor,
factor_constructor as factor_constructor,
ts_constructor as ts_constructor,
)
rdata-0.11.2/rdata/conversion/_conversion.py 0000664 0000000 0000000 00000063424 14571337522 0021066 0 ustar 00root root 0000000 0000000 from __future__ import annotations
import abc
import warnings
from collections import ChainMap
from collections.abc import Callable, Mapping, MutableMapping, Sequence
from dataclasses import dataclass
from fractions import Fraction
from types import MappingProxyType, SimpleNamespace
from typing import Any, Final, NamedTuple, Union, cast
import numpy as np
import pandas as pd
import xarray
from typing_extensions import override
from .. import parser
ConversionFunction = Callable[[Union[parser.RData, parser.RObject]], Any]
class RLanguage(NamedTuple):
"""R language construct."""
elements: list[Any]
attributes: Mapping[str, Any]
class RExpression(NamedTuple):
"""R expression."""
elements: list[RLanguage]
@dataclass
class RBuiltin:
"""R builtin."""
name: str
@dataclass
class RFunction:
"""R function."""
environment: Mapping[str, Any]
formals: Mapping[str, Any] | None
body: RLanguage
attributes: Mapping[str, Any]
@property
def source(self) -> str:
return "\n".join(self.attributes["srcref"].srcfile.lines)
@dataclass
class RExternalPointer:
"""R bytecode."""
protected: Any
tag: Any
@dataclass
class RBytecode:
"""R bytecode."""
code: xarray.DataArray
constants: Sequence[Any]
attributes: Mapping[str, Any]
class REnvironment(ChainMap[str, Any]):
"""R environment."""
def __init__(
self,
*maps: MutableMapping[str, Any],
frame: Mapping[str, Any] | None = None,
) -> None:
super().__init__(*maps)
self.frame = frame
def convert_list(
r_list: parser.RObject,
conversion_function: ConversionFunction,
) -> Mapping[str, Any] | list[Any]:
"""
Expand a tagged R pairlist to a Python dictionary.
Args:
r_list: Pairlist R object, with tags.
conversion_function: Conversion function to apply to the elements of
the list. By default is the identity function.
Returns:
A dictionary with the tags of the pairwise list as keys and their
corresponding values as values.
See Also:
convert_vector
"""
if r_list.info.type is parser.RObjectType.NILVALUE:
return {}
if r_list.info.type not in {
parser.RObjectType.LIST,
parser.RObjectType.LANG,
}:
msg = "Must receive a LIST, LANG or NILVALUE object"
raise TypeError(msg)
tag = None if r_list.tag is None else conversion_function(r_list.tag)
cdr = conversion_function(r_list.value[1])
if tag is not None:
if cdr is None:
cdr = {}
return {tag: conversion_function(r_list.value[0]), **cdr}
if cdr is None:
cdr = []
return [conversion_function(r_list.value[0]), *cdr]
def convert_env(
r_env: parser.RObject,
conversion_function: ConversionFunction,
) -> REnvironment:
"""Convert environment objects."""
if r_env.info.type is not parser.RObjectType.ENV:
msg = "Must receive a ENV object"
raise TypeError(msg)
frame = conversion_function(r_env.value.frame)
enclosure = conversion_function(r_env.value.enclosure)
hash_table = conversion_function(r_env.value.hash_table)
dictionary = {}
if hash_table is not None:
for d in hash_table:
if d is not None:
dictionary.update(d)
return REnvironment(dictionary, enclosure, frame=frame)
def convert_attrs(
r_obj: parser.RObject,
conversion_function: ConversionFunction,
) -> Mapping[str, Any]:
"""
Return the attributes of an object as a Python dictionary.
Args:
r_obj: R object.
conversion_function: Conversion function to apply to the elements of
the attribute list. By default is the identity function.
Returns:
A dictionary with the names of the attributes as keys and their
corresponding values as values.
See Also:
convert_list
"""
if r_obj.attributes:
attrs = cast(
Mapping[str, Any],
conversion_function(r_obj.attributes),
)
else:
attrs = {}
return attrs
def convert_vector(
r_vec: parser.RObject,
conversion_function: ConversionFunction,
attrs: Mapping[str, Any] | None = None,
) -> list[Any] | Mapping[str, Any]:
"""
Convert a R vector to a Python list or dictionary.
If the vector has a ``names`` attribute, the result is a dictionary with
the names as keys. Otherwise, the result is a Python list.
Args:
r_vec: R vector.
conversion_function: Conversion function to apply to the elements of
the vector. By default is the identity function.
attrs: Attributes of the vector.
Returns:
A dictionary with the ``names`` of the vector as keys and their
corresponding values as values. If the vector does not have an
argument ``names``, then a normal Python list is returned.
See Also:
convert_list
"""
if attrs is None:
attrs = {}
if r_vec.info.type not in {
parser.RObjectType.VEC,
parser.RObjectType.EXPR,
}:
msg = "Must receive a VEC or EXPR object"
raise TypeError(msg)
value: list[Any] | Mapping[str, Any] = [
conversion_function(o) for o in r_vec.value
]
# If it has the name attribute, use a dict instead
field_names = attrs.get("names")
if field_names is not None:
value = dict(zip(field_names, value))
return value
def safe_decode(byte_str: bytes, encoding: str) -> str | bytes:
"""Decode a (possibly malformed) string."""
try:
return byte_str.decode(encoding)
except UnicodeDecodeError as e:
warnings.warn( # noqa: B028
f"Exception while decoding {byte_str!r}: {e}",
)
return byte_str
def convert_char(
r_char: parser.RObject,
*,
default_encoding: str | None = None,
force_default_encoding: bool = False,
) -> str | bytes | None:
"""
Decode a R character array to a Python string or bytes.
The bits that signal the encoding are in the general pointer. The
string can be encoded in UTF8, LATIN1 or ASCII, or can be a sequence
of bytes.
Args:
r_char: R character array.
default_encoding: Default encoding to apply when encoding info
is not available.
force_default_encoding: Always use the default encoding.
Returns:
Decoded string.
See Also:
convert_symbol
"""
if r_char.info.type is not parser.RObjectType.CHAR:
msg = "Must receive a CHAR object"
raise TypeError(msg)
if r_char.value is None:
return None
assert isinstance(r_char.value, bytes)
encoding = None
if not force_default_encoding:
if r_char.info.gp & parser.CharFlags.UTF8:
encoding = "utf_8"
elif r_char.info.gp & parser.CharFlags.LATIN1:
encoding = "latin_1"
elif r_char.info.gp & parser.CharFlags.ASCII:
encoding = "ascii"
elif r_char.info.gp & parser.CharFlags.BYTES:
encoding = "bytes"
if encoding is None:
if default_encoding:
encoding = default_encoding
else:
# Assume ASCII if no encoding is marked
warnings.warn("Unknown encoding. Assumed ASCII.") # noqa: B028
encoding = "ascii"
return (
r_char.value
if encoding == "bytes"
else safe_decode(r_char.value, encoding)
)
def convert_symbol(
r_symbol: parser.RObject,
conversion_function: ConversionFunction,
) -> str | bytes:
"""
Decode a R symbol to a Python string or bytes.
Args:
r_symbol: R symbol.
conversion_function: Conversion function to apply to the char element
of the symbol. By default is the identity function.
Returns:
Decoded string.
See Also:
convert_char
"""
if r_symbol.info.type is parser.RObjectType.SYM:
symbol = conversion_function(r_symbol.value)
assert isinstance(symbol, str)
return symbol
msg = "Must receive a SYM object"
raise TypeError(msg)
def convert_array(
r_array: parser.RObject,
attrs: Mapping[str, Any] | None = None,
) -> np.ndarray[Any, Any] | xarray.DataArray:
"""
Convert a R array to a Numpy ndarray or a Xarray DataArray.
If the array has attribute ``dimnames`` the output will be a
Xarray DataArray, preserving the dimension names.
Args:
r_array: R array.
attrs: Attributes of the array.
Returns:
Array.
See Also:
convert_vector
"""
if attrs is None:
attrs = {}
if r_array.info.type not in {
parser.RObjectType.LGL,
parser.RObjectType.INT,
parser.RObjectType.REAL,
parser.RObjectType.CPLX,
}:
msg = "Must receive an array object"
raise TypeError(msg)
value = r_array.value
shape = attrs.get("dim")
if shape is not None:
# R matrix order is like FORTRAN
value = np.reshape(value, shape, order="F")
dimension_names = None
coords = None
dimnames = attrs.get("dimnames")
if dimnames:
if isinstance(dimnames, Mapping):
dimension_names = list(dimnames.keys())
coords = dimnames
else:
dimension_names = [f"dim_{i}" for i, _ in enumerate(dimnames)]
coords = {
dimension_names[i]: d
for i, d in enumerate(dimnames)
if d is not None
}
value = xarray.DataArray(
value,
dims=dimension_names,
coords=coords,
)
return value # type: ignore [no-any-return]
R_INT_MIN = -2**31
def _dataframe_column_transform(source: Any) -> Any: # noqa: ANN401
if isinstance(source, np.ndarray):
if np.issubdtype(source.dtype, np.integer):
return pd.Series(source, dtype=pd.Int32Dtype()).array
if np.issubdtype(source.dtype, np.bool_):
return pd.Series(source, dtype=pd.BooleanDtype()).array
if np.issubdtype(source.dtype, np.str_):
return pd.Series(source, dtype=pd.StringDtype()).array
return source
def dataframe_constructor(
obj: Mapping[str, Any],
attrs: Mapping[str, Any],
) -> pd.DataFrame:
row_names = attrs["row.names"]
obj = {key: _dataframe_column_transform(val) for key, val in obj.items()}
# Default row names are stored as [R_INT_NA, -len]
default_row_names_len = 2
index: pd.RangeIndex | tuple[str, ...] = (
pd.RangeIndex(1, abs(row_names[1]) + 1)
if (
len(row_names) == default_row_names_len
and isinstance(row_names, np.ma.MaskedArray)
and row_names.mask[0]
)
else tuple(row_names)
)
return pd.DataFrame(obj, columns=obj, index=index)
def _factor_constructor_internal(
obj: np.ndarray[Any, np.dtype[np.integer[Any]]],
attrs: Mapping[str, Any],
*,
ordered: bool,
) -> pd.Categorical:
values = [attrs["levels"][i - 1] if i >= 0 else None for i in obj]
return pd.Categorical(values, attrs["levels"], ordered=ordered)
def factor_constructor(
obj: np.ndarray[Any, np.dtype[np.integer[Any]]],
attrs: Mapping[str, Any],
) -> pd.Categorical:
"""Construct a factor objects."""
return _factor_constructor_internal(obj, attrs, ordered=False)
def ordered_constructor(
obj: np.ndarray[Any, np.dtype[np.integer[Any]]],
attrs: Mapping[str, Any],
) -> pd.Categorical:
"""Contruct an ordered factor."""
return _factor_constructor_internal(obj, attrs, ordered=True)
def ts_constructor(
obj: np.ndarray[Any, Any],
attrs: Mapping[str, Any],
) -> pd.Series[Any]:
"""Construct a time series object."""
start, end, frequency = attrs["tsp"]
frequency = int(frequency)
real_start = Fraction(int(round(start * frequency)), frequency)
real_end = Fraction(int(round(end * frequency)), frequency)
index: np.ndarray[Any, Any] = np.arange(
real_start,
real_end + Fraction(1, frequency),
Fraction(1, frequency),
)
if frequency == 1:
index = index.astype(int)
return pd.Series(obj, index=index)
@dataclass
class SrcRef:
"""Reference to a source file location."""
first_line: int
first_byte: int
last_line: int
last_byte: int
first_column: int
last_column: int
first_parsed: int
last_parsed: int
srcfile: SrcFile
def srcref_constructor(
obj: tuple[int, int, int, int, int, int, int, int],
attrs: Mapping[str, Any],
) -> SrcRef:
return SrcRef(*obj, srcfile=attrs["srcfile"])
@dataclass
class SrcFile:
"""Source file."""
filename: str
file_encoding: str | None
string_encoding: str | None
def srcfile_constructor(
obj: REnvironment,
attrs: Mapping[str, Any], # noqa: ARG001
) -> SrcFile:
frame = obj.frame
assert frame is not None
filename = frame["filename"][0]
file_encoding = frame.get("encoding")
string_encoding = frame.get("Enc")
return SrcFile(
filename=filename,
file_encoding=file_encoding,
string_encoding=string_encoding,
)
@dataclass
class SrcFileCopy(SrcFile):
"""Source file with a copy of its lines."""
lines: Sequence[str]
def srcfilecopy_constructor(
obj: REnvironment,
attrs: Mapping[str, Any], # noqa: ARG001
) -> SrcFileCopy:
frame = obj.frame
assert frame is not None
filename = frame["filename"][0]
file_encoding = frame.get("encoding", (None,))[0]
string_encoding = frame.get("Enc", (None,))[0]
lines = frame["lines"]
return SrcFileCopy(
filename=filename,
file_encoding=file_encoding,
string_encoding=string_encoding,
lines=lines,
)
Constructor = Callable[[Any, Mapping[str, Any]], Any]
ConstructorDict = Mapping[
Union[str, bytes],
Constructor,
]
default_class_map_dict: Final[ConstructorDict] = {
"data.frame": dataframe_constructor,
"factor": factor_constructor,
"ordered": ordered_constructor,
"ts": ts_constructor,
"srcref": srcref_constructor,
"srcfile": srcfile_constructor,
"srcfilecopy": srcfilecopy_constructor,
}
#: Default mapping of constructor functions.
DEFAULT_CLASS_MAP: Final = MappingProxyType(default_class_map_dict)
class Converter(abc.ABC):
"""Interface of a class converting R objects in Python objects."""
@abc.abstractmethod
def convert(self, data: parser.RData | parser.RObject) -> Any: # noqa: ANN401
"""Convert a R object to a Python one."""
@dataclass
class UnresolvedReference:
references: MutableMapping[int, Any]
index: int
class SimpleConverter(Converter):
"""
Class converting R objects to Python objects.
Args:
constructor_dict:
Dictionary mapping names of R classes to constructor functions with
the following prototype:
.. code-block :: python
def constructor(obj, attrs):
...
This dictionary can be used to support custom R classes. By
default, the dictionary used is
:data:`~rdata.conversion._conversion.DEFAULT_CLASS_MAP`
which has support for several common classes.
default_encoding:
Default encoding used for strings with unknown encoding. If `None`,
the one stored in the file will be used, or ASCII as a fallback.
force_default_encoding:
Use the default encoding even if the strings specify other
encoding.
global_environment: Global environment to use. By default is an empty
environment.
base_environment: Base environment to use. By default is an empty
environment.
"""
def __init__(
self,
constructor_dict: ConstructorDict = DEFAULT_CLASS_MAP,
*,
default_encoding: str | None = None,
force_default_encoding: bool = False,
global_environment: MutableMapping[str, Any] | None = None,
base_environment: MutableMapping[str, Any] | None = None,
) -> None:
self.constructor_dict = constructor_dict
self.default_encoding = default_encoding
self.force_default_encoding = force_default_encoding
self.global_environment = REnvironment(
{} if global_environment is None
else global_environment,
)
self.base_environment = REnvironment(
{} if base_environment is None
else base_environment,
)
self.empty_environment: Mapping[str, Any] = REnvironment({})
self._reset()
def _reset(self) -> None:
self.references: MutableMapping[int, Any] = {}
self.default_encoding_used = self.default_encoding
@override
def convert(
self,
data: parser.RData | parser.RObject,
) -> Any:
self._reset()
return self._convert_next(data)
def _convert_next( # noqa: C901, PLR0912, PLR0915
self,
data: parser.RData | parser.RObject,
) -> Any: # noqa: ANN401
"""Convert a R object to a Python one."""
obj: parser.RObject
if isinstance(data, parser.RData):
obj = data.object
if self.default_encoding is None:
self.default_encoding_used = data.extra.encoding
else:
obj = data
attrs = convert_attrs(obj, self._convert_next)
reference_id = id(obj)
# Return the value if previously referenced
value: Any = self.references.get(id(obj))
if value is not None:
pass
if obj.info.type == parser.RObjectType.SYM:
# Return the internal string
value = convert_symbol(obj, self._convert_next)
elif obj.info.type == parser.RObjectType.LIST:
# Expand the list and process the elements
value = convert_list(obj, self._convert_next)
elif obj.info.type == parser.RObjectType.CLO:
assert obj.tag is not None
assert obj.attributes is not None
environment = self._convert_next(obj.tag)
formals = self._convert_next(obj.value[0])
body = self._convert_next(obj.value[1])
attributes = self._convert_next(obj.attributes)
value = RFunction(
environment=environment,
formals=formals,
body=body,
attributes=attributes,
)
elif obj.info.type == parser.RObjectType.ENV:
# Return a ChainMap of the environments
value = convert_env(obj, self._convert_next)
elif obj.info.type == parser.RObjectType.LANG:
# Expand the list and process the elements, returning a
# special object
rlanguage_list = convert_list(obj, self._convert_next)
assert isinstance(rlanguage_list, list)
attributes = self._convert_next(
obj.attributes,
) if obj.attributes else {}
value = RLanguage(rlanguage_list, attributes)
elif obj.info.type in {
parser.RObjectType.SPECIAL,
parser.RObjectType.BUILTIN,
}:
value = RBuiltin(name=obj.value.decode("ascii"))
elif obj.info.type == parser.RObjectType.CHAR:
# Return the internal string
value = convert_char(
obj,
default_encoding=self.default_encoding_used,
force_default_encoding=self.force_default_encoding,
)
elif obj.info.type in {
parser.RObjectType.LGL,
parser.RObjectType.INT,
parser.RObjectType.REAL,
parser.RObjectType.CPLX,
}:
# Return the internal array
value = convert_array(obj, attrs=attrs)
elif obj.info.type == parser.RObjectType.STR:
# Convert the internal strings
value = np.array([self._convert_next(o) for o in obj.value])
elif obj.info.type == parser.RObjectType.VEC:
# Convert the internal objects
value = convert_vector(obj, self._convert_next, attrs=attrs)
elif obj.info.type == parser.RObjectType.EXPR:
rexpression_list = convert_vector(
obj,
self._convert_next,
attrs=attrs,
)
assert isinstance(rexpression_list, list)
# Convert the internal objects returning a special object
value = RExpression(rexpression_list)
elif obj.info.type == parser.RObjectType.BCODE:
value = RBytecode(
code=self._convert_next(obj.value[0]),
constants=[self._convert_next(c) for c in obj.value[1]],
attributes=attrs,
)
elif obj.info.type == parser.RObjectType.EXTPTR:
value = RExternalPointer(
protected=self._convert_next(obj.value[0]),
tag=self._convert_next(obj.value[1]),
)
elif obj.info.type == parser.RObjectType.S4:
value = SimpleNamespace(**attrs)
elif obj.info.type == parser.RObjectType.BASEENV:
value = self.base_environment
elif obj.info.type == parser.RObjectType.EMPTYENV:
value = self.empty_environment
elif obj.info.type == parser.RObjectType.MISSINGARG:
value = NotImplemented
elif obj.info.type == parser.RObjectType.GLOBALENV:
value = self.global_environment
elif obj.info.type == parser.RObjectType.REF:
# Return the referenced value
value = self.references.get(id(obj.referenced_object))
if value is None:
reference_id = id(obj.referenced_object)
assert obj.referenced_object is not None
self.references[reference_id] = UnresolvedReference(
self.references,
reference_id,
)
value = self._convert_next(obj.referenced_object)
elif obj.info.type == parser.RObjectType.NILVALUE:
value = None
else:
msg = f"Type {obj.info.type} not implemented"
raise NotImplementedError(msg)
if obj.info.object and attrs is not None:
classname = attrs.get("class", ())
for i, c in enumerate(classname):
constructor = self.constructor_dict.get(c, None)
new_value = (
constructor(value, attrs)
if constructor
else NotImplemented
)
if new_value is NotImplemented:
missing_msg = (
f"Missing constructor for R class \"{c}\". "
)
if len(classname) > (i + 1):
solution_msg = (
f"The constructor for class "
f"\"{classname[i+1]}\" will be "
f"used instead."
)
else:
solution_msg = (
"The underlying R object is "
"returned instead."
)
warnings.warn(
missing_msg + solution_msg,
stacklevel=1,
)
else:
value = new_value
break
self.references[reference_id] = value
return value
def convert(
data: parser.RData | parser.RObject,
constructor_dict: ConstructorDict = DEFAULT_CLASS_MAP,
*,
default_encoding: str | None = None,
force_default_encoding: bool = False,
global_environment: MutableMapping[str, Any] | None = None,
base_environment: MutableMapping[str, Any] | None = None,
) -> Any: # noqa: ANN401
"""
Use the default converter (:func:`SimpleConverter`) to convert the data.
Args:
data: Parsed data.
constructor_dict: Dictionary mapping names of R classes to constructor
functions with the following prototype:
.. code-block :: python
def constructor(obj, attrs):
...
This dictionary can be used to support custom R classes. By
default, the dictionary used is
:data:`~rdata.conversion._conversion.DEFAULT_CLASS_MAP`
which has support for several common classes.
default_encoding: Default encoding used for strings with unknown
encoding. If `None`, the one stored in the file will be used, or
ASCII as a fallback.
force_default_encoding:
Use the default encoding even if the strings specify other
encoding.
global_environment: Global environment to use. By default is an empty
environment.
base_environment: Base environment to use. By default is an empty
environment.
Examples:
Parse one of the included examples, containing a vector
>>> import rdata
>>>
>>> parsed = rdata.parser.parse_file(
... rdata.TESTDATA_PATH / "test_vector.rda")
>>> converted = rdata.conversion.convert(parsed)
>>> converted
{'test_vector': array([1., 2., 3.])}
Parse another example, containing a dataframe
>>> import rdata
>>>
>>> parsed = rdata.parser.parse_file(
... rdata.TESTDATA_PATH / "test_dataframe.rda")
>>> converted = rdata.conversion.convert(parsed)
>>> converted
{'test_dataframe': class value
1 a 1
2 b 2
3 b 3}
"""
return SimpleConverter(
constructor_dict=constructor_dict,
default_encoding=default_encoding,
force_default_encoding=force_default_encoding,
global_environment=global_environment,
base_environment=base_environment,
).convert(data)
rdata-0.11.2/rdata/parser/ 0000775 0000000 0000000 00000000000 14571337522 0015266 5 ustar 00root root 0000000 0000000 rdata-0.11.2/rdata/parser/__init__.py 0000664 0000000 0000000 00000000466 14571337522 0017405 0 ustar 00root root 0000000 0000000 """Utilities for parsing a rdata file."""
from ._parser import (
DEFAULT_ALTREP_MAP as DEFAULT_ALTREP_MAP,
CharFlags as CharFlags,
RData as RData,
RObject as RObject,
RObjectInfo as RObjectInfo,
RObjectType as RObjectType,
parse_data as parse_data,
parse_file as parse_file,
)
rdata-0.11.2/rdata/parser/_ascii.py 0000664 0000000 0000000 00000006127 14571337522 0017075 0 ustar 00root root 0000000 0000000 from __future__ import annotations
import io
from typing import Any
import numpy as np
import numpy.typing as npt
from ._parser import R_INT_NA, AltRepConstructorMap, Parser
class ParserASCII(Parser):
"""Parser for data in ASCII format."""
def __init__(
self,
data: memoryview,
*,
expand_altrep: bool,
altrep_constructor_dict: AltRepConstructorMap,
) -> None:
super().__init__(
expand_altrep=expand_altrep,
altrep_constructor_dict=altrep_constructor_dict,
)
self.file = io.TextIOWrapper(io.BytesIO(data), encoding="ascii")
def _readline(self) -> str:
r"""Read a line without trailing \n."""
return self.file.readline()[:-1]
def _parse_array_values(
self,
dtype: npt.DTypeLike,
length: int,
) -> npt.NDArray[Any]:
array = np.empty(length, dtype=dtype)
value: int | float | complex
for i in range(length):
line = self._readline()
if np.issubdtype(dtype, np.integer):
value = R_INT_NA if line == "NA" else int(line)
elif np.issubdtype(dtype, np.floating):
value = float(line)
elif np.issubdtype(dtype, np.complexfloating):
line2 = self._readline()
value = complex(float(line), float(line2))
else:
msg = f"Unknown dtype: {dtype}"
raise ValueError(msg)
array[i] = value
return array
def parse_string(self, length: int) -> bytes:
# Non-ascii characters in strings are written using octal byte codes,
# for example, a string 'aä' (2 chars) in UTF-8 is written as an ascii
# string r'a\303\244' (9 chars). We want to transform this to a byte
# string b'a\303\244' (3 bytes) corresponding to the byte
# representation of the original UTF-8 string.
# Let's use this string as an example to go through the code below
# Read the ascii string
s = self._readline()
# Now s = r'a\303\244' (9 chars)
# Convert characters to bytes (all characters are ascii)
b = s.encode("ascii")
# Now b = br'a\303\244' (9 bytes)
# There is a special 'unicode_escape' encoding that does
# basically two things here:
# 1) interpret e.g. br'\303' (4 bytes) as a single byte b'\303'
# 2) decode so-transformed byte string to a string with latin1 encoding
s = b.decode("unicode_escape")
# Now s = 'aä' (3 chars)
# We don't really want the latter latin1 decoding step done by
# the previous line of code, so we undo it by encoding in latin1
# back to bytes
b = s.encode("latin1")
# Now b = b'a\303\244' (3 bytes)
# We return this byte representation here. Later in the code there
# will be the decoding step from b'a\303\244' to 'aä',
# that is, s = b.decode('utf8')
assert len(b) == length
return b
def check_complete(self) -> None:
assert self.file.read(1) == ""
rdata-0.11.2/rdata/parser/_parser.py 0000664 0000000 0000000 00000113377 14571337522 0017307 0 ustar 00root root 0000000 0000000 from __future__ import annotations
import abc
import bz2
import enum
import gzip
import lzma
import os
import pathlib
import warnings
from collections.abc import Callable, Iterator, Mapping, Sequence
from dataclasses import dataclass
from types import MappingProxyType
from typing import (
TYPE_CHECKING,
Any,
Final,
Protocol,
Union,
runtime_checkable,
)
import numpy as np
import numpy.typing as npt
if TYPE_CHECKING:
from ._ascii import ParserASCII
from ._xdr import ParserXDR
#: Value used to represent a missing integer in R.
R_INT_NA: Final = -2**31
@runtime_checkable
class BinaryFileLike(Protocol):
"""Protocol for binary files."""
def read(self) -> bytes:
"""Read the contents of the file."""
@runtime_checkable
class BinaryBufferFileLike(Protocol):
"""Protocol for binary files."""
@property
def buffer(self) -> BinaryFileLike:
"""Get the underlying buffer."""
AcceptableFile = Union[BinaryFileLike, BinaryBufferFileLike]
try:
from importlib.resources.abc import Traversable as Traversable
except ImportError:
@runtime_checkable
class Traversable(Protocol): # type: ignore [no-redef]
"""Definition of Traversable protocol for Python < 3.11."""
def iterdir(self) -> Iterator[Traversable]:
pass
def read_bytes(self) -> bytes:
pass
def read_text(self, encoding: str | None = None) -> str:
pass
def is_dir(self) -> bool:
pass
def is_file(self) -> bool:
pass
def joinpath(
self,
*descendants: str | os.PathLike[str],
) -> Traversable:
pass
def __truediv__(
self,
child: str | os.PathLike[str],
) -> Traversable:
pass
def open(
self,
mode: str = "r",
) -> AcceptableFile:
pass
def name(self) -> str:
pass
class FileTypes(enum.Enum):
"""Type of file containing a R file."""
bzip2 = "bz2"
gzip = "gzip"
xz = "xz"
rdata_binary_v2 = "rdata version 2 (binary)"
rdata_binary_v3 = "rdata version 3 (binary)"
rdata_ascii_v2 = "rdata version 2 (ascii)"
rdata_ascii_v3 = "rdata version 3 (ascii)"
magic_dict = {
FileTypes.bzip2: b"\x42\x5a\x68",
FileTypes.gzip: b"\x1f\x8b",
FileTypes.xz: b"\xFD7zXZ\x00",
FileTypes.rdata_binary_v2: b"RDX2\n",
FileTypes.rdata_binary_v3: b"RDX3\n",
FileTypes.rdata_ascii_v2: b"RDA2\n",
FileTypes.rdata_ascii_v3: b"RDA3\n",
}
def file_type(data: memoryview) -> FileTypes | None:
"""Return the type of the file."""
for filetype, magic in magic_dict.items():
if data[:len(magic)] == magic:
return filetype
return None
class RdataFormats(enum.Enum):
"""Format of a R file."""
XDR = "XDR"
ASCII = "ASCII"
ASCII_CRLF = "ASCII_CRLF"
binary = "binary"
format_dict: Final = MappingProxyType({
RdataFormats.XDR: b"X\n",
RdataFormats.ASCII: b"A\n",
RdataFormats.ASCII_CRLF: b"A\r\n",
RdataFormats.binary: b"B\n",
})
def rdata_format(data: memoryview) -> RdataFormats | None:
"""Return the format of the data."""
for format_type, magic in format_dict.items():
if data[:len(magic)] == magic:
return format_type
return None
class RObjectType(enum.Enum):
"""Type of a R object."""
NIL = 0 # NULL
SYM = 1 # symbols
LIST = 2 # pairlists
CLO = 3 # closures
ENV = 4 # environments
PROM = 5 # promises
LANG = 6 # language objects
SPECIAL = 7 # special functions
BUILTIN = 8 # builtin functions
CHAR = 9 # internal character strings
LGL = 10 # logical vectors
INT = 13 # integer vectors
REAL = 14 # numeric vectors
CPLX = 15 # complex vectors
STR = 16 # character vectors
DOT = 17 # dot-dot-dot object
ANY = 18 # make “any” args work
VEC = 19 # list (generic vector)
EXPR = 20 # expression vector
BCODE = 21 # byte code
EXTPTR = 22 # external pointer
WEAKREF = 23 # weak reference
RAW = 24 # raw vector
S4 = 25 # S4 classes not of simple type
ALTREP = 238 # Alternative representations
ATTRLIST = 239 # Bytecode attribute
ATTRLANG = 240 # Bytecode attribute
BASEENV = 241 # Base environment
EMPTYENV = 242 # Empty environment
BCREPREF = 243 # Bytecode repetition reference
BCREPDEF = 244 # Bytecode repetition definition
MISSINGARG = 251 # Missinf argument
GLOBALENV = 253 # Global environment
NILVALUE = 254 # NIL value
REF = 255 # Reference
BYTECODE_SPECIAL_SET: Final = frozenset((
RObjectType.BCODE,
RObjectType.BCREPREF,
RObjectType.BCREPDEF,
RObjectType.LANG,
RObjectType.LIST,
RObjectType.ATTRLANG,
RObjectType.ATTRLIST,
))
class CharFlags(enum.IntFlag):
"""Flags for R objects of type char."""
HAS_HASH = 1
BYTES = 1 << 1
LATIN1 = 1 << 2
UTF8 = 1 << 3
CACHED = 1 << 5
ASCII = 1 << 6
@dataclass
class RVersions:
"""R versions."""
format: int
serialized: int
minimum: int
@dataclass
class RExtraInfo:
"""
Extra information.
Contains the default encoding (only in version 3).
"""
encoding: str | None = None
@dataclass
class RObjectInfo:
"""Internal attributes of a R object."""
type: RObjectType
object: bool
attributes: bool
tag: bool
gp: int
reference: int
def _str_internal( # noqa: PLR0912, C901
obj: RObject | Sequence[RObject],
indent: int = 0,
used_references: set[int] | None = None,
) -> str:
if used_references is None:
used_references = set()
small_indent = indent + 2
big_indent = indent + 4
indent_spaces = " " * indent
small_indent_spaces = " " * small_indent
big_indent_spaces = " " * big_indent
string = ""
if isinstance(obj, Sequence):
string += f"{indent_spaces}[\n"
for elem in obj:
string += _str_internal(
elem,
big_indent,
used_references.copy(),
)
string += f"{indent_spaces}]\n"
return string
string += f"{indent_spaces}{obj.info.type}\n"
if obj.tag:
tag_string = _str_internal(
obj.tag,
big_indent,
used_references.copy(),
)
string += f"{small_indent_spaces}tag:\n{tag_string}\n"
if obj.info.reference:
assert obj.referenced_object
reference_string = (
f"{big_indent_spaces}..."
if obj.info.reference in used_references
else _str_internal(
obj.referenced_object,
indent + 4, used_references.copy())
)
string += (
f"{small_indent_spaces}reference: "
f"{obj.info.reference}\n{reference_string}\n"
)
string += f"{small_indent_spaces}value:\n"
if isinstance(obj.value, RObject):
string += _str_internal(
obj.value,
big_indent,
used_references.copy(),
)
elif isinstance(obj.value, (tuple, list)):
for elem in obj.value:
string += _str_internal(
elem,
big_indent,
used_references.copy(),
)
elif isinstance(obj.value, np.ndarray):
max_displayed_elements: Final = 4
string += big_indent_spaces
if len(obj.value) > max_displayed_elements:
string += (
f"[{obj.value[0]}, {obj.value[1]} ... "
f"{obj.value[-2]}, {obj.value[-1]}]\n"
)
else:
string += f"{obj.value}\n"
else:
string += f"{big_indent_spaces}{obj.value}\n"
if obj.attributes:
attr_string = _str_internal(
obj.attributes,
big_indent,
used_references.copy(),
)
string += f"{small_indent_spaces}attributes:\n{attr_string}\n"
return string
@dataclass
class RObject:
"""Representation of a R object."""
info: RObjectInfo
value: Any
attributes: RObject | None
tag: RObject | None = None
referenced_object: RObject | None = None
def __str__(self) -> str:
return _str_internal(self)
@dataclass
class RData:
"""Data contained in a R file."""
versions: RVersions
extra: RExtraInfo
object: RObject
def __str__(self) -> str:
return (
"RData(\n"
f" versions: {self.versions}\n"
f" extra: {self.extra}\n"
f" object: \n{_str_internal(self.object, indent=4)}\n"
")\n"
)
@dataclass
class EnvironmentValue:
"""Value of an environment."""
locked: bool
enclosure: RObject
frame: RObject
hash_table: RObject
AltRepConstructor = Callable[
[RObject],
tuple[RObjectInfo, Any],
]
AltRepConstructorMap = Mapping[bytes, AltRepConstructor]
def format_float_with_scipen(number: float, scipen: int) -> bytes:
"""Format a floating point value as in R."""
fixed = np.format_float_positional(number, trim="-")
scientific = np.format_float_scientific(number, trim="-")
assert isinstance(fixed, str)
assert isinstance(scientific, str)
return (
scientific if len(fixed) - len(scientific) > scipen
else fixed
).encode()
def deferred_string_constructor(
state: RObject,
) -> tuple[RObjectInfo, Any]:
"""Expand a deferred string ALTREP."""
new_info = RObjectInfo(
type=RObjectType.STR,
object=False,
attributes=False,
tag=False,
gp=0,
reference=0,
)
object_to_format = state.value[0].value
scipen = state.value[1].value
value = [
RObject(
info=RObjectInfo(
type=RObjectType.CHAR,
object=False,
attributes=False,
tag=False,
gp=CharFlags.ASCII,
reference=0,
),
value=format_float_with_scipen(num, scipen),
attributes=None,
tag=None,
referenced_object=None,
)
for num in object_to_format
]
return new_info, value
def compact_seq_constructor(
state: RObject,
*,
is_int: bool = False,
) -> tuple[RObjectInfo, Any]:
"""Expand a compact_seq ALTREP."""
new_info = RObjectInfo(
type=RObjectType.INT if is_int else RObjectType.REAL,
object=False,
attributes=False,
tag=False,
gp=0,
reference=0,
)
n = int(state.value[0])
start = state.value[1]
step = state.value[2]
if is_int:
start = int(start)
step = int(step)
# Calculate stop with integer arithmetic
# and use built-in range() for numerical stability
stop = start + (n - 1) * step
value = np.array(range(start, stop + 1, step))
else:
# Calculate stop with floating-point arithmetic
stop = start + (n - 1) * step
value = np.linspace(start, stop, n)
return new_info, value
def compact_intseq_constructor(
state: RObject,
) -> tuple[RObjectInfo, Any]:
"""Expand a compact_intseq ALTREP."""
return compact_seq_constructor(state, is_int=True)
def compact_realseq_constructor(
state: RObject,
) -> tuple[RObjectInfo, Any]:
"""Expand a compact_realseq ALTREP."""
return compact_seq_constructor(state, is_int=False)
def wrap_constructor(
state: RObject,
) -> tuple[RObjectInfo, Any]:
"""Expand any wrap_* ALTREP."""
new_info = RObjectInfo(
type=state.value[0].info.type,
object=False,
attributes=False,
tag=False,
gp=0,
reference=0,
)
value = state.value[0].value
return new_info, value
default_altrep_map_dict: Final[Mapping[bytes, AltRepConstructor]] = {
b"deferred_string": deferred_string_constructor,
b"compact_intseq": compact_intseq_constructor,
b"compact_realseq": compact_realseq_constructor,
b"wrap_real": wrap_constructor,
b"wrap_string": wrap_constructor,
b"wrap_logical": wrap_constructor,
b"wrap_integer": wrap_constructor,
b"wrap_complex": wrap_constructor,
b"wrap_raw": wrap_constructor,
}
DEFAULT_ALTREP_MAP: Final = MappingProxyType(default_altrep_map_dict)
class Parser(abc.ABC):
"""Parser interface for a R file."""
def __init__(
self,
*,
expand_altrep: bool = True,
altrep_constructor_dict: AltRepConstructorMap = DEFAULT_ALTREP_MAP,
) -> None:
self.expand_altrep = expand_altrep
self.altrep_constructor_dict = altrep_constructor_dict
def _parse_array(
self,
dtype: npt.DTypeLike,
) -> npt.NDArray[Any]:
"""Parse an array composed of an integer (array size) and values."""
length = self.parse_int()
return self._parse_array_values(dtype, length)
@abc.abstractmethod
def _parse_array_values(
self,
dtype: npt.DTypeLike,
length: int,
) -> npt.NDArray[Any]:
"""Parse values of an array."""
def parse_bool(self) -> bool:
"""Parse a boolean."""
return bool(self.parse_int())
def parse_int(self) -> int:
"""Parse an integer."""
return int(self._parse_array_values(np.int32, 1)[0])
def parse_nullable_bool_array(
self,
*,
fill_value: bool = True,
) -> npt.NDArray[np.bool_] | np.ma.MaskedArray[Any, Any]:
"""Parse a boolean array."""
return self.parse_nullable_int_array(
fill_value=fill_value,
).astype(np.bool_)
def parse_nullable_int_array(
self,
*,
fill_value: int = R_INT_NA,
) -> npt.NDArray[np.int32] | np.ma.MaskedArray[Any, Any]:
"""Parse an integer array."""
data = self._parse_array(np.int32)
mask = (data == R_INT_NA)
data[mask] = fill_value
if np.any(mask):
return np.ma.array( # type: ignore [no-untyped-call,no-any-return]
data=data,
mask=mask,
fill_value=fill_value,
)
return data
def parse_double_array(self) -> npt.NDArray[np.float64]:
"""Parse a double array."""
return self._parse_array(np.float64)
def parse_complex_array(self) -> npt.NDArray[np.complex128]:
"""Parse a complex array."""
return self._parse_array(np.complex128)
@abc.abstractmethod
def parse_string(self, length: int) -> bytes:
"""Parse a string."""
def check_complete(self) -> None:
"""Check that parsing was completed."""
return
def parse_all(self) -> RData:
"""Parse all the file."""
versions = self.parse_versions()
extra_info = self.parse_extra_info(versions)
obj = self.parse_R_object()
return RData(versions, extra_info, obj)
def parse_versions(self) -> RVersions:
"""Parse the versions header."""
format_version = self.parse_int()
r_version = self.parse_int()
minimum_r_version = self.parse_int()
if format_version not in {2, 3}:
msg = f"Format version {format_version} unsupported"
raise NotImplementedError(msg)
return RVersions(format_version, r_version, minimum_r_version)
def parse_extra_info(self, versions: RVersions) -> RExtraInfo:
"""
Parse the extra info.
Parses the encoding in version 3 format.
"""
encoding = None
minimum_version_with_encoding = 3
if versions.format >= minimum_version_with_encoding:
encoding_len = self.parse_int()
encoding = self.parse_string(encoding_len).decode("ASCII")
return RExtraInfo(encoding)
def expand_altrep_to_object(
self,
info: RObject,
state: RObject,
) -> tuple[RObjectInfo, Any]:
"""Expand alternative representation to normal object."""
assert info.info.type == RObjectType.LIST
class_sym = info.value[0]
while class_sym.info.type == RObjectType.REF:
class_sym = class_sym.referenced_object
assert class_sym.info.type == RObjectType.SYM
assert class_sym.value.info.type == RObjectType.CHAR
altrep_name = class_sym.value.value
assert isinstance(altrep_name, bytes)
constructor = self.altrep_constructor_dict[altrep_name]
return constructor(state)
def _parse_bytecode_constant(
self,
reference_list: list[RObject] | None,
bytecode_rep_list: list[RObject | None] | None = None,
) -> RObject:
obj_type = self.parse_int()
return self.parse_R_object(
reference_list,
bytecode_rep_list,
info_int=obj_type,
)
def _parse_bytecode(
self,
reference_list: list[RObject] | None,
bytecode_rep_list: list[RObject | None] | None = None,
) -> tuple[RObject, Sequence[RObject]]:
"""Parse R bytecode."""
if bytecode_rep_list is None:
n_repeated = self.parse_int()
code = self.parse_R_object(reference_list, bytecode_rep_list)
if bytecode_rep_list is None:
bytecode_rep_list = [None] * n_repeated
n_constants = self.parse_int()
constants = [
self._parse_bytecode_constant(
reference_list,
bytecode_rep_list,
)
for _ in range(n_constants)
]
return (code, constants)
def parse_R_object( # noqa: N802, C901, PLR0912, PLR0915
self,
reference_list: list[RObject] | None = None,
bytecode_rep_list: list[RObject | None] | None = None,
info_int: int | None = None,
) -> RObject:
"""Parse a R object."""
if reference_list is None:
# Index is 1-based, so we insert a dummy object
reference_list = []
original_info_int = info_int
if (
info_int is not None
and RObjectType(info_int) in BYTECODE_SPECIAL_SET
):
info = parse_r_object_info(info_int)
info.tag = info.type not in {
RObjectType.BCREPREF,
RObjectType.BCODE,
}
else:
info_int = self.parse_int()
info = parse_r_object_info(info_int)
tag = None
attributes = None
referenced_object = None
bytecode_rep_position = -1
tag_read = False
attributes_read = False
add_reference = False
result = None
value: Any
if info.type == RObjectType.BCREPDEF:
assert bytecode_rep_list
bytecode_rep_position = self.parse_int()
info.type = RObjectType(self.parse_int())
if info.type == RObjectType.NIL:
value = None
elif info.type == RObjectType.SYM:
# Read Char
value = self.parse_R_object(reference_list, bytecode_rep_list)
# Symbols can be referenced
add_reference = True
elif info.type in {
RObjectType.LIST,
RObjectType.LANG,
RObjectType.CLO,
RObjectType.PROM,
RObjectType.DOT,
RObjectType.ATTRLANG,
}:
if info.type is RObjectType.ATTRLANG:
info.type = RObjectType.LANG
info.attributes = True
tag = None
if info.attributes:
attributes = self.parse_R_object(
reference_list,
bytecode_rep_list,
)
attributes_read = True
if info.tag:
tag = self.parse_R_object(reference_list, bytecode_rep_list)
tag_read = True
# Read CAR and CDR
car = self.parse_R_object(
reference_list,
bytecode_rep_list,
info_int=(
None if original_info_int is None
else self.parse_int()
),
)
cdr = self.parse_R_object(
reference_list,
bytecode_rep_list,
info_int=(
None if original_info_int is None
else self.parse_int()
),
)
value = (car, cdr)
elif info.type == RObjectType.ENV:
info.object = True
result = RObject(
info=info,
tag=tag,
attributes=attributes,
value=None,
referenced_object=referenced_object,
)
reference_list.append(result)
locked = self.parse_bool()
enclosure = self.parse_R_object(reference_list, bytecode_rep_list)
frame = self.parse_R_object(reference_list, bytecode_rep_list)
hash_table = self.parse_R_object(reference_list, bytecode_rep_list)
attributes = self.parse_R_object(reference_list, bytecode_rep_list)
value = EnvironmentValue(
locked=locked,
enclosure=enclosure,
frame=frame,
hash_table=hash_table,
)
elif info.type in {RObjectType.SPECIAL, RObjectType.BUILTIN}:
length = self.parse_int()
if length > 0:
value = self.parse_string(length=length)
elif info.type == RObjectType.CHAR:
length = self.parse_int()
if length > 0:
value = self.parse_string(length=length)
elif length == 0:
value = b""
elif length == -1:
value = None
else:
msg = f"Length of CHAR cannot be {length}"
raise NotImplementedError(msg)
elif info.type == RObjectType.LGL:
value = self.parse_nullable_bool_array()
elif info.type == RObjectType.INT:
value = self.parse_nullable_int_array()
elif info.type == RObjectType.REAL:
value = self.parse_double_array()
elif info.type == RObjectType.CPLX:
value = self.parse_complex_array()
elif info.type in {
RObjectType.STR,
RObjectType.VEC,
RObjectType.EXPR,
}:
length = self.parse_int()
value = [None] * length
for i in range(length):
value[i] = self.parse_R_object(
reference_list, bytecode_rep_list)
elif info.type == RObjectType.BCODE:
value = self._parse_bytecode(reference_list, bytecode_rep_list)
tag_read = True
elif info.type == RObjectType.EXTPTR:
result = RObject(
info=info,
tag=tag,
attributes=attributes,
value=None,
referenced_object=referenced_object,
)
reference_list.append(result)
protected = self.parse_R_object(
reference_list,
bytecode_rep_list,
)
extptr_tag = self.parse_R_object(
reference_list,
bytecode_rep_list,
)
value = (protected, extptr_tag)
elif info.type == RObjectType.S4:
value = None
elif info.type == RObjectType.ALTREP:
altrep_info = self.parse_R_object(
reference_list,
bytecode_rep_list,
)
altrep_state = self.parse_R_object(
reference_list,
bytecode_rep_list,
)
altrep_attr = self.parse_R_object(
reference_list,
bytecode_rep_list,
)
if self.expand_altrep:
info, value = self.expand_altrep_to_object(
info=altrep_info,
state=altrep_state,
)
attributes = altrep_attr
else:
value = (altrep_info, altrep_state, altrep_attr)
elif info.type == RObjectType.BASEENV: # noqa: SIM114
value = None
elif info.type == RObjectType.EMPTYENV:
value = None
elif info.type == RObjectType.BCREPREF:
assert bytecode_rep_list
position = self.parse_int()
result = bytecode_rep_list[position]
assert result
return result
elif info.type == RObjectType.MISSINGARG: # noqa: SIM114
value = None
elif info.type == RObjectType.GLOBALENV: # noqa: SIM114
value = None
elif info.type == RObjectType.NILVALUE:
value = None
elif info.type == RObjectType.REF:
value = None
# Index is 1-based
referenced_object = reference_list[info.reference - 1]
else:
msg = f"Type {info.type} not implemented"
raise NotImplementedError(msg)
if info.tag and not tag_read:
warnings.warn( # noqa: B028
f"Tag not implemented for type {info.type} "
"and ignored",
)
if info.attributes and not attributes_read:
attributes = self.parse_R_object(reference_list, bytecode_rep_list)
if result is None:
result = RObject(
info=info,
tag=tag,
attributes=attributes,
value=value,
referenced_object=referenced_object,
)
else:
result.info = info
result.attributes = attributes
result.value = value
result.referenced_object = referenced_object
if add_reference:
reference_list.append(result)
if bytecode_rep_position >= 0:
assert bytecode_rep_list
bytecode_rep_list[bytecode_rep_position] = result
return result
def parse_file(
file_or_path: AcceptableFile | os.PathLike[Any] | Traversable | str,
*,
expand_altrep: bool = True,
altrep_constructor_dict: AltRepConstructorMap = DEFAULT_ALTREP_MAP,
extension: str | None = None,
) -> RData:
"""
Parse a R file (.rda or .rdata).
Args:
file_or_path: File in the R serialization format.
expand_altrep: Whether to translate ALTREPs to normal objects.
altrep_constructor_dict: Dictionary mapping each ALTREP to
its constructor.
extension: Extension of the file.
Returns:
Data contained in the file (versions and object).
See Also:
:func:`parse_data`: Similar function that receives the data directly.
Examples:
Parse one of the included examples, containing a vector
>>> import rdata
>>>
>>> parsed = rdata.parser.parse_file(
... rdata.TESTDATA_PATH / "test_vector.rda")
>>> parsed
RData(versions=RVersions(format=2,
serialized=196610,
minimum=131840),
extra=RExtraInfo(encoding=None),
object=RObject(info=RObjectInfo(type=,
object=False,
attributes=False,
tag=True,
gp=0,
reference=0),
value=(RObject(info=RObjectInfo(type=,
object=False,
attributes=False,
tag=False,
gp=0,
reference=0),
value=array([1., 2., 3.]),
attributes=None,
tag=None,
referenced_object=None),
RObject(info=RObjectInfo(type=,
object=False,
attributes=False,
tag=False,
gp=0,
reference=0),
value=None,
attributes=None,
tag=None,
referenced_object=None)),
attributes=None,
tag=RObject(info=RObjectInfo(type=,
object=False,
attributes=False,
tag=False,
gp=0,
reference=0),
value=RObject(info=RObjectInfo(\
type=,
object=False,
attributes=False,
tag=False,
gp=64,
reference=0),
value=b'test_vector',
attributes=None,
tag=None,
referenced_object=None),
attributes=None,
tag=None,
referenced_object=None),
referenced_object=None))
"""
path = None
if isinstance(file_or_path, Traversable):
path = file_or_path
elif isinstance(file_or_path, (os.PathLike, str)):
path = pathlib.Path(file_or_path)
else:
# file is a pre-opened file
binary_file = (
file_or_path.buffer
if isinstance(file_or_path, BinaryBufferFileLike)
else file_or_path
)
data = binary_file.read()
if path is not None:
# file was a path-like
if extension is None:
extension = getattr(path, "suffix", None)
data = path.read_bytes()
return parse_data(
data,
expand_altrep=expand_altrep,
altrep_constructor_dict=altrep_constructor_dict,
extension=extension,
)
def parse_data(
data: bytes,
*,
expand_altrep: bool = True,
altrep_constructor_dict: AltRepConstructorMap = DEFAULT_ALTREP_MAP,
extension: str | None = None,
) -> RData:
"""
Parse the data of a R file, received as a sequence of bytes.
Args:
data: Data extracted of a R file.
expand_altrep: Whether to translate ALTREPs to normal objects.
altrep_constructor_dict: Dictionary mapping each ALTREP to
its constructor.
extension: Extension of the file.
Returns:
Data contained in the file (versions and object).
See Also:
:func:`parse_file`: Similar function that parses a file directly.
Examples:
Parse one of the included examples, containing a vector
>>> import rdata
>>>
>>> with open(rdata.TESTDATA_PATH / "test_vector.rda", "rb") as f:
... parsed = rdata.parser.parse_data(f.read())
>>>
>>> parsed
RData(versions=RVersions(format=2,
serialized=196610,
minimum=131840),
extra=RExtraInfo(encoding=None),
object=RObject(info=RObjectInfo(type=,
object=False,
attributes=False,
tag=True,
gp=0,
reference=0),
value=(RObject(info=RObjectInfo(type=,
object=False,
attributes=False,
tag=False,
gp=0,
reference=0),
value=array([1., 2., 3.]),
attributes=None,
tag=None,
referenced_object=None),
RObject(info=RObjectInfo(type=,
object=False,
attributes=False,
tag=False,
gp=0,
reference=0),
value=None,
attributes=None,
tag=None,
referenced_object=None)),
attributes=None,
tag=RObject(info=RObjectInfo(type=,
object=False,
attributes=False,
tag=False,
gp=0,
reference=0),
value=RObject(info=RObjectInfo(\
type=,
object=False,
attributes=False,
tag=False,
gp=64,
reference=0),
value=b'test_vector',
attributes=None,
tag=None,
referenced_object=None),
attributes=None,
tag=None,
referenced_object=None),
referenced_object=None))
"""
view = memoryview(data)
filetype = file_type(view)
parse_function = (
parse_rdata_binary
if filetype in {
FileTypes.rdata_binary_v2,
FileTypes.rdata_binary_v3,
FileTypes.rdata_ascii_v2,
FileTypes.rdata_ascii_v3,
None,
} else parse_data
)
if filetype is FileTypes.bzip2:
new_data = bz2.decompress(data)
elif filetype is FileTypes.gzip:
new_data = gzip.decompress(data)
elif filetype is FileTypes.xz:
new_data = lzma.decompress(data)
elif filetype in {FileTypes.rdata_binary_v2,
FileTypes.rdata_binary_v3,
FileTypes.rdata_ascii_v2,
FileTypes.rdata_ascii_v3,
}:
if extension == ".rds":
warnings.warn( # noqa: B028
f"Wrong extension {extension} for file in RDATA format",
)
view = view[len(magic_dict[filetype]):]
new_data = view
else:
new_data = view
if extension != ".rds":
warnings.warn("Unknown file type: assumed RDS") # noqa: B028
if extension not in {None, ".rds"}:
warnings.warn(f"Wrong extension {extension} for file in RDS format") # noqa: B028
return parse_function(
new_data, # type: ignore [arg-type]
expand_altrep=expand_altrep,
altrep_constructor_dict=altrep_constructor_dict,
extension=extension,
)
def parse_rdata_binary(
data: memoryview,
*,
expand_altrep: bool = True,
altrep_constructor_dict: AltRepConstructorMap = DEFAULT_ALTREP_MAP,
extension: str | None = None, # noqa: ARG001
) -> RData:
"""Select the appropiate parser and parse all the info."""
format_type = rdata_format(data)
if format_type:
data = data[len(format_dict[format_type]):]
Parser: type[ParserXDR | ParserASCII] # noqa: N806
if format_type is RdataFormats.XDR:
from ._xdr import ParserXDR as Parser
elif format_type in (RdataFormats.ASCII, RdataFormats.ASCII_CRLF):
from ._ascii import ParserASCII as Parser
else:
msg = "Unknown file format"
raise NotImplementedError(msg)
parser = Parser(
data,
expand_altrep=expand_altrep,
altrep_constructor_dict=altrep_constructor_dict,
)
r_data = parser.parse_all()
parser.check_complete()
return r_data
def bits(data: int, start: int, stop: int) -> int:
"""Read bits [start, stop) of an integer."""
count = stop - start
mask = ((1 << count) - 1) << start
bitvalue = data & mask
return bitvalue >> start
def is_special_r_object_type(r_object_type: RObjectType) -> bool:
"""Check if a R type has a different serialization than the usual one."""
return (
r_object_type is RObjectType.NILVALUE
or r_object_type is RObjectType.REF
)
def parse_r_object_info(info_int: int) -> RObjectInfo:
"""Parse the internal information of an object."""
type_exp = RObjectType(bits(info_int, 0, 8))
reference = 0
if is_special_r_object_type(type_exp):
object_flag = False
attributes = False
tag = False
gp = 0
else:
object_flag = bool(bits(info_int, 8, 9))
attributes = bool(bits(info_int, 9, 10))
tag = bool(bits(info_int, 10, 11))
gp = bits(info_int, 12, 28)
if type_exp == RObjectType.REF:
reference = bits(info_int, 8, 32)
return RObjectInfo(
type=type_exp,
object=object_flag,
attributes=attributes,
tag=tag,
gp=gp,
reference=reference,
)
rdata-0.11.2/rdata/parser/_xdr.py 0000664 0000000 0000000 00000002217 14571337522 0016576 0 ustar 00root root 0000000 0000000 from __future__ import annotations
import io
from typing import Any
import numpy as np
import numpy.typing as npt
from ._parser import AltRepConstructorMap, Parser
class ParserXDR(Parser):
"""Parser for data in XDR format."""
def __init__(
self,
data: memoryview,
*,
expand_altrep: bool,
altrep_constructor_dict: AltRepConstructorMap,
) -> None:
super().__init__(
expand_altrep=expand_altrep,
altrep_constructor_dict=altrep_constructor_dict,
)
self.file = io.BytesIO(data)
def _parse_array_values(
self,
dtype: npt.DTypeLike,
length: int,
) -> npt.NDArray[Any]:
dtype = np.dtype(dtype)
buffer = self.file.read(length * dtype.itemsize)
# Read in big-endian order and convert to native byte order
return np.frombuffer(
buffer,
dtype=dtype.newbyteorder(">"),
).astype(dtype, copy=False)
def parse_string(self, length: int) -> bytes:
return self.file.read(length)
def check_complete(self) -> None:
assert self.file.read(1) == b""
rdata-0.11.2/rdata/py.typed 0000664 0000000 0000000 00000000100 14571337522 0015460 0 ustar 00root root 0000000 0000000 # Marker file for PEP 561. The rdata package uses inline types. rdata-0.11.2/rdata/testing.py 0000664 0000000 0000000 00000002516 14571337522 0016025 0 ustar 00root root 0000000 0000000 """Utilities for testing with R files."""
from __future__ import annotations
import subprocess
import tempfile
from typing import Any, Protocol
R_CODE_PREFIX = """::: """
class HasDoc(Protocol):
"""Python object having a docstring."""
__doc__: str | None
def get_data_source(
function_or_class: HasDoc,
*,
prefix: str = R_CODE_PREFIX,
) -> str:
"""Get the part of the docstring containing the data source."""
doc = function_or_class.__doc__
if doc is None:
return ""
source = ""
for line in doc.splitlines(keepends=True):
stripped_line = line.lstrip()
if stripped_line.startswith(prefix):
source += stripped_line.removeprefix(prefix)
return source
def execute_r_data_source(
function_or_class: HasDoc,
*,
prefix: str = R_CODE_PREFIX,
**kwargs: Any, # noqa: ANN401
) -> None:
"""Execute R data source."""
source = get_data_source(
function_or_class,
prefix=prefix,
)
if not source:
return
inits = ""
for key, value in kwargs.items():
inits += f"{key} <- {value!r}\n"
source = inits + source
with tempfile.NamedTemporaryFile("w") as file:
file.write(source)
file.flush()
subprocess.check_call(
["Rscript", file.name], # noqa: S603, S607
)
rdata-0.11.2/rdata/tests/ 0000775 0000000 0000000 00000000000 14571337522 0015134 5 ustar 00root root 0000000 0000000 rdata-0.11.2/rdata/tests/__init__.py 0000664 0000000 0000000 00000000043 14571337522 0017242 0 ustar 00root root 0000000 0000000 """Tests for the rdata package."""
rdata-0.11.2/rdata/tests/data/ 0000775 0000000 0000000 00000000000 14571337522 0016045 5 ustar 00root root 0000000 0000000 rdata-0.11.2/rdata/tests/data/test_altrep_compact_intseq.rda 0000664 0000000 0000000 00000000176 14571337522 0024160 0 ustar 00root root 0000000 0000000 r0b```f`a`e`f2XCCt-X F'*I-.O))J-O-HL.+)N-ʾbd|*eYSb`q~d` b'c rdata-0.11.2/rdata/tests/data/test_altrep_compact_intseq_asymmetric.rda 0000664 0000000 0000000 00000000215 14571337522 0026407 0 ustar 00root root 0000000 0000000 r0b```f`afd`f2XCCt-X FN ZZ\SRZ[\WRZX\ZR T5*FeIJ,N!?N 5 = rdata-0.11.2/rdata/tests/data/test_altrep_compact_realseq.rda 0000664 0000000 0000000 00000000200 14571337522 0024275 0 ustar 00root root 0000000 0000000 r0b```f`a`e`f2XCCt-X F'.I-.O))J-O-HL./JM)N-JbdJYSb`q> 8 3ȩ rdata-0.11.2/rdata/tests/data/test_altrep_compact_realseq_asymmetric.rda 0000664 0000000 0000000 00000000220 14571337522 0026534 0 ustar 00root root 0000000 0000000 r0b```f`afd`f2XCCt-X FN VZ\SRZ[\_SZX\ZR TuFfIJ,N fvPc " k rdata-0.11.2/rdata/tests/data/test_altrep_deferred_string.rda 0000664 0000000 0000000 00000000256 14571337522 0024314 0 ustar 00root root 0000000 0000000 r0b```f`a`e`f2XCCt-X F'.I-.O))J-OIMK-*JM/.)KJbdJYSb`q bN `; Y` ʷHq {)Geu8# P; 5 rdata-0.11.2/rdata/tests/data/test_altrep_wrap_logical.rda 0000664 0000000 0000000 00000000175 14571337522 0023611 0 ustar 00root root 0000000 0000000 r0b```f`a`e`f2XCCt-X F'(I-.O))J-//J,OLNʽbd}ʈ1ne}Ē.Һt# rdata-0.11.2/rdata/tests/data/test_dataframe_rownames.rda 0000664 0000000 0000000 00000000330 14571337522 0023427 0 ustar 00root root 0000000 0000000 ]O
P $B@m!nD;~55.ysf epz {ƍ6,8T0s*NO>$ld 8p*b2&W$ś*<)L EZ46`^ݯ;aBd3IHjcI
Ff9 rdata-0.11.2/rdata/tests/data/test_dataframe_v3.rda 0000664 0000000 0000000 00000000272 14571337522 0022131 0 ustar 00root root 0000000 0000000 ]P
0 m" '~~xMNΫ?jڭ=$}/}/ =o8 0`ENyW j,%@%k#`dRlN鹋?g!(&s5 n>o% 55"4Q#YA+ Q>Ed+HߔYlwͦ/%AgT rdata-0.11.2/rdata/tests/data/test_dataframe_v3.rds 0000664 0000000 0000000 00000000244 14571337522 0022152 0 ustar 00root root 0000000 0000000 ]NI0 t6VBB x҅+ /I>8qf< @ ^/3͎D$} ciź!Lo:cM'⑊xmFteXcBXPvU ޓ%^&, rdata-0.11.2/rdata/tests/data/test_empty_function.rda 0000664 0000000 0000000 00000000503 14571337522 0022635 0 ustar 00root root 0000000 0000000 RKk@Fz h=xꥇ Kk M$@LB6KNY#xd ٙ}3 0uZh0"pz12s= Z⩗2 : ckJu dh ΏbA8})xМ{7qK|
R'gKg+ ^2R^/ABQM$_CSrN|9Q"^H#ɧIXgNJͮ oqȞ ҚpͧWRSj_!_iG*ls|j| ]H#FOKС rdata-0.11.2/rdata/tests/data/test_empty_function_uncompiled.rda 0000664 0000000 0000000 00000000417 14571337522 0025060 0 ustar 00root root 0000000 0000000 r0b```f`a@&khXH˕ħTƧ%d܂̜6f4 lEEi@Sy< 2և|F4@i@L5'3/@4N]Bu- KsSb#P'Tbj4+/fb`J!zH|h2'VtpU9b)F0cx&X
? rdata-0.11.2/rdata/tests/data/test_empty_str.rda 0000664 0000000 0000000 00000000110 14571337522 0021612 0 ustar 00root root 0000000 0000000 r0b```b`fcf`b2Y# '+I-.O-(/.) ɂ? {$
=yQ>V1-وJɢdnR(]ٙoN.Ϻn L m4>@Xz=f< C/aύ T>$mmt.hKx M 2F]q e5N;
y>pa1Ze
CibS'. rV7d JĴYh
.nJ
VQvԲ}{
hilen*gzYeϡnZUϞfU6P
ö|"#|Qh'$ eF^$_ PE Ϧ-;0ی]gÈq3o c&8ՔE.e%n\,Ks Q rdata-0.11.2/rdata/tests/data/test_half_named_matrix.rda 0000664 0000000 0000000 00000000230 14571337522 0023231 0 ustar 00root root 0000000 0000000 r0b```f`adb`f2XCCt-X FN -VZ\XRYe PJ@i(-@59%3H13) *\ d Ci<P gda 5T rdata-0.11.2/rdata/tests/data/test_list.rda 0000664 0000000 0000000 00000000147 14571337522 0020551 0 ustar 00root root 0000000 0000000 r0b```b`b&f H020piΒ G$ | `@ a$#ށE+.L Y(h rdata-0.11.2/rdata/tests/data/test_list_attrs.rda 0000664 0000000 0000000 00000000157 14571337522 0021767 0 ustar 00root root 0000000 0000000 r0b```b`faa`b2Y# '/I-.%%E@LhJs+r@ 0H,,14&2l/ HA@j rdata-0.11.2/rdata/tests/data/test_logical.rda 0000664 0000000 0000000 00000000111 14571337522 0021177 0 ustar 00root root 0000000 0000000 r0b```b`b&f H020pi , H? O rdata-0.11.2/rdata/tests/data/test_matrix.rda 0000664 0000000 0000000 00000000146 14571337522 0021101 0 ustar 00root root 0000000 0000000 r0b```b`b&f H020piĒ
*>0p PZJs@i )@l+ .G rdata-0.11.2/rdata/tests/data/test_minimal_function.rda 0000664 0000000 0000000 00000000423 14571337522 0023126 0 ustar 00root root 0000000 0000000 r0b```f`a@&khXHfe&ħ%d101)d+.J.JM qe3fNI2Y ̼֜b C IPlt`l
MP .48@%榢 bLPAn *1=5?YJBC 3 <@,cAr /i t\3`)ZQPZ\ bϼ
f ( y2 rdata-0.11.2/rdata/tests/data/test_minimal_function_uncompiled.rda 0000664 0000000 0000000 00000000331 14571337522 0025343 0 ustar 00root root 0000000 0000000 r0b```f`a@&khXH+fe&ħ%d܂̜6f4=lEEi@sy< *6|F4@i@L5'3/@4!R]_ sSbCP$Tb j4+/ l rdata-0.11.2/rdata/tests/data/test_na_string.rda 0000664 0000000 0000000 00000000116 14571337522 0021556 0 ustar 00root root 0000000 0000000 r0b```b`fcf`b2Y# '+I-.K/.)Ke8 RnE rdata-0.11.2/rdata/tests/data/test_named_matrix.rda 0000664 0000000 0000000 00000000241 14571337522 0022241 0 ustar 00root root 0000000 0000000 r0b```f`adb`f2XCCt-X FN -XZ\XRY` PJ@i(-@59%3H13) *\ d Ci<P gUÌ$jgda Oϳ rdata-0.11.2/rdata/tests/data/test_nullable_int.rda 0000664 0000000 0000000 00000000136 14571337522 0022244 0 ustar 00root root 0000000 0000000 r0b```f`afd`f2XCCt-X FN -XZ\WWd`0 U rdata-0.11.2/rdata/tests/data/test_nullable_logical.rda 0000664 0000000 0000000 00000000135 14571337522 0023063 0 ustar 00root root 0000000 0000000 r0b```f`afd`f2XCCt-X FN -ZZ\W)0 P&Y rdata-0.11.2/rdata/tests/data/test_s4.rda 0000664 0000000 0000000 00000000213 14571337522 0020116 0 ustar 00root root 0000000 0000000 r0b```b`faa`b2Y# 'f/I-./6a`dDbKMHblΉE9h*A
@B6 P59' ŴԢ 80@i(UsIqkq
V;w 0
ZXs0 2lf Ʈ rdata-0.11.2/rdata/tests/data/test_vector.rda 0000664 0000000 0000000 00000000116 14571337522 0021074 0 ustar 00root root 0000000 0000000 r0b```b`b&f H020pi" ?0` cQTR rdata-0.11.2/rdata/tests/test_rdata.py 0000664 0000000 0000000 00000054067 14571337522 0017654 0 ustar 00root root 0000000 0000000 """Tests of parsing and conversion."""
import itertools
import unittest
from collections import ChainMap
from fractions import Fraction
from types import SimpleNamespace
from typing import Any
import numpy as np
import pandas as pd
import pytest
import xarray
import rdata
TESTDATA_PATH = rdata.TESTDATA_PATH
class SimpleTests(unittest.TestCase):
"""Collection of simple test cases."""
def test_opened_file(self) -> None:
"""Test that an opened file can be passed to parse_file."""
with (TESTDATA_PATH / "test_vector.rda").open("rb") as f:
parsed = rdata.parser.parse_file(f)
converted = rdata.conversion.convert(parsed)
assert isinstance(converted, dict)
def test_opened_string(self) -> None:
"""Test that a string can be passed to parse_file."""
parsed = rdata.parser.parse_file(
str(TESTDATA_PATH / "test_vector.rda"),
)
converted = rdata.conversion.convert(parsed)
assert isinstance(converted, dict)
def test_logical(self) -> None:
"""Test parsing of logical vectors."""
data = rdata.read_rda(TESTDATA_PATH / "test_logical.rda")
np.testing.assert_equal(data, {
"test_logical": np.array([True, True, False, True, False]),
})
def test_nullable_logical(self) -> None:
"""Test parsing of logical vectors containing NA."""
data = rdata.read_rda(TESTDATA_PATH / "test_nullable_logical.rda")
array = data["test_nullable_logical"]
np.testing.assert_array_equal(
array.data,
np.array([True, False, True]),
)
np.testing.assert_array_equal(
array.mask,
np.array([False, False, True]),
)
def test_nullable_int(self) -> None:
"""Test parsing of integer vectors containing NA."""
data = rdata.read_rda(TESTDATA_PATH / "test_nullable_int.rda")
array = data["test_nullable_int"]
np.testing.assert_array_equal(
array.data,
np.array([313, -12, -2**31]),
)
np.testing.assert_array_equal(
array.mask,
np.array([False, False, True]),
)
def test_vector(self) -> None:
"""Test parsing of numerical vectors."""
data = rdata.read_rda(TESTDATA_PATH / "test_vector.rda")
np.testing.assert_equal(data, {
"test_vector": np.array([1.0, 2.0, 3.0]),
})
def test_empty_string(self) -> None:
"""Test that the empty string is parsed correctly."""
data = rdata.read_rda(TESTDATA_PATH / "test_empty_str.rda")
np.testing.assert_equal(data, {
"test_empty_str": [""],
})
def test_na_string(self) -> None:
"""Test that the NA string is parsed correctly."""
data = rdata.read_rda(TESTDATA_PATH / "test_na_string.rda")
np.testing.assert_equal(data, {
"test_na_string": [None],
})
def test_complex(self) -> None:
"""Test that complex numbers can be parsed."""
data = rdata.read_rda(TESTDATA_PATH / "test_complex.rda")
np.testing.assert_equal(data, {
"test_complex": np.array([1 + 2j, 2, 0, 1 + 3j, -1j]),
})
def test_matrix(self) -> None:
"""Test that a matrix can be parsed."""
data = rdata.read_rda(TESTDATA_PATH / "test_matrix.rda")
np.testing.assert_equal(data, {
"test_matrix": np.array([
[1.0, 2.0, 3.0],
[4.0, 5.0, 6.0],
]),
})
def test_named_matrix(self) -> None:
"""Test that a named matrix can be parsed."""
data = rdata.read_rda(TESTDATA_PATH / "test_named_matrix.rda")
reference = xarray.DataArray(
[
[1.0, 2.0, 3.0],
[4.0, 5.0, 6.0],
],
dims=["dim_0", "dim_1"],
coords={
"dim_0": ["dim0_0", "dim0_1"],
"dim_1": ["dim1_0", "dim1_1", "dim1_2"],
},
)
xarray.testing.assert_identical(
data["test_named_matrix"],
reference,
)
def test_half_named_matrix(self) -> None:
"""Test that a named matrix with no name for a dim can be parsed."""
data = rdata.read_rda(TESTDATA_PATH / "test_half_named_matrix.rda")
reference = xarray.DataArray(
[
[1.0, 2.0, 3.0],
[4.0, 5.0, 6.0],
],
dims=["dim_0", "dim_1"],
coords={
"dim_0": ["dim0_0", "dim0_1"],
},
)
xarray.testing.assert_identical(
data["test_half_named_matrix"],
reference,
)
def test_full_named_matrix(self) -> None:
"""Test that a named matrix with dim names can be parsed."""
data = rdata.read_rda(TESTDATA_PATH / "test_full_named_matrix.rda")
reference = xarray.DataArray(
[
[1.0, 2.0, 3.0],
[4.0, 5.0, 6.0],
],
dims=["my_dim_0", "my_dim_1"],
coords={
"my_dim_0": ["dim0_0", "dim0_1"],
"my_dim_1": ["dim1_0", "dim1_1", "dim1_2"],
},
)
xarray.testing.assert_identical(
data["test_full_named_matrix"],
reference,
)
def test_full_named_matrix_rds(self) -> None:
"""Test that a named matrix with dim names can be parsed."""
data = rdata.read_rds(TESTDATA_PATH / "test_full_named_matrix.rds")
reference = xarray.DataArray(
[
[1.0, 2.0, 3.0],
[4.0, 5.0, 6.0],
],
dims=["my_dim_0", "my_dim_1"],
coords={
"my_dim_0": ["dim0_0", "dim0_1"],
"my_dim_1": ["dim1_0", "dim1_1", "dim1_2"],
},
)
xarray.testing.assert_identical(
data,
reference,
)
def test_list(self) -> None:
"""Test that list can be parsed."""
data = rdata.read_rda(TESTDATA_PATH / "test_list.rda")
np.testing.assert_equal(data, {
"test_list":
[
np.array([1.0]),
["a", "b", "c"],
np.array([2.0, 3.0]),
["hi"],
],
})
@pytest.mark.filterwarnings("ignore:Missing constructor")
def test_file(self) -> None:
"""Test that external pointers can be parsed."""
data = rdata.read_rda(TESTDATA_PATH / "test_file.rda")
np.testing.assert_equal(data, {
"test_file": [5],
})
def test_expression(self) -> None:
"""Test that expressions can be parsed."""
data = rdata.read_rda(TESTDATA_PATH / "test_expression.rda")
np.testing.assert_equal(data, {
"test_expression": rdata.conversion.RExpression([
rdata.conversion.RLanguage(
["^", "base", "exponent"],
attributes={},
),
]),
})
def test_builtin(self) -> None:
"""Test that builtin functions can be parsed."""
data = rdata.read_rda(TESTDATA_PATH / "test_builtin.rda")
np.testing.assert_equal(data, {
"test_builtin": rdata.conversion.RBuiltin(name="abs"),
})
def test_minimal_function_uncompiled(self) -> None:
"""Test that a minimal function can be parsed."""
data = rdata.read_rda(
TESTDATA_PATH / "test_minimal_function_uncompiled.rda",
)
converted_fun = data["test_minimal_function_uncompiled"]
assert isinstance(
converted_fun,
rdata.conversion.RFunction,
)
np.testing.assert_equal(converted_fun.environment, ChainMap({}))
np.testing.assert_equal(converted_fun.formals, None)
np.testing.assert_equal(converted_fun.body, None)
np.testing.assert_equal(
converted_fun.source,
"test_minimal_function_uncompiled <- function() NULL\n",
)
@pytest.mark.filterwarnings("ignore:Missing constructor")
def test_minimal_function(self) -> None:
"""Test that a minimal function (compiled) can be parsed."""
data = rdata.read_rda(TESTDATA_PATH / "test_minimal_function.rda")
converted_fun = data["test_minimal_function"]
assert isinstance(
converted_fun,
rdata.conversion.RFunction,
)
np.testing.assert_equal(converted_fun.environment, ChainMap({}))
np.testing.assert_equal(converted_fun.formals, None)
converted_body = converted_fun.body
assert isinstance(
converted_body,
rdata.conversion.RBytecode,
)
np.testing.assert_equal(converted_body.code, np.array([12, 17, 1]))
np.testing.assert_equal(converted_body.attributes, {})
np.testing.assert_equal(
converted_fun.source,
"test_minimal_function <- function() NULL\n",
)
def test_empty_function_uncompiled(self) -> None:
"""Test that a simple function can be parsed."""
data = rdata.read_rda(
TESTDATA_PATH / "test_empty_function_uncompiled.rda",
)
converted_fun = data["test_empty_function_uncompiled"]
assert isinstance(
converted_fun,
rdata.conversion.RFunction,
)
np.testing.assert_equal(converted_fun.environment, ChainMap({}))
np.testing.assert_equal(converted_fun.formals, None)
assert isinstance(converted_fun.body, rdata.conversion.RLanguage)
np.testing.assert_equal(
converted_fun.source,
"test_empty_function_uncompiled <- function() {}\n",
)
@pytest.mark.filterwarnings("ignore:Missing constructor")
def test_empty_function(self) -> None:
"""Test that a simple function (compiled) can be parsed."""
data = rdata.read_rda(TESTDATA_PATH / "test_empty_function.rda")
converted_fun = data["test_empty_function"]
assert isinstance(
converted_fun,
rdata.conversion.RFunction,
)
np.testing.assert_equal(converted_fun.environment, ChainMap({}))
np.testing.assert_equal(converted_fun.formals, None)
converted_body = converted_fun.body
assert isinstance(
converted_body,
rdata.conversion.RBytecode,
)
np.testing.assert_equal(converted_body.code, np.array([12, 17, 1]))
np.testing.assert_equal(converted_body.attributes, {})
np.testing.assert_equal(
converted_fun.source,
"test_empty_function <- function() {}\n",
)
@pytest.mark.filterwarnings("ignore:Missing constructor")
def test_function(self) -> None:
"""Test that functions can be parsed."""
data = rdata.read_rda(TESTDATA_PATH / "test_function.rda")
converted_fun = data["test_function"]
assert isinstance(
converted_fun,
rdata.conversion.RFunction,
)
np.testing.assert_equal(converted_fun.environment, ChainMap({}))
np.testing.assert_equal(converted_fun.formals, None)
converted_body = converted_fun.body
assert isinstance(
converted_body,
rdata.conversion.RBytecode,
)
np.testing.assert_equal(
converted_body.code,
np.array([12, 23, 1, 34, 4, 38, 2, 1]),
)
np.testing.assert_equal(converted_body.attributes, {})
np.testing.assert_equal(
converted_fun.source,
"test_function <- function() {print(\"Hello\")}\n",
)
@pytest.mark.filterwarnings("ignore:Missing constructor")
def test_function_arg(self) -> None:
"""Test that functions can be parsed."""
data = rdata.read_rda(TESTDATA_PATH / "test_function_arg.rda")
converted_fun = data["test_function_arg"]
assert isinstance(
converted_fun,
rdata.conversion.RFunction,
)
np.testing.assert_equal(converted_fun.environment, ChainMap({}))
np.testing.assert_equal(converted_fun.formals, {"a": NotImplemented})
converted_body = converted_fun.body
assert isinstance(
converted_body,
rdata.conversion.RBytecode,
)
np.testing.assert_equal(
converted_body.code,
np.array([12, 23, 1, 29, 4, 38, 2, 1]),
)
np.testing.assert_equal(converted_body.attributes, {})
np.testing.assert_equal(
converted_fun.source,
"test_function_arg <- function(a) {print(a)}\n",
)
def test_encodings(self) -> None:
"""Test of differents encodings."""
with self.assertWarns(
UserWarning,
msg="Unknown encoding. Assumed ASCII.",
):
data = rdata.read_rda(
TESTDATA_PATH / "test_encodings.rda",
)
np.testing.assert_equal(data, {
"test_encoding_utf8": ["eĥoŝanĝo ĉiuĵaŭde"],
"test_encoding_latin1": ["cañón"],
"test_encoding_bytes": [b"reba\xf1o"],
"test_encoding_latin1_implicit": [b"\xcd\xf1igo"],
})
def test_encodings_v3(self) -> None:
"""Test encodings in version 3 format."""
data = rdata.read_rda(TESTDATA_PATH / "test_encodings_v3.rda")
np.testing.assert_equal(data, {
"test_encoding_utf8": ["eĥoŝanĝo ĉiuĵaŭde"],
"test_encoding_latin1": ["cañón"],
"test_encoding_bytes": [b"reba\xf1o"],
"test_encoding_latin1_implicit": ["Íñigo"],
})
def test_dataframe(self) -> None:
"""Test dataframe conversion."""
for f in ("test_dataframe.rda", "test_dataframe_v3.rda"):
with self.subTest(file=f):
data = rdata.read_rda(TESTDATA_PATH / f)
pd.testing.assert_frame_equal(
data["test_dataframe"],
pd.DataFrame(
{
"class": pd.Categorical(
["a", "b", "b"],
),
"value": pd.Series(
[1, 2, 3],
dtype=pd.Int32Dtype(),
).array,
},
index=pd.RangeIndex(start=1, stop=4),
),
)
def test_dataframe_rds(self) -> None:
"""Test dataframe conversion."""
for f in ("test_dataframe.rds", "test_dataframe_v3.rds"):
with self.subTest(file=f):
data = rdata.read_rds(TESTDATA_PATH / f)
pd.testing.assert_frame_equal(
data,
pd.DataFrame(
{
"class": pd.Categorical(
["a", "b", "b"],
),
"value": pd.Series(
[1, 2, 3],
dtype=pd.Int32Dtype(),
).array,
},
index=pd.RangeIndex(start=1, stop=4),
),
)
def test_dataframe_rownames(self) -> None:
"""Test dataframe conversion."""
data = rdata.read_rda(TESTDATA_PATH / "test_dataframe_rownames.rda")
pd.testing.assert_frame_equal(
data["test_dataframe_rownames"],
pd.DataFrame(
{
"class": pd.Categorical(
["a", "b", "b"],
),
"value": pd.Series(
[1, 2, 3],
dtype=pd.Int32Dtype(),
).array,
},
index=("Madrid", "Frankfurt", "Herzberg am Harz"),
),
)
def test_ts(self) -> None:
"""Test time series conversion."""
data = rdata.read_rda(TESTDATA_PATH / "test_ts.rda")
pd.testing.assert_series_equal(
data["test_ts"],
pd.Series({
2000 + Fraction(2, 12): 1.0,
2000 + Fraction(3, 12): 2.0,
2000 + Fraction(4, 12): 3.0,
}),
)
def test_s4(self) -> None:
"""Test parsing of S4 classes."""
with pytest.warns(UserWarning, match="Missing constructor"):
data = rdata.read_rda(TESTDATA_PATH / "test_s4.rda")
np.testing.assert_equal(data, {
"test_s4": SimpleNamespace(
age=np.array(28),
name=["Carlos"],
**{"class": ["Person"]},
),
})
def test_environment(self) -> None:
"""Test parsing of environments."""
parsed = rdata.parser.parse_file(
TESTDATA_PATH / "test_environment.rda",
)
converted = rdata.conversion.convert(parsed)
dict_env = {"string": ["test"]}
empty_global_env: dict[str, Any] = {}
np.testing.assert_equal(converted, {
"test_environment": ChainMap(dict_env, ChainMap(empty_global_env)),
})
global_env = {"global": "test"}
converted_global = rdata.conversion.convert(
parsed,
global_environment=global_env,
)
np.testing.assert_equal(converted_global, {
"test_environment": ChainMap(dict_env, ChainMap(global_env)),
})
def test_emptyenv(self) -> None:
"""Test parsing the empty environment."""
data = rdata.read_rda(TESTDATA_PATH / "test_emptyenv.rda")
assert data == {
"test_emptyenv": ChainMap({}),
}
def test_list_attrs(self) -> None:
"""Test that lists accept attributes."""
data = rdata.read_rda(TESTDATA_PATH / "test_list_attrs.rda")
np.testing.assert_equal(data, {
"test_list_attrs": [["list"], [5]],
})
def test_altrep_compact_intseq(self) -> None:
"""Test alternative representation of sequences of ints."""
data = rdata.read_rda(TESTDATA_PATH / "test_altrep_compact_intseq.rda")
np.testing.assert_equal(data, {
"test_altrep_compact_intseq": np.arange(1000),
})
def test_altrep_compact_intseq_asymmetric(self) -> None:
"""
Test alternative representation of sequences of ints.
This test an origin different from 0, to reproduce
issue #29.
"""
data = rdata.read_rda(
TESTDATA_PATH / "test_altrep_compact_intseq_asymmetric.rda",
)
np.testing.assert_equal(data, {
"test_altrep_compact_intseq_asymmetric": np.arange(-5, 6),
})
def test_altrep_compact_realseq(self) -> None:
"""Test alternative representation of sequences of ints."""
data = rdata.read_rda(
TESTDATA_PATH / "test_altrep_compact_realseq.rda",
)
np.testing.assert_equal(data, {
"test_altrep_compact_realseq": np.arange(1000.0),
})
def test_altrep_compact_realseq_asymmetric(self) -> None:
"""
Test alternative representation of sequences of ints.
This test an origin different from 0, to reproduce
issue #29.
"""
data = rdata.read_rda(
TESTDATA_PATH / "test_altrep_compact_realseq_asymmetric.rda",
)
np.testing.assert_equal(data, {
"test_altrep_compact_realseq_asymmetric": np.arange(-5.0, 6.0),
})
def test_altrep_deferred_string(self) -> None:
"""Test alternative representation of deferred strings."""
data = rdata.read_rda(
TESTDATA_PATH / "test_altrep_deferred_string.rda",
)
np.testing.assert_equal(data, {
"test_altrep_deferred_string": [
"1", "2.3", "10000",
"1e+05", "-10000", "-1e+05",
"0.001", "1e-04", "1e-05",
],
})
def test_altrep_wrap_real(self) -> None:
"""Test alternative representation of wrap_real."""
data = rdata.read_rda(
TESTDATA_PATH / "test_altrep_wrap_real.rda",
)
np.testing.assert_equal(data, {
"test_altrep_wrap_real": [3],
})
def test_altrep_wrap_string(self) -> None:
"""Test alternative representation of wrap_string."""
data = rdata.read_rda(TESTDATA_PATH / "test_altrep_wrap_string.rda")
np.testing.assert_equal(data, {
"test_altrep_wrap_string": ["Hello"],
})
def test_altrep_wrap_logical(self) -> None:
"""Test alternative representation of wrap_logical."""
data = rdata.read_rda(TESTDATA_PATH / "test_altrep_wrap_logical.rda")
np.testing.assert_equal(data, {
"test_altrep_wrap_logical": [True],
})
def test_ascii(self) -> None:
"""Test ascii files."""
ref_ma = np.ma.array( # type: ignore[no-untyped-call]
data=[True],
mask=[True],
fill_value=True,
)
ref = [[1.1], [2], [3. + 4.j], ref_ma, ["aä"]]
for tag, v, ext in itertools.product(
("", "win_"),
(2, 3),
("rda", "rds"),
):
f = f"test_ascii_{tag}v{v}.{ext}"
with self.subTest(file=f):
parsed = rdata.parser.parse_file(
TESTDATA_PATH / f,
)
converted = rdata.conversion.convert(parsed)
if ext == "rda":
np.testing.assert_equal(converted, {"data": ref})
ma = converted["data"][3]
else:
np.testing.assert_equal(converted, ref)
ma = converted[3]
# Test masked array separately
np.testing.assert_equal(ma.data, ref_ma.data)
np.testing.assert_equal(ma.mask, ref_ma.mask)
np.testing.assert_equal(ma.mask, ref_ma.mask)
np.testing.assert_equal(ma.get_fill_value(),
ref_ma.get_fill_value())
if __name__ == "__main__":
unittest.main()
rdata-0.11.2/readthedocs.yml 0000664 0000000 0000000 00000001170 14571337522 0015706 0 ustar 00root root 0000000 0000000 # .readthedocs.yml
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
# Required
version: 2
build:
os: ubuntu-22.04
tools:
python: "3.9"
# Build documentation in the docs/ directory with Sphinx
sphinx:
builder: html
configuration: docs/conf.py
# Build documentation with MkDocs
#mkdocs:
# configuration: mkdocs.yml
# Optionally build your docs in additional formats such as PDF
# Optionally set the version of Python and requirements required to build your docs
python:
install:
- method: pip
path: .
extra_requirements:
- docs
Add your contributions