././@PaxHeader 0000000 0000000 0000000 00000000034 00000000000 011452 x ustar 00 0000000 0000000 28 mtime=1612763830.9979815
visidata-2.2.1/ 0000770 0001750 0001750 00000000000 00000000000 014006 5 ustar 00kefala kefala 0000000 0000000 ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1593229243.0
visidata-2.2.1/LICENSE.gpl3 0000660 0001750 0001750 00000104513 00000000000 015664 0 ustar 00kefala kefala 0000000 0000000 GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc.
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU General Public License is a free, copyleft license for
software and other kinds of works.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
the GNU General Public License is intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users. We, the Free Software Foundation, use the
GNU General Public License for most of our software; it applies also to
any other work released this way by its authors. You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
To protect your rights, we need to prevent others from denying you
these rights or asking you to surrender the rights. Therefore, you have
certain responsibilities if you distribute copies of the software, or if
you modify it: responsibilities to respect the freedom of others.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must pass on to the recipients the same
freedoms that you received. You must make sure that they, too, receive
or can get the source code. And you must show them these terms so they
know their rights.
Developers that use the GNU GPL protect your rights with two steps:
(1) assert copyright on the software, and (2) offer you this License
giving you legal permission to copy, distribute and/or modify it.
For the developers' and authors' protection, the GPL clearly explains
that there is no warranty for this free software. For both users' and
authors' sake, the GPL requires that modified versions be marked as
changed, so that their problems will not be attributed erroneously to
authors of previous versions.
Some devices are designed to deny users access to install or run
modified versions of the software inside them, although the manufacturer
can do so. This is fundamentally incompatible with the aim of
protecting users' freedom to change the software. The systematic
pattern of such abuse occurs in the area of products for individuals to
use, which is precisely where it is most unacceptable. Therefore, we
have designed this version of the GPL to prohibit the practice for those
products. If such problems arise substantially in other domains, we
stand ready to extend this provision to those domains in future versions
of the GPL, as needed to protect the freedom of users.
Finally, every program is threatened constantly by software patents.
States should not allow patents to restrict development and use of
software on general-purpose computers, but in those that do, we wish to
avoid the special danger that patents applied to a free program could
make it effectively proprietary. To prevent this, the GPL assures that
patents cannot be used to render the program non-free.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Use with the GNU Affero General Public License.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU Affero General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the special requirements of the GNU Affero General Public License,
section 13, concerning interaction through a network will apply to the
combination as such.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
Copyright (C)
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see .
Also add information on how to contact you by electronic and paper mail.
If the program does terminal interaction, make it output a short
notice like this when it starts in an interactive mode:
Copyright (C)
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, your program's commands
might be different; for a GUI interface, you would use an "about box".
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU GPL, see
.
The GNU General Public License does not permit incorporating your program
into proprietary programs. If your program is a subroutine library, you
may consider it more useful to permit linking proprietary applications with
the library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License. But first, please read
.
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612763790.0
visidata-2.2.1/MANIFEST.in 0000660 0001750 0001750 00000000175 00000000000 015550 0 ustar 00kefala kefala 0000000 0000000 include README.md
include LICENSE.gpl3
include visidata/man/vd.1
include visidata/man/vd.txt
include visidata/man/visidata.1
././@PaxHeader 0000000 0000000 0000000 00000000034 00000000000 011452 x ustar 00 0000000 0000000 28 mtime=1612763830.9979815
visidata-2.2.1/PKG-INFO 0000660 0001750 0001750 00000010515 00000000000 015106 0 ustar 00kefala kefala 0000000 0000000 Metadata-Version: 2.1
Name: visidata
Version: 2.2.1
Summary: terminal interface for exploring and arranging tabular data
Home-page: https://visidata.org
Author: Saul Pwanson
Author-email: visidata@saul.pw
License: GPLv3
Download-URL: https://github.com/saulpw/visidata/tarball/2.2.1
Description:
# VisiData v2.2.1 [![twitter @VisiData][1.1]][1] [](https://circleci.com/gh/saulpw/visidata/tree/stable) [](https://gitpod.io/#https://github.com/saulpw/visidata)
A terminal interface for exploring and arranging tabular data.
VisiData supports tsv, csv, sqlite, json, xlsx (Excel), hdf5, and [many other formats](https://visidata.org/formats).
## Platform requirements
- Linux, OS/X, or Windows (with WSL)
- Python 3.6+
- additional Python modules are required for certain formats and sources
## Install
To install the latest release from PyPi:
pip3 install visidata
To install the cutting edge `develop` branch (no warranty expressed or implied):
pip3 install git+https://github.com/saulpw/visidata.git@develop
See [visidata.org/install](https://visidata.org/install) for detailed instructions for all available platforms and package managers.
### Usage
$ vd
$ | vd
Press `Ctrl+Q` to quit at any time.
Hundreds of other commands and options are also available; see the documentation.
### Documentation
* [VisiData documentation](https://visidata.org/docs)
* [Plugin Author's Guide and API Reference](https://visidata.org/docs/api)
* [Quick reference](https://visidata.org/man) (available within `vd` with `Ctrl+H`), which has a list of commands and options.
* [Intro to VisiData Tutorial](https://jsvine.github.io/intro-to-visidata/) by [Jeremy Singer-Vine](https://www.jsvine.com/)
### Help and Support
If you have a question, issue, or suggestion regarding VisiData, please [create an issue on Github](https://github.com/saulpw/visidata/issues) or chat with us at #visidata on [freenode.net](https://webchat.freenode.net/).
If you use VisiData regularly, please [support me on Patreon](https://www.patreon.com/saulpw)!
## License
Code in the `stable` branch of this repository, including the main `vd` application, loaders, and plugins, is available for use and redistribution under GPLv3.
## Credits
VisiData is conceived and developed by Saul Pwanson ``.
Anja Kefala `` maintains the documentation and packages for all platforms.
Many thanks to numerous other [contributors](https://visidata.org/credits/), and to those wonderful users who provide feedback, for helping to make VisiData the awesome tool that it is.
[1.1]: http://i.imgur.com/tXSoThF.png
[1]: http://www.twitter.com/VisiData
Keywords: console tabular data spreadsheet terminal viewer textpunkcurses csv hdf5 h5 xlsx excel tsv
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Environment :: Console :: Curses
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Database :: Front-Ends
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Office/Business :: Financial :: Spreadsheet
Classifier: Topic :: Scientific/Engineering :: Visualization
Classifier: Topic :: Utilities
Requires-Python: >=3.6
Description-Content-Type: text/markdown
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612763790.0
visidata-2.2.1/README.md 0000660 0001750 0001750 00000005250 00000000000 015270 0 ustar 00kefala kefala 0000000 0000000
# VisiData v2.2.1 [![twitter @VisiData][1.1]][1] [](https://circleci.com/gh/saulpw/visidata/tree/stable) [](https://gitpod.io/#https://github.com/saulpw/visidata)
A terminal interface for exploring and arranging tabular data.
VisiData supports tsv, csv, sqlite, json, xlsx (Excel), hdf5, and [many other formats](https://visidata.org/formats).
## Platform requirements
- Linux, OS/X, or Windows (with WSL)
- Python 3.6+
- additional Python modules are required for certain formats and sources
## Install
To install the latest release from PyPi:
pip3 install visidata
To install the cutting edge `develop` branch (no warranty expressed or implied):
pip3 install git+https://github.com/saulpw/visidata.git@develop
See [visidata.org/install](https://visidata.org/install) for detailed instructions for all available platforms and package managers.
### Usage
$ vd
$ | vd
Press `Ctrl+Q` to quit at any time.
Hundreds of other commands and options are also available; see the documentation.
### Documentation
* [VisiData documentation](https://visidata.org/docs)
* [Plugin Author's Guide and API Reference](https://visidata.org/docs/api)
* [Quick reference](https://visidata.org/man) (available within `vd` with `Ctrl+H`), which has a list of commands and options.
* [Intro to VisiData Tutorial](https://jsvine.github.io/intro-to-visidata/) by [Jeremy Singer-Vine](https://www.jsvine.com/)
### Help and Support
If you have a question, issue, or suggestion regarding VisiData, please [create an issue on Github](https://github.com/saulpw/visidata/issues) or chat with us at #visidata on [freenode.net](https://webchat.freenode.net/).
If you use VisiData regularly, please [support me on Patreon](https://www.patreon.com/saulpw)!
## License
Code in the `stable` branch of this repository, including the main `vd` application, loaders, and plugins, is available for use and redistribution under GPLv3.
## Credits
VisiData is conceived and developed by Saul Pwanson ``.
Anja Kefala `` maintains the documentation and packages for all platforms.
Many thanks to numerous other [contributors](https://visidata.org/credits/), and to those wonderful users who provide feedback, for helping to make VisiData the awesome tool that it is.
[1.1]: http://i.imgur.com/tXSoThF.png
[1]: http://www.twitter.com/VisiData
././@PaxHeader 0000000 0000000 0000000 00000000034 00000000000 011452 x ustar 00 0000000 0000000 28 mtime=1612763830.8819814
visidata-2.2.1/bin/ 0000770 0001750 0001750 00000000000 00000000000 014556 5 ustar 00kefala kefala 0000000 0000000 ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1606688014.0
visidata-2.2.1/bin/vd 0000770 0001750 0001750 00000000144 00000000000 015114 0 ustar 00kefala kefala 0000000 0000000 #!/usr/bin/env python3
import visidata.main
if __name__ == '__main__':
visidata.main.vd_cli()
././@PaxHeader 0000000 0000000 0000000 00000000034 00000000000 011452 x ustar 00 0000000 0000000 28 mtime=1612763830.9979815
visidata-2.2.1/setup.cfg 0000660 0001750 0001750 00000000046 00000000000 015630 0 ustar 00kefala kefala 0000000 0000000 [egg_info]
tag_build =
tag_date = 0
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612763790.0
visidata-2.2.1/setup.py 0000770 0001750 0001750 00000003726 00000000000 015533 0 ustar 00kefala kefala 0000000 0000000 #!/usr/bin/env python3
from setuptools import setup
# tox can't actually run python3 setup.py: https://github.com/tox-dev/tox/issues/96
#from visidata import __version__
__version__ = '2.2.1'
setup(name='visidata',
version=__version__,
description='terminal interface for exploring and arranging tabular data',
long_description=open('README.md').read(),
long_description_content_type='text/markdown',
author='Saul Pwanson',
python_requires='>=3.6',
author_email='visidata@saul.pw',
url='https://visidata.org',
download_url='https://github.com/saulpw/visidata/tarball/' + __version__,
scripts=['bin/vd'],
entry_points={'console_scripts': [
'visidata=visidata.main:vd_cli'
],
},
py_modules = ['visidata'],
install_requires=['python-dateutil'],
packages=['visidata', 'visidata.loaders', 'visidata.tests'],
include_package_data=True,
data_files = [('share/man/man1', ['visidata/man/vd.1', 'visidata/man/visidata.1'])],
package_data={'visidata': ['man/vd.1', 'man/vd.txt']},
license='GPLv3',
classifiers=[
'Development Status :: 5 - Production/Stable',
'Environment :: Console',
'Environment :: Console :: Curses',
'Intended Audience :: Developers',
'Intended Audience :: Science/Research',
'Intended Audience :: System Administrators',
'License :: OSI Approved :: GNU General Public License v3 (GPLv3)',
'Operating System :: OS Independent',
'Programming Language :: Python :: 3',
'Topic :: Database :: Front-Ends',
'Topic :: Scientific/Engineering',
'Topic :: Office/Business :: Financial :: Spreadsheet',
'Topic :: Scientific/Engineering :: Visualization',
'Topic :: Utilities',
],
keywords=('console tabular data spreadsheet terminal viewer textpunk'
'curses csv hdf5 h5 xlsx excel tsv'),
)
././@PaxHeader 0000000 0000000 0000000 00000000034 00000000000 011452 x ustar 00 0000000 0000000 28 mtime=1612763830.9379814
visidata-2.2.1/visidata/ 0000770 0001750 0001750 00000000000 00000000000 015612 5 ustar 00kefala kefala 0000000 0000000 ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612763790.0
visidata-2.2.1/visidata/__init__.py 0000660 0001750 0001750 00000006762 00000000000 017737 0 ustar 00kefala kefala 0000000 0000000 'VisiData: a curses interface for exploring and arranging tabular data'
__version__ = '2.2.1'
__version_info__ = 'VisiData v' + __version__
__author__ = 'Saul Pwanson '
__status__ = 'Production/Stable'
__copyright__ = 'Copyright (c) 2016-2019 ' + __author__
class EscapeException(BaseException):
'Inherits from BaseException to avoid "except Exception" clauses. Do not use a blanket "except:" or the task will be uncancelable.'
pass
def addGlobals(g):
'''Update the VisiData globals dict with items from *g*, which is a mapping of names to functions.
Importers can call ``addGlobals(globals())`` to have their globals accessible to execstrings.'''
globals().update(g)
def getGlobals():
'Return the VisiData globals dict.'
return globals()
from builtins import *
from copy import copy, deepcopy
from .utils import *
from .extensible import *
from .vdobj import *
vd = VisiData()
vd.addGlobals = addGlobals
vd.getGlobals = getGlobals
from .basesheet import *
from .settings import *
from .errors import *
from .editor import *
from .cliptext import *
from .color import *
from .mainloop import *
from .wrappers import *
from .undo import *
from ._types import *
from .column import *
theme = option # convert theme(...) to option(...) and move this down, eventually into deprecated.py
from .sheets import *
from .statusbar import *
from .textsheet import *
from .threads import *
from ._input import *
from .movement import *
from .path import *
from .urlcache import *
from .selection import *
from .loaders.tsv import *
from .pyobj import *
from .loaders.json import *
from ._open import *
from .save import *
from .clipboard import *
from .slide import *
from .search import *
from .expr import *
from .choose import *
from .metasheets import *
from .join import *
from .aggregators import *
from .describe import *
from .pivot import *
from .freqtbl import *
from .melt import *
from .cmdlog import *
from .freeze import *
from .regex import *
from .canvas import *
from .graph import *
from .motd import *
from .transpose import *
from .shell import *
from .layout import *
from .main import *
from .help import *
from .modify import *
import visidata.sort
import visidata.unfurl
import visidata.fill
import visidata.incr
import visidata.customdate
import visidata.misc
from .macros import *
from .menu import *
from .loaders.csv import *
from .loaders.archive import *
from .loaders.xlsx import *
from .loaders.xlsb import *
from .loaders.hdf5 import *
from .loaders.sqlite import *
from .loaders.fixed_width import *
from .loaders.postgres import *
from .loaders.mysql import *
from .loaders.shp import *
from .loaders.geojson import *
from .loaders.mbtiles import *
from .loaders.http import *
from .loaders.html import *
from .loaders.markdown import *
from .loaders.pcap import *
from .loaders.png import *
from .loaders.ttf import *
from .loaders.sas import *
from .loaders.spss import *
from .loaders.xml import *
from .loaders.yaml import *
from .loaders._pandas import *
from .loaders.graphviz import *
from .loaders.npy import *
from .loaders.usv import *
from .loaders.frictionless import *
from .loaders.imap import *
from .loaders.pdf import *
from .loaders.pandas_freqtbl import *
from .loaders.xword import *
from .loaders.vcf import *
from .loaders.texttables import *
from .loaders.rec import *
from .loaders.eml import *
from .loaders.vds import *
from .plugins import *
from .colorsheet import *
from .deprecated import *
from math import *
vd.finalInit()
vd.addGlobals(globals())
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608541899.0
visidata-2.2.1/visidata/__main__.py 0000660 0001750 0001750 00000000043 00000000000 017702 0 ustar 00kefala kefala 0000000 0000000 from .main import vd_cli
vd_cli()
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612075960.0
visidata-2.2.1/visidata/_input.py 0000660 0001750 0001750 00000034544 00000000000 017475 0 ustar 00kefala kefala 0000000 0000000 from contextlib import suppress
import collections
import curses
import visidata
from visidata import EscapeException, ExpectedException, clipdraw, Sheet, VisiData
from visidata import vd, option, options, theme, colors
from visidata import launchExternalEditor, suspend, ColumnItem, AttrDict
__all__ = ['confirm', 'CompleteKey']
theme('color_edit_cell', 'normal', 'cell color to use when editing cell')
theme('disp_edit_fill', '_', 'edit field fill character')
theme('disp_unprintable', '·', 'substitute character for unprintables')
option('input_history', '', 'basename of file to store persistent input history')
class AcceptInput(Exception):
'*args[0]* is the input to be accepted'
visidata.vd._nextCommands = []
@VisiData.api
def queueCommand(vd, longname): #, input=None, sheet=None, col=None, row=None):
vd._nextCommands.append(longname)
def acceptThenFunc(*longnames):
def _acceptthen(v, i):
for longname in longnames:
vd.queueCommand(longname)
raise AcceptInput(v)
return _acceptthen
# editline helpers
class EnableCursor:
def __enter__(self):
with suppress(curses.error):
curses.mousemask(0)
curses.curs_set(1)
def __exit__(self, exc_type, exc_val, tb):
with suppress(curses.error):
curses.curs_set(0)
curses.mousemask(-1)
def until_get_wch(scr):
'Ignores get_wch timeouts'
ret = None
while not ret:
try:
ret = scr.get_wch()
except curses.error:
pass
return ret
def splice(v, i, s):
'Insert `s` into string `v` at `i` (such that v[i] == s[0]).'
return v if i < 0 else v[:i] + s + v[i:]
def clean_printable(s):
'Escape unprintable characters.'
return ''.join(c if c.isprintable() else options.disp_unprintable for c in str(s))
def delchar(s, i, remove=1):
'Delete `remove` characters from str `s` beginning at position `i`.'
return s if i < 0 else s[:i] + s[i+remove:]
class CompleteState:
def __init__(self, completer_func):
self.comps_idx = -1
self.completer_func = completer_func
self.former_i = None
self.just_completed = False
def complete(self, v, i, state_incr):
self.just_completed = True
self.comps_idx += state_incr
if self.former_i is None:
self.former_i = i
try:
r = self.completer_func(v[:self.former_i], self.comps_idx)
except Exception as e:
# raise # beep/flash; how to report exception?
return v, i
if not r:
# beep/flash to indicate no matches?
return v, i
v = r + v[i:]
return v, len(v)
def reset(self):
if self.just_completed:
self.just_completed = False
else:
self.former_i = None
self.comps_idx = -1
class HistoryState:
def __init__(self, history):
self.history = history
self.hist_idx = None
self.prev_val = None
def up(self, v, i):
if self.hist_idx is None:
self.hist_idx = len(self.history)
self.prev_val = v
if self.hist_idx > 0:
self.hist_idx -= 1
v = self.history[self.hist_idx]
i = len(v)
return v, i
def down(self, v, i):
if self.hist_idx is None:
return v, i
elif self.hist_idx < len(self.history)-1:
self.hist_idx += 1
v = self.history[self.hist_idx]
else:
v = self.prev_val
self.hist_idx = None
i = len(v)
return v, i
# history: earliest entry first
@VisiData.api
def editline(vd, scr, y, x, w, i=0, attr=curses.A_NORMAL, value='', fillchar=' ', truncchar='-', unprintablechar='.', completer=lambda text,idx: None, history=[], display=True, updater=lambda val: None, bindings={}):
'A better curses line editing widget.'
with EnableCursor():
ESC='^['
TAB='^I'
history_state = HistoryState(history)
complete_state = CompleteState(completer)
insert_mode = True
first_action = True
v = str(value) # value under edit
# i = 0 # index into v, initial value can be passed in as argument as of 1.2
if i != 0:
first_action = False
left_truncchar = right_truncchar = truncchar
def find_nonword(s, a, b, incr):
if not s: return 0
a = min(max(a, 0), len(s)-1)
b = min(max(b, 0), len(s)-1)
if incr < 0:
while not s[b].isalnum() and b >= a: # first skip non-word chars
b += incr
while s[b].isalnum() and b >= a:
b += incr
return min(max(b, -1), len(s))
else:
while not s[a].isalnum() and a < b: # first skip non-word chars
a += incr
while s[a].isalnum() and a < b:
a += incr
return min(max(a, 0), len(s))
while True:
updater(v)
if display:
dispval = clean_printable(v)
else:
dispval = '*' * len(v)
dispi = i # the onscreen offset within the field where v[i] is displayed
if len(dispval) < w: # entire value fits
dispval += fillchar*(w-len(dispval)-1)
elif i == len(dispval): # cursor after value (will append)
dispi = w-1
dispval = left_truncchar + dispval[len(dispval)-w+2:] + fillchar
elif i >= len(dispval)-w//2: # cursor within halfwidth of end
dispi = w-(len(dispval)-i)
dispval = left_truncchar + dispval[len(dispval)-w+1:]
elif i <= w//2: # cursor within halfwidth of beginning
dispval = dispval[:w-1] + right_truncchar
else:
dispi = w//2 # visual cursor stays right in the middle
k = 1 if w%2==0 else 0 # odd widths have one character more
dispval = left_truncchar + dispval[i-w//2+1:i+w//2-k] + right_truncchar
prew = clipdraw(scr, y, x, dispval[:dispi], attr, w)
clipdraw(scr, y, x+prew, dispval[dispi:], attr, w-prew+1)
scr.move(y, x+prew)
ch = vd.getkeystroke(scr)
if ch == '': continue
elif ch == 'KEY_IC': insert_mode = not insert_mode
elif ch == '^A' or ch == 'KEY_HOME': i = 0
elif ch == '^B' or ch == 'KEY_LEFT': i -= 1
elif ch in ('^C', '^Q', ESC): raise EscapeException(ch)
elif ch == '^D' or ch == 'KEY_DC': v = delchar(v, i)
elif ch == '^E' or ch == 'KEY_END': i = len(v)
elif ch == '^F' or ch == 'KEY_RIGHT': i += 1
elif ch in ('^H', 'KEY_BACKSPACE', '^?'): i -= 1; v = delchar(v, i)
elif ch == TAB: v, i = complete_state.complete(v, i, +1)
elif ch == 'KEY_BTAB': v, i = complete_state.complete(v, i, -1)
elif ch in ['^J', '^M']: break # ENTER to accept value
elif ch == '^K': v = v[:i] # ^Kill to end-of-line
elif ch == '^O': v = launchExternalEditor(v)
elif ch == '^R': v = str(value) # ^Reload initial value
elif ch == '^T': v = delchar(splice(v, i-2, v[i-1]), i) # swap chars
elif ch == '^U': v = v[i:]; i = 0 # clear to beginning
elif ch == '^V': v = splice(v, i, until_get_wch(scr)); i += 1 # literal character
elif ch == '^W': j = find_nonword(v, 0, i-1, -1); v = v[:j+1] + v[i:]; i = j+1 # erase word
elif ch == '^Y': v = splice(v, i, vd.clipcells[0])
elif ch == '^Z': suspend()
# CTRL+arrow
elif ch == 'kLFT5': i = find_nonword(v, 0, i-1, -1)+1; # word left
elif ch == 'kRIT5': i = find_nonword(v, i+1, len(v)-1, +1)+1; # word right
elif ch == 'kUP5': pass
elif ch == 'kDN5': pass
elif history and ch == 'KEY_UP': v, i = history_state.up(v, i)
elif history and ch == 'KEY_DOWN': v, i = history_state.down(v, i)
elif ch in bindings: v, i = bindings[ch](v, i)
elif len(ch) > 1: pass
else:
if first_action:
v = ''
if insert_mode:
v = splice(v, i, ch)
else:
v = v[:i] + ch + v[i+1:]
i += 1
if i < 0: i = 0
if i > len(v): i = len(v)
first_action = False
complete_state.reset()
return v
@VisiData.api
def editText(vd, y, x, w, record=True, display=True, **kwargs):
'Invoke modal single-line editor at (*y*, *x*) for *w* terminal chars. Use *display* is False for sensitive input like passphrases. If *record* is True, get input from the cmdlog in batch mode, and save input to the cmdlog if *display* is also True. Return new value as string.'
v = None
if record and vd.cmdlog:
v = vd.getLastArgs()
if v is None:
try:
v = vd.editline(vd.sheets[0]._scr, y, x, w, display=display, **kwargs)
except AcceptInput as e:
v = e.args[0]
# clear keyboard buffer to neutralize multi-line pastes (issue#585)
curses.flushinp()
if display:
vd.status('"%s"' % v)
if record and vd.cmdlog:
vd.setLastArgs(v)
return v
@VisiData.api
def inputsingle(vd, prompt, record=True):
'Display prompt and return single character of user input.'
sheet = vd.sheets[0]
rstatuslen = vd.drawRightStatus(sheet._scr, sheet)
v = None
if record and vd.cmdlog:
v = vd.getLastArgs()
if v is not None:
return v
y = sheet.windowHeight-1
w = sheet.windowWidth
rstatuslen = vd.drawRightStatus(sheet._scr, sheet)
promptlen = clipdraw(sheet._scr, y, 0, prompt, 0, w=w-rstatuslen-1)
sheet._scr.move(y, w-promptlen-rstatuslen-2)
v = vd.getkeystroke(sheet._scr)
if record and vd.cmdlog:
vd.setLastArgs(v)
return v
@VisiData.api
def input(self, prompt, type=None, defaultLast=False, history=[], **kwargs):
'''Display *prompt* and return line of user input.
- *type*: string indicating the type of input to use for history.
- *history*: list of strings to use for input history.
- *defaultLast*: on empty input, if True, return last history item.
- *display*: pass False to not display input (for sensitive input, e.g. a password).
- *record*: pass False to not record input on cmdlog (for sensitive or inconsequential input).
- *completer*: ``completer(val, idx)`` is called on TAB to get next completed value.
- *updater*: ``updater(val)`` is called every keypress or timeout.
- *bindings*: dict of keystroke to func(v, i) that returns updated (v, i)
'''
history = self.lastInputsSheet.history(type)
sheet = self.activeSheet
rstatuslen = self.drawRightStatus(sheet._scr, sheet)
attr = 0
promptlen = clipdraw(sheet._scr, sheet.windowHeight-1, 0, prompt, attr, w=sheet.windowWidth-rstatuslen-1)
ret = self.editText(sheet.windowHeight-1, promptlen, sheet.windowWidth-promptlen-rstatuslen-2,
attr=colors.color_edit_cell,
unprintablechar=options.disp_unprintable,
truncchar=options.disp_truncator,
history=history,
**kwargs)
if ret:
self.lastInputsSheet.appendRow(AttrDict(type=type, input=ret))
elif defaultLast:
history or vd.fail("no previous input")
ret = history[-1]
return ret
@VisiData.global_api
def confirm(vd, prompt, exc=EscapeException):
'Display *prompt* on status line and demand input that starts with "Y" or "y" to proceed. Raise *exc* otherwise. Return True.'
if options.batch:
return vd.fail('cannot confirm in batch mode: ' + prompt)
yn = vd.input(prompt, value='no', record=False)[:1]
if not yn or yn not in 'Yy':
msg = 'disconfirmed: ' + prompt
if exc:
raise exc(msg)
vd.warning(msg)
return False
return True
class CompleteKey:
def __init__(self, items):
self.items = items
def __call__(self, val, state):
opts = [x for x in self.items if x.startswith(val)]
return opts[state%len(opts)] if opts else val
@Sheet.api
def editCell(self, vcolidx=None, rowidx=None, value=None, **kwargs):
'''Call vd.editText for the cell at (*rowidx*, *vcolidx*). Return the new value, properly typed.
- *rowidx*: numeric index into ``self.rows``. If negative, indicates the column name in the header.
- *value*: if given, the starting input; otherwise the starting input is the cell value or column name as appropriate.
- *kwargs*: passthrough args to ``vd.editText``.
'''
if vcolidx is None:
vcolidx = self.cursorVisibleColIndex
x, w = self._visibleColLayout.get(vcolidx, (0, 0))
col = self.visibleCols[vcolidx]
if rowidx is None:
rowidx = self.cursorRowIndex
if rowidx < 0: # header
y = 0
value = value or col.name
else:
y, h = self._rowLayout.get(rowidx, (0, 0))
value = value or col.getDisplayValue(self.rows[self.cursorRowIndex])
bindings={
'kUP': acceptThenFunc('go-up', 'rename-col' if rowidx < 0 else 'edit-cell'),
'KEY_SR': acceptThenFunc('go-up', 'rename-col' if rowidx < 0 else 'edit-cell'),
'kDN': acceptThenFunc('go-down', 'rename-col' if rowidx < 0 else 'edit-cell'),
'KEY_SF': acceptThenFunc('go-down', 'rename-col' if rowidx < 0 else 'edit-cell'),
'KEY_SRIGHT': acceptThenFunc('go-right', 'rename-col' if rowidx < 0 else 'edit-cell'),
'KEY_SLEFT': acceptThenFunc('go-left', 'rename-col' if rowidx < 0 else 'edit-cell'),
}
bindings.update(kwargs.get('bindings', {}))
kwargs['bindings'] = bindings
editargs = dict(value=value,
fillchar=options.disp_edit_fill,
truncchar=options.disp_truncator)
editargs.update(kwargs) # update with user-specified args
r = vd.editText(y, x, w, **editargs)
if rowidx >= 0: # if not header
r = col.type(r) # convert input to column type, let exceptions be raised
return r
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612075960.0
visidata-2.2.1/visidata/_open.py 0000660 0001750 0001750 00000007341 00000000000 017272 0 ustar 00kefala kefala 0000000 0000000 from visidata import *
__all__ = ['open_txt']
option('filetype', '', 'specify file type', replay=True)
@VisiData.api
def inputFilename(vd, prompt, *args, **kwargs):
return vd.input(prompt, type="filename", *args, completer=_completeFilename, **kwargs)
@VisiData.api
def inputPath(vd, *args, **kwargs):
return Path(vd.inputFilename(*args, **kwargs))
def _completeFilename(val, state):
i = val.rfind('/')
if i < 0: # no /
base = ''
partial = val
elif i == 0: # root /
base = '/'
partial = val[1:]
else:
base = val[:i]
partial = val[i+1:]
files = []
for f in os.listdir(Path(base or '.')):
if f.startswith(partial):
files.append(os.path.join(base, f))
files.sort()
return files[state%len(files)]
@VisiData.api
def openPath(vd, p, filetype=None, create=False):
'''Call ``open_(p)`` or ``openurl_(p, filetype)``. Return constructed but unloaded sheet of appropriate type.
If True, *create* will return a new, blank **Sheet** if file does not exist.'''
if p.scheme and not p.fp: # isinstance(p, UrlPath):
openfunc = 'openurl_' + p.scheme
try:
return vd.getGlobals()[openfunc](p, filetype=filetype)
except KeyError:
vd.fail(f'no loader for url scheme: {p.scheme}')
if not filetype:
if p.is_dir():
filetype = 'dir'
else:
filetype = p.ext or options.filetype or 'txt'
if not p.exists():
if not create:
return None
vd.warning('%s does not exist, creating new sheet' % p)
return vd.newSheet(p.name, 1, source=p)
filetype = filetype.lower()
openfunc = getattr(vd, 'open_' + filetype, vd.getGlobals().get('open_' + filetype))
if not openfunc:
vd.warning('unknown "%s" filetype' % filetype)
filetype = 'txt'
openfunc = vd.getGlobals().get('open_txt')
vd.status('opening %s as %s' % (p.given, filetype))
return openfunc(p)
@VisiData.global_api
def openSource(vd, p, filetype=None, create=False, **kwargs):
'''Return unloaded sheet object for *p* opened as the given *filetype* and with *kwargs* as option overrides. *p* can be a Path or a string (filename, url, or "-" for stdin).
when true, *create* will return a blank sheet, if file does not exist.'''
if not filetype:
filetype = options.getonly('filetype', 'global', '')
vs = None
if isinstance(p, str):
if '://' in p:
vs = vd.openPath(Path(p), filetype=filetype) # convert to Path and recurse
elif p == '-':
vs = vd.openPath(Path('-', fp=vd._stdin), filetype=filetype)
else:
vs = vd.openPath(Path(p), filetype=filetype, create=create) # convert to Path and recurse
else:
vs = vs or vd.openPath(p, filetype=filetype, create=create)
for optname, optval in kwargs.items():
vs.options[optname] = optval
return vs
#### enable external addons
def open_txt(p):
'Create sheet from `.txt` file at Path `p`, checking whether it is TSV.'
with p.open_text() as fp:
if options.delimiter in next(fp): # peek at the first line
return open_tsv(p) # TSV often have .txt extension
return TextSheet(p.name, source=p)
@VisiData.api
def loadInternalSheet(vd, cls, p, **kwargs):
'Load internal sheet of given class. Internal sheets are always tsv.'
vs = cls(p.name, source=p, **kwargs)
options._set('encoding', 'utf8', vs)
if p.exists():
vd.sheets.insert(0, vs)
vs.reload.__wrapped__(vs)
vd.sheets.pop(0)
return vs
BaseSheet.addCommand('o', 'open-file', 'vd.push(openSource(inputFilename("open: "), create=True))', 'open input in VisiData')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612075960.0
visidata-2.2.1/visidata/_types.py 0000660 0001750 0001750 00000015164 00000000000 017477 0 ustar 00kefala kefala 0000000 0000000 # VisiData uses Python native int, float, str, and adds simple date, currency, and anytype.
import collections
import functools
import datetime
import locale
from visidata import option, options, TypedWrapper, vd, VisiData
#__all__ = ['anytype', 'vdtype', ]
option('disp_currency_fmt', '%.02f', 'default fmtstr to format for currency values', replay=True)
option('disp_float_fmt', '{:.02f}', 'default fmtstr to format for float values', replay=True)
option('disp_int_fmt', '{:.0f}', 'default fmtstr to format for int values', replay=True)
option('disp_date_fmt','%Y-%m-%d', 'default fmtstr to strftime for date values', replay=True)
try:
import dateutil.parser
except ImportError:
pass
# VisiDataType .typetype are e.g. int, float, str, and used internally in these ways:
#
# o = typetype(val) # for interpreting raw value
# o = typetype(str) # for conversion from string (when setting)
# o = typetype() # for default value to be used when conversion fails
#
# The resulting object o must be orderable and convertible to a string for display and certain outputs (like csv).
#
# .icon is a single character that appears in the notes field of cells and column headers.
# .formatter(fmtstr, typedvalue) returns a string of the formatted typedvalue according to fmtstr.
# .fmtstr is the default fmtstr passed to .formatter.
def anytype(r=None):
'minimalist "any" passthrough type'
return r
anytype.__name__ = ''
def numericFormatter(fmtstr, typedval):
try:
fmtstr = fmtstr or options['disp_'+type(typedval).__name__+'_fmt']
if fmtstr[0] == '%':
return locale.format_string(fmtstr, typedval, grouping=False)
else:
return fmtstr.format(typedval)
except ValueError:
return str(typedval)
vd.si_prefixes='p n u m . kK M G T P Q'.split()
def floatsi(*args):
if not args:
return 0.0
if not isinstance(args[0], str):
return args[0]
s=args[0].strip()
for i, p in enumerate(vd.si_prefixes):
if s[-1] in p:
return float(s[:-1]) * (1000 ** (i-4))
return float(s)
def SIFormatter(fmtstr, val):
level = 4
if val != 0:
while abs(val) > 1000:
val /= 1000
level += 1
while abs(val) < 0.001:
val *= 1000
level -= 1
return numericFormatter(fmtstr, val) + (vd.si_prefixes[level][0] if level != 4 else '')
class VisiDataType:
'Register *typetype* in the typemap.'
def __init__(self, typetype=None, icon=None, fmtstr='', formatter=numericFormatter, key='', name=None):
self.typetype = typetype or anytype # int or float or other constructor
self.name = name or getattr(typetype, '__name__', str(typetype))
self.icon = icon # show in rightmost char of column
self.fmtstr = fmtstr
self.formatter = formatter
self.key = key
@VisiData.api
def addType(vd, typetype=None, icon=None, fmtstr='', formatter=numericFormatter, key='', name=None):
'''Add type to type map.
- *typetype*: actual type class *TYPE* above
- *icon*: unicode character in column header
- *fmtstr*: format string to use if fmtstr not given
- *formatter*: formatting function to call as ``formatter(fmtstr, typedvalue)``
'''
t = VisiDataType(typetype=typetype, icon=icon, fmtstr=fmtstr, formatter=formatter, key=key, name=name)
if typetype:
vd.typemap[typetype] = t
return t
vdtype = vd.addType
# typemap [vtype] -> VisiDataType
vd.typemap = {}
@VisiData.api
def getType(vd, typetype):
return vd.typemap.get(typetype) or VisiDataType()
vdtype(None, '∅')
vdtype(anytype, '', formatter=lambda _,v: str(v))
vdtype(str, '~', formatter=lambda _,v: v)
vdtype(int, '#')
vdtype(float, '%')
vdtype(dict, '')
vdtype(list, '')
@VisiData.api
def isNumeric(vd, col):
return col.type in (int,vlen,float,currency,date,floatsi,floatlocale)
##
floatchars='+-0123456789.'
def currency(*args):
'dirty float (strip non-numeric characters)'
if args and isinstance(args[0], str):
args = [''.join(ch for ch in args[0] if ch in floatchars)]
return float(*args)
def floatlocale(*args):
'Calculate float() using system locale set in LC_NUMERIC.'
if not args:
return 0.0
return locale.atof(*args)
class vlen(int):
def __new__(cls, v=0):
if isinstance(v, (vlen, int, float)):
return super(vlen, cls).__new__(cls, v)
else:
return super(vlen, cls).__new__(cls, len(v))
def __len__(self):
return self
class date(datetime.datetime):
'datetime wrapper, constructed from time_t or from str with dateutil.parse'
def __new__(cls, *args, **kwargs):
'datetime is immutable so needs __new__ instead of __init__'
if not args:
return datetime.datetime.now()
elif len(args) > 1:
return super().__new__(cls, *args, **kwargs)
s = args[0]
if isinstance(s, int) or isinstance(s, float):
r = datetime.datetime.fromtimestamp(s)
elif isinstance(s, str):
r = dateutil.parser.parse(s)
elif isinstance(s, (datetime.datetime, datetime.date)):
r = s
else:
raise Exception('invalid type for date %s' % type(s).__name__)
t = r.timetuple()
ms = getattr(r, 'microsecond', 0)
tzinfo = getattr(r, 'tzinfo', None)
return super().__new__(cls, *t[:6], microsecond=ms, tzinfo=tzinfo, **kwargs)
def __str__(self):
return self.strftime(options.disp_date_fmt)
def __float__(self):
return self.timestamp()
def __radd__(self, n):
return self.__add__(n)
def __add__(self, n):
'add n days (int or float) to the date'
if isinstance(n, (int, float)):
n = datetime.timedelta(days=n)
return date(super().__add__(n))
def __sub__(self, n):
'subtract n days (int or float) from the date. or subtract another date for a timedelta'
if isinstance(n, (int, float)):
n = datetime.timedelta(days=n)
elif isinstance(n, (date, datetime.datetime)):
return datedelta(super().__sub__(n).total_seconds()/(24*60*60))
return super().__sub__(n)
class datedelta(datetime.timedelta):
def __float__(self):
return self.total_seconds()
vdtype(vlen, '♯', '%.0f')
vdtype(floatlocale, '%')
vdtype(date, '@', '', formatter=lambda fmtstr,val: val.strftime(fmtstr or options.disp_date_fmt))
vdtype(currency, '$')
vdtype(floatsi, '‱', formatter=SIFormatter)
# simple constants, for expressions like 'timestamp+15*minutes'
years=365.25
months=30.0
weeks=7.0
days=1.0
hours=days/24
minutes=days/(24*60)
seconds=days/(24*60*60)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612075960.0
visidata-2.2.1/visidata/aggregators.py 0000660 0001750 0001750 00000013740 00000000000 020477 0 ustar 00kefala kefala 0000000 0000000 import math
import functools
import collections
from statistics import mode, stdev
from visidata import Progress, Column
from visidata import *
@Column.api
def getValueRows(self, rows):
'Generate (value, row) for each row in *rows* at this column, excluding null and error values.'
f = self.sheet.isNullFunc()
for r in Progress(rows, 'calculating'):
try:
v = self.getTypedValue(r)
if not f(v):
yield v, r
except Exception:
pass
@Column.api
def getValues(self, rows):
'Generate value for each row in *rows* at this column, excluding null and error values.'
for v, r in self.getValueRows(rows):
yield v
vd.aggregators = collections.OrderedDict() # [aggname] -> annotated func, or list of same
Column.init('aggregators', list)
def aggstr_get(col):
'A space-separated names of aggregators on this column.'
return ' '.join(aggr.name for aggr in col.aggregators)
def aggstr_set(col, v):
col.aggregators = list(vd.aggregators[k] for k in (v or '').split())
Column.aggstr = property(aggstr_get, aggstr_set)
class Aggregator:
def __init__(self, name, type, func, helpstr='foo'):
'Define aggregator `name` that calls func(col, rows)'
self.type = type
self.func = func
self.helpstr = helpstr
self.name = name
def __call__(self, *args, **kwargs):
return self.func(*args, **kwargs)
_defaggr = Aggregator
@VisiData.global_api
def aggregator(vd, name, func, helpstr='', *args, type=None):
'Define simple aggregator *name* that calls ``func(values, *args)`` to aggregate *values*. Use *type* to force the default type of the aggregated column.'
def _func(col, rows): # wrap builtins so they can have a .type
vals = list(col.getValues(rows))
try:
return func(vals, *args)
except Exception as e:
if len(vals) == 0:
return None
return e
vd.aggregators[name] = _defaggr(name, type, _func, helpstr)
## specific aggregator implementations
def mean(vals):
vals = list(vals)
if vals:
return float(sum(vals))/len(vals)
def median(values):
L = sorted(values)
return L[len(L)//2]
# http://code.activestate.com/recipes/511478-finding-the-percentile-of-the-values/
def _percentile(N, percent, key=lambda x:x):
"""
Find the percentile of a list of values.
@parameter N - is a list of values. Note N MUST BE already sorted.
@parameter percent - a float value from 0.0 to 1.0.
@parameter key - optional key function to compute value from each element of N.
@return - the percentile of the values
"""
if not N:
return None
k = (len(N)-1) * percent
f = math.floor(k)
c = math.ceil(k)
if f == c:
return key(N[int(k)])
d0 = key(N[int(f)]) * (c-k)
d1 = key(N[int(c)]) * (k-f)
return d0+d1
@functools.lru_cache(100)
def percentile(pct, helpstr=''):
return _defaggr('p%s'%pct, None, lambda col,rows,pct=pct: _percentile(sorted(col.getValues(rows)), pct/100), helpstr)
def quantiles(q, helpstr):
return [percentile(round(100*i/q), helpstr) for i in range(1, q)]
vd.aggregator('min', min, 'minimum value')
vd.aggregator('max', max, 'maximum value')
vd.aggregator('avg', mean, 'arithmetic mean of values', type=float)
vd.aggregator('mean', mean, 'arithmetic mean of values', type=float)
vd.aggregator('median', median, 'median of values')
vd.aggregator('mode', mode, 'mode of values')
vd.aggregator('sum', sum, 'sum of values')
vd.aggregator('distinct', set, 'distinct values', type=vlen)
vd.aggregator('count', lambda values: sum(1 for v in values), 'number of values', type=int)
vd.aggregator('list', list, 'list of values')
vd.aggregator('stdev', stdev, 'standard deviation of values', type=float)
vd.aggregators['q3'] = quantiles(3, 'tertiles (33/66th pctile)')
vd.aggregators['q4'] = quantiles(4, 'quartiles (25/50/75th pctile)')
vd.aggregators['q5'] = quantiles(5, 'quintiles (20/40/60/80th pctiles)')
vd.aggregators['q10'] = quantiles(10, 'deciles (10/20/30/40/50/60/70/80/80th pctiles)')
# returns keys of the row with the max value
vd.aggregators['keymax'] = _defaggr('keymax', anytype, lambda col, rows: col.sheet.rowkey(max(col.getValueRows(rows))[1]), 'key of the maximum value')
ColumnsSheet.columns += [ColumnAttr('aggregators','aggstr')]
@Sheet.api
def addAggregators(sheet, cols, aggrnames):
'Add each aggregator in list of *aggrnames* to each of *cols*.'
for aggrname in aggrnames:
aggrs = vd.aggregators.get(aggrname)
aggrs = aggrs if isinstance(aggrs, list) else [aggrs]
for aggr in aggrs:
for c in cols:
if not hasattr(c, 'aggregators'):
c.aggregators = []
if aggr and aggr not in c.aggregators:
c.aggregators += [aggr]
@Column.api
def aggname(col, agg):
'Consistent formatting of the name of given aggregator for this column. e.g. "col1_sum"'
return '%s_%s' % (col.name, agg.name)
@Column.api
@asyncthread
def show_aggregate(col, agg, rows):
'Show aggregated value in status, and add to memory.'
aggval = agg(col, rows)
typedval = wrapply(agg.type or col.type, aggval)
dispval = col.format(typedval)
vd.status(dispval)
@VisiData.property
def aggregator_choices(vd):
return [
{'key': agg, 'desc': v[0].helpstr if isinstance(v, list) else v.helpstr} for agg, v in vd.aggregators.items()
]
vd.addGlobals(globals())
Sheet.addCommand('+', 'aggregate-col', 'addAggregators([cursorCol], chooseMany(aggregator_choices))', 'add aggregator to current column')
Sheet.addCommand('z+', 'show-aggregate', 'for agg in chooseMany(aggregator_choices): cursorCol.show_aggregate(aggregators[agg], selectedRows or rows)', 'display result of aggregator over values in selected rows for current column')
ColumnsSheet.addCommand('g+', 'aggregate-cols', 'addAggregators(selectedRows or source[0].nonKeyVisibleCols, chooseMany(aggregator_choices))', 'add aggregators to selected source columns')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1609740788.0
visidata-2.2.1/visidata/basesheet.py 0000660 0001750 0001750 00000017112 00000000000 020132 0 ustar 00kefala kefala 0000000 0000000 import os
import visidata
from visidata import Extensible, VisiData, vd, EscapeException
from unittest import mock
UNLOADED = tuple() # sentinel for a sheet not yet loaded for the first time
vd.beforeExecHooks = [] # func(sheet, cmd, args, keystrokes) called before the exec()
class LazyChainMap:
'provides a lazy mapping to obj attributes. useful when some attributes are expensive properties.'
def __init__(self, *objs):
self.locals = {}
self.objs = {} # [k] -> obj
for obj in objs:
for k in dir(obj):
if k not in self.objs:
self.objs[k] = obj
def keys(self):
return list(self.objs.keys()) # sum(set(dir(obj)) for obj in self.objs))
def clear(self):
self.locals.clear()
def __getitem__(self, k):
obj = self.objs.get(k, None)
if obj:
return getattr(obj, k)
return self.locals[k]
def __setitem__(self, k, v):
obj = self.objs.get(k, None)
if obj:
return setattr(obj, k, v)
self.locals[k] = v
class BaseSheet(Extensible):
'Base class for all sheet types.'
_rowtype = object # callable (no parms) that returns new empty item
_coltype = None # callable (no parms) that returns new settable view into that item
rowtype = 'objects' # one word, plural, describing the items
precious = True # False for a few discardable metasheets
defer = False # False for not deferring changes until save
@visidata.classproperty
def class_options(cls):
return vd.OptionsObject(vd._options, obj=cls)
@property
def options(self):
return vd.OptionsObject(vd._options, obj=self)
def __init__(self, *names, **kwargs):
self._name = None
self.name = self.options.name_joiner.join(str(x) for x in names)
self.source = None
self.rows = UNLOADED # list of opaque objects
self._scr = mock.MagicMock(__bool__=mock.Mock(return_value=False)) # disable curses in batch mode
self.__dict__.update(kwargs)
def __lt__(self, other):
if self.name != other.name:
return self.name < other.name
else:
return id(self) < id(other)
def __copy__(self):
'Return shallow copy of sheet.'
cls = self.__class__
ret = cls.__new__(cls)
ret.__dict__.update(self.__dict__)
ret.precious = True # copies can be precious even if originals aren't
return ret
def __bool__(self):
'an instantiated Sheet always tests true'
return True
def __len__(self):
'Number of elements on this sheet.'
return self.nRows
def __str__(self):
return self.name
@property
def nRows(self):
'Number of rows on this sheet. Override in subclass.'
return 0
def __contains__(self, vs):
if self.source is vs:
return True
if isinstance(self.source, BaseSheet):
return vs in self.source
return False
@property
def windowHeight(self):
'Height of the current sheet window, in terminal lines.'
return self._scr.getmaxyx()[0] if self._scr else 25
@property
def windowWidth(self):
'Width of the current sheet window, in single-width characters.'
return self._scr.getmaxyx()[1] if self._scr else 80
def execCommand(self, cmd, args='', vdglobals=None, keystrokes=None):
"""Execute `cmd` tuple with `vdglobals` as globals and this sheet's attributes as locals. Return True if user cancelled.
`cmd` can be a longname, a keystroke, or a Command object."""
cmd = self.getCommand(cmd or keystrokes)
if not cmd:
if keystrokes:
vd.debug('no command "%s"' % keystrokes)
return True
escaped = False
err = ''
if vdglobals is None:
vdglobals = vd.getGlobals()
self.sheet = self
try:
for hookfunc in vd.beforeExecHooks:
hookfunc(self, cmd, '', keystrokes)
code = compile(cmd.execstr, cmd.longname, 'exec')
vd.debug(cmd.longname)
exec(code, vdglobals, LazyChainMap(vd, self))
except EscapeException as e: # user aborted
vd.warning(str(e))
escaped = True
except Exception as e:
vd.debug(cmd.execstr)
err = vd.exceptionCaught(e)
escaped = True
try:
if vd.cmdlog:
# sheet may have changed
vd.cmdlog.afterExecSheet(vd.sheets[0] if vd.sheets else None, escaped, err)
except Exception as e:
vd.exceptionCaught(e)
self.checkCursorNoExceptions()
vd.clearCaches()
return escaped
@property
def name(self):
'Name of this sheet.'
try:
return self._name
except AttributeError:
return self.rowtype
@name.setter
def name(self, name):
'Set name without spaces.'
if self._name:
vd.addUndo(setattr, self, '_name', self._name)
self._name = visidata.maybe_clean(str(name), self)
def recalc(self):
'Clear any calculated value caches.'
pass
def draw(self, scr):
'Draw the sheet on the terminal window *scr*. Overrideable.'
vd.error('no draw')
def refresh(self):
'Clear the terminal screen and let the next draw cycle redraw everything.'
self._scr.clear()
self._scr.refresh()
def ensureLoaded(self):
'Call ``reload()`` if not already loaded.'
if self.rows is UNLOADED:
self.rows = [] # prevent auto-reload from running twice
return self.reload() # likely launches new thread
def reload(self):
'Load sheet from *self.source*. Override in subclass.'
vd.error('no reload')
@property
def cursorRow(self):
'The row object at the row cursor. Overrideable.'
return None
def checkCursor(self):
'Check cursor and fix if out-of-bounds. Overrideable.'
pass
def checkCursorNoExceptions(self):
try:
return self.checkCursor()
except Exception as e:
vd.exceptionCaught(e)
def evalExpr(self, expr, **kwargs):
'Evaluate Python expression *expr* in the context of *kwargs* (may vary by sheet type).'
return eval(expr, vd.getGlobals(), None)
@VisiData.api
def redraw(vd):
'Clear the terminal screen and let the next draw cycle recreate the windows and redraw everything.'
vd.scrFull.clear()
vd.win1.clear()
vd.win2.clear()
vd.setWindows(vd.scrFull)
@VisiData.property
def sheet(self):
'the top sheet on the stack'
return self.sheets[0] if self.sheets else None
@VisiData.api
def isLongname(self, ks):
'Return True if *ks* is a longname.'
return ('-' in ks) and (ks[-1] != '-') or (len(ks) > 3 and ks.islower())
@VisiData.api
def getSheet(vd, sheetname):
'Return Sheet from the sheet stack. *sheetname* can be a sheet name or a sheet number indexing directly into ``vd.sheets``.'
if isinstance(sheetname, BaseSheet):
return sheetname
matchingSheets = [x for x in vd.sheets if x.name == sheetname]
if matchingSheets:
if len(matchingSheets) > 1:
vd.warning('more than one sheet named "%s"' % sheetname)
return matchingSheets[0]
try:
sheetidx = int(sheetname)
return vd.sheets[sheetidx]
except ValueError:
pass
if sheetname == 'options':
vs = self.optionsSheet
vs.reload()
vs.vd = vd
return vs
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608541899.0
visidata-2.2.1/visidata/canvas.py 0000660 0001750 0001750 00000072220 00000000000 017443 0 ustar 00kefala kefala 0000000 0000000
from collections import defaultdict, Counter
from visidata import *
# see www/design/graphics.md
option('show_graph_labels', True, 'show axes and legend on graph')
theme('plot_colors', 'green red yellow cyan magenta white 38 136 168', 'list of distinct colors to use for plotting distinct objects')
theme('disp_pixel_random', False, 'randomly choose attr from set of pixels instead of most common')
option('zoom_incr', 2.0, 'amount to multiply current zoomlevel when zooming')
theme('color_graph_hidden', '238 blue', 'color of legend for hidden attribute')
theme('color_graph_selected', 'bold', 'color of selected graph points')
class Point:
def __init__(self, x, y):
self.x = x
self.y = y
def __repr__(self):
if isinstance(self.x, int):
return '(%d,%d)' % (self.x, self.y)
else:
return '(%.02f,%.02f)' % (self.x, self.y)
@property
def xy(self):
return (self.x, self.y)
class Box:
def __init__(self, x, y, w=0, h=0):
self.xmin = x
self.ymin = y
self.w = w
self.h = h
def __repr__(self):
return '[%s+%s,%s+%s]' % (self.xmin, self.w, self.ymin, self.h)
@property
def xymin(self):
return Point(self.xmin, self.ymin)
@property
def xmax(self):
return self.xmin + self.w
@property
def ymax(self):
return self.ymin + self.h
@property
def center(self):
return Point(self.xcenter, self.ycenter)
@property
def xcenter(self):
return self.xmin + self.w/2
@property
def ycenter(self):
return self.ymin + self.h/2
def contains(self, x, y):
return x >= self.xmin and \
x < self.xmax and \
y >= self.ymin and \
y < self.ymax
def BoundingBox(x1, y1, x2, y2):
return Box(min(x1, x2), min(y1, y2), abs(x2-x1), abs(y2-y1))
def clipline(x1, y1, x2, y2, xmin, ymin, xmax, ymax):
'Liang-Barsky algorithm, returns [xn1,yn1,xn2,yn2] of clipped line within given area, or None'
dx = x2-x1
dy = y2-y1
pq = [
(-dx, x1-xmin), # left
( dx, xmax-x1), # right
(-dy, y1-ymin), # bottom
( dy, ymax-y1), # top
]
u1, u2 = 0, 1
for p, q in pq:
if p < 0: # from outside to inside
u1 = max(u1, q/p)
elif p > 0: # from inside to outside
u2 = min(u2, q/p)
else: # p == 0: # parallel to bbox
if q < 0: # completely outside bbox
return None
if u1 > u2: # completely outside bbox
return None
xn1 = x1 + dx*u1
yn1 = y1 + dy*u1
xn2 = x1 + dx*u2
yn2 = y1 + dy*u2
return xn1, yn1, xn2, yn2
def iterline(x1, y1, x2, y2):
'Yields (x, y) coords of line from (x1, y1) to (x2, y2)'
xdiff = abs(x2-x1)
ydiff = abs(y2-y1)
xdir = 1 if x1 <= x2 else -1
ydir = 1 if y1 <= y2 else -1
r = math.ceil(max(xdiff, ydiff))
if r == 0: # point, not line
yield x1, y1
else:
x, y = math.floor(x1), math.floor(y1)
i = 0
while i < r:
x += xdir * xdiff / r
y += ydir * ydiff / r
yield x, y
i += 1
def anySelected(vs, rows):
for r in rows:
if vs.isSelected(r):
return True
# - width/height are exactly equal to the number of pixels displayable, and can change at any time.
# - needs to refresh from source on resize
class Plotter(BaseSheet):
'pixel-addressable display of entire terminal with (x,y) integer pixel coordinates'
columns=[Column('_')] # to eliminate errors outside of draw()
rowtype='pixels'
def __init__(self, *names, **kwargs):
super().__init__(*names, **kwargs)
self.labels = [] # (x, y, text, attr, row)
self.hiddenAttrs = set()
self.needsRefresh = False
self.resetCanvasDimensions(self.windowHeight, self.windowWidth)
@property
def nRows(self):
return (self.plotwidth* self.plotheight)
def resetCanvasDimensions(self, windowHeight, windowWidth):
'sets total available canvas dimensions to (windowHeight, windowWidth) (in char cells)'
self.plotwidth = windowWidth*2
self.plotheight = (windowHeight-1)*4 # exclude status line
# pixels[y][x] = { attr: list(rows), ... }
self.pixels = [[defaultdict(list) for x in range(self.plotwidth)] for y in range(self.plotheight)]
def plotpixel(self, x, y, attr=0, row=None):
self.pixels[y][x][attr].append(row)
def plotline(self, x1, y1, x2, y2, attr=0, row=None):
for x, y in iterline(x1, y1, x2, y2):
self.plotpixel(math.ceil(x), math.ceil(y), attr, row)
def plotlabel(self, x, y, text, attr=0, row=None):
self.labels.append((x, y, text, attr, row))
def plotlegend(self, i, txt, attr=0, width=15):
self.plotlabel(self.plotwidth-width*2, i*4, txt, attr)
@property
def plotterCursorBox(self):
'Returns pixel bounds of cursor as a Box. Override to provide a cursor.'
return Box(0,0,0,0)
@property
def plotterMouse(self):
return Point(*self.plotterFromTerminalCoord(self.mouseX, self.mouseY))
def plotterFromTerminalCoord(self, x, y):
return x*2, y*4
def getPixelAttrRandom(self, x, y):
'weighted-random choice of attr at this pixel.'
c = list(attr for attr, rows in self.pixels[y][x].items()
for r in rows if attr and attr not in self.hiddenAttrs)
return random.choice(c) if c else 0
def getPixelAttrMost(self, x, y):
'most common attr at this pixel.'
r = self.pixels[y][x]
c = [(len(rows), attr, rows) for attr, rows in list(r.items()) if attr and attr not in self.hiddenAttrs]
if not c:
return 0
_, attr, rows = max(c)
if isinstance(self.source, BaseSheet) and anySelected(self.source, rows):
attr = update_attr(ColorAttr(attr, 0, 8, attr), colors.color_graph_selected, 10).attr
return attr
def hideAttr(self, attr, hide=True):
if hide:
self.hiddenAttrs.add(attr)
else:
self.hiddenAttrs.remove(attr)
self.plotlegends()
def rowsWithin(self, bbox):
'return list of deduped rows within bbox'
ret = {}
for y in range(bbox.ymin, bbox.ymax+1):
for x in range(bbox.xmin, min(len(self.pixels[y]), bbox.xmax+1)):
for attr, rows in self.pixels[y][x].items():
if attr not in self.hiddenAttrs:
for r in rows:
ret[self.source.rowid(r)] = r
return list(ret.values())
def draw(self, scr):
windowHeight, windowWidth = scr.getmaxyx()
if self.needsRefresh:
self.render(windowHeight, windowWidth)
if self.pixels:
cursorBBox = self.plotterCursorBox
getPixelAttr = self.getPixelAttrRandom if options.disp_pixel_random else self.getPixelAttrMost
for char_y in range(0, self.plotheight//4):
for char_x in range(0, self.plotwidth//2):
block_attrs = [
getPixelAttr(char_x*2 , char_y*4 ),
getPixelAttr(char_x*2 , char_y*4+1),
getPixelAttr(char_x*2 , char_y*4+2),
getPixelAttr(char_x*2+1, char_y*4 ),
getPixelAttr(char_x*2+1, char_y*4+1),
getPixelAttr(char_x*2+1, char_y*4+2),
getPixelAttr(char_x*2 , char_y*4+3),
getPixelAttr(char_x*2+1, char_y*4+3),
]
pow2 = 1
braille_num = 0
for c in block_attrs:
if c:
braille_num += pow2
pow2 *= 2
if braille_num != 0:
attr = Counter(c for c in block_attrs if c).most_common(1)[0][0]
else:
attr = 0
if cursorBBox.contains(char_x*2, char_y*4) or \
cursorBBox.contains(char_x*2+1, char_y*4+3):
attr = update_attr(ColorAttr(attr, 0, 0, attr), colors.color_current_row).attr
if attr:
scr.addstr(char_y, char_x, chr(0x2800+braille_num), attr)
def _mark_overlap_text(labels, textobj):
def _overlaps(a, b):
a_x1, _, a_txt, _, _ = a
b_x1, _, b_txt, _, _ = b
a_x2 = a_x1 + len(a_txt)
b_x2 = b_x1 + len(b_txt)
if a_x1 < b_x1 < a_x2 or a_x1 < b_x2 < a_x2 or \
b_x1 < a_x1 < b_x2 or b_x1 < a_x2 < b_x2:
return True
else:
return False
label_fldraw = [textobj, True]
labels.append(label_fldraw)
for o in labels:
if _overlaps(o[0], textobj):
o[1] = False
label_fldraw[1] = False
if options.show_graph_labels:
labels_by_line = defaultdict(list) # y -> text labels
for pix_x, pix_y, txt, attr, row in self.labels:
if attr in self.hiddenAttrs:
continue
if row is not None:
pix_x -= len(txt)/2*2
char_y = int(pix_y/4)
char_x = int(pix_x/2)
o = (char_x, char_y, txt, attr, row)
_mark_overlap_text(labels_by_line[char_y], o)
for line in labels_by_line.values():
for o, fldraw in line:
if fldraw:
char_x, char_y, txt, attr, row = o
clipdraw(scr, char_y, char_x, txt, attr, len(txt))
# - has a cursor, of arbitrary position and width/height (not restricted to current zoom)
class Canvas(Plotter):
'zoomable/scrollable virtual canvas with (x,y) coordinates in arbitrary units'
rowtype = 'plots'
leftMarginPixels = 10*2
rightMarginPixels = 4*2
topMarginPixels = 0
bottomMarginPixels = 1*4 # reserve bottom line for x axis
def __init__(self, *names, **kwargs):
super().__init__(*names, **kwargs)
self.canvasBox = None # bounding box of entire canvas, in canvas units
self.visibleBox = None # bounding box of visible canvas, in canvas units
self.cursorBox = None # bounding box of cursor, in canvas units
self.aspectRatio = 0.0
self.xzoomlevel = 1.0
self.yzoomlevel = 1.0
self.needsRefresh = False
self.polylines = [] # list of ([(canvas_x, canvas_y), ...], attr, row)
self.gridlabels = [] # list of (grid_x, grid_y, label, attr, row)
self.legends = collections.OrderedDict() # txt: attr (visible legends only)
self.plotAttrs = {} # key: attr (all keys, for speed)
self.reset()
@property
def nRows(self):
return len(self.polylines)
def reset(self):
'clear everything in preparation for a fresh reload()'
self.polylines.clear()
self.legends.clear()
self.legendwidth = 0
self.plotAttrs.clear()
self.unusedAttrs = list(colors[colorname.translate(str.maketrans('_', ' '))] for colorname in options.plot_colors.split())
def plotColor(self, k):
attr = self.plotAttrs.get(k, None)
if attr is None:
if len(self.unusedAttrs) > 1:
attr = self.unusedAttrs.pop(0)
legend = ' '.join(str(x) for x in k)
else:
attr = self.unusedAttrs[0]
legend = '[other]'
self.legendwidth = max(self.legendwidth, len(legend))
self.legends[legend] = attr
self.plotAttrs[k] = attr
self.plotlegends()
return attr
def resetCanvasDimensions(self, windowHeight, windowWidth):
super().resetCanvasDimensions(windowHeight, windowWidth)
self.plotviewBox = BoundingBox(self.leftMarginPixels, self.topMarginPixels,
self.plotwidth-self.rightMarginPixels, self.plotheight-self.bottomMarginPixels-1)
@property
def statusLine(self):
return 'canvas %s visible %s cursor %s' % (self.canvasBox, self.visibleBox, self.cursorBox)
@property
def canvasMouse(self):
return self.canvasFromPlotterCoord(self.plotterMouse.x, self.plotterMouse.y)
def canvasFromPlotterCoord(self, plotter_x, plotter_y):
return Point(self.visibleBox.xmin + (plotter_x-self.plotviewBox.xmin)/self.xScaler, self.visibleBox.ymin + (plotter_y-self.plotviewBox.ymin)/self.yScaler)
def canvasFromTerminalCoord(self, x, y):
return self.canvasFromPlotterCoord(*self.plotterFromTerminalCoord(x, y))
def setCursorSize(self, p):
'sets width based on diagonal corner p'
self.cursorBox = BoundingBox(self.cursorBox.xmin, self.cursorBox.ymin, p.x, p.y)
self.cursorBox.w = max(self.cursorBox.w, self.canvasCharWidth)
self.cursorBox.h = max(self.cursorBox.h, self.canvasCharHeight)
@property
def canvasCharWidth(self):
'Width in canvas units of a single char in the terminal'
return self.visibleBox.w*2/self.plotviewBox.w
@property
def canvasCharHeight(self):
'Height in canvas units of a single char in the terminal'
return self.visibleBox.h*4/self.plotviewBox.h
@property
def plotterVisibleBox(self):
return BoundingBox(self.scaleX(self.visibleBox.xmin),
self.scaleY(self.visibleBox.ymin),
self.scaleX(self.visibleBox.xmax),
self.scaleY(self.visibleBox.ymax))
@property
def plotterCursorBox(self):
if self.cursorBox is None:
return Box(0,0,0,0)
return BoundingBox(self.scaleX(self.cursorBox.xmin),
self.scaleY(self.cursorBox.ymin),
self.scaleX(self.cursorBox.xmax),
self.scaleY(self.cursorBox.ymax))
def point(self, x, y, attr=0, row=None):
self.polylines.append(([(x, y)], attr, row))
def line(self, x1, y1, x2, y2, attr=0, row=None):
self.polylines.append(([(x1, y1), (x2, y2)], attr, row))
def polyline(self, vertexes, attr=0, row=None):
'adds lines for (x,y) vertexes of a polygon'
self.polylines.append((vertexes, attr, row))
def polygon(self, vertexes, attr=0, row=None):
'adds lines for (x,y) vertexes of a polygon'
self.polylines.append((vertexes + [vertexes[0]], attr, row))
def qcurve(self, vertexes, attr=0, row=None):
'Draw quadratic curve from vertexes[0] to vertexes[2] with control point at vertexes[1]'
if len(vertexes) != 3:
vd.fail('need exactly 3 points for qcurve (got %d)' % len(vertexes))
x1, y1 = vertexes[0]
x2, y2 = vertexes[1]
x3, y3 = vertexes[2]
self.point(x1, y1, attr, row)
self._recursive_bezier(x1, y1, x2, y2, x3, y3, attr, row)
self.point(x3, y3, attr, row)
def _recursive_bezier(self, x1, y1, x2, y2, x3, y3, attr, row, level=0):
'from http://www.antigrain.com/research/adaptive_bezier/'
m_approximation_scale = 10.0
m_distance_tolerance = (0.5 / m_approximation_scale) ** 2
m_angle_tolerance = 1 * 2*math.pi/360 # 15 degrees in rads
curve_angle_tolerance_epsilon = 0.01
curve_recursion_limit = 32
curve_collinearity_epsilon = 1e-30
if level > curve_recursion_limit:
return
# Calculate all the mid-points of the line segments
x12 = (x1 + x2) / 2
y12 = (y1 + y2) / 2
x23 = (x2 + x3) / 2
y23 = (y2 + y3) / 2
x123 = (x12 + x23) / 2
y123 = (y12 + y23) / 2
dx = x3-x1
dy = y3-y1
d = abs(((x2 - x3) * dy - (y2 - y3) * dx))
if d > curve_collinearity_epsilon:
# Regular care
if d*d <= m_distance_tolerance * (dx*dx + dy*dy):
# If the curvature doesn't exceed the distance_tolerance value, we tend to finish subdivisions.
if m_angle_tolerance < curve_angle_tolerance_epsilon:
self.point(x123, y123, attr, row)
return
# Angle & Cusp Condition
da = abs(math.atan2(y3 - y2, x3 - x2) - math.atan2(y2 - y1, x2 - x1))
if da >= math.pi:
da = 2*math.pi - da
if da < m_angle_tolerance:
# Finally we can stop the recursion
self.point(x123, y123, attr, row)
return
else:
# Collinear case
dx = x123 - (x1 + x3) / 2
dy = y123 - (y1 + y3) / 2
if dx*dx + dy*dy <= m_distance_tolerance:
self.point(x123, y123, attr, row)
return
# Continue subdivision
self._recursive_bezier(x1, y1, x12, y12, x123, y123, attr, row, level + 1)
self._recursive_bezier(x123, y123, x23, y23, x3, y3, attr, row, level + 1)
def label(self, x, y, text, attr=0, row=None):
self.gridlabels.append((x, y, text, attr, row))
def fixPoint(self, plotterPoint, canvasPoint):
'adjust visibleBox.xymin so that canvasPoint is plotted at plotterPoint'
self.visibleBox.xmin = canvasPoint.x - self.canvasW(plotterPoint.x-self.plotviewBox.xmin)
self.visibleBox.ymin = canvasPoint.y - self.canvasH(plotterPoint.y-self.plotviewBox.ymin)
self.refresh()
def zoomTo(self, bbox):
'set visible area to bbox, maintaining aspectRatio if applicable'
self.fixPoint(self.plotviewBox.xymin, bbox.xymin)
self.xzoomlevel=bbox.w/self.canvasBox.w
self.yzoomlevel=bbox.h/self.canvasBox.h
def incrZoom(self, incr):
self.xzoomlevel *= incr
self.yzoomlevel *= incr
self.resetBounds()
def resetBounds(self):
'create canvasBox and cursorBox if necessary, and set visibleBox w/h according to zoomlevels. then redisplay labels.'
if not self.canvasBox:
xmin, ymin, xmax, ymax = None, None, None, None
for vertexes, attr, row in self.polylines:
for x, y in vertexes:
if xmin is None or x < xmin: xmin = x
if ymin is None or y < ymin: ymin = y
if xmax is None or x > xmax: xmax = x
if ymax is None or y > ymax: ymax = y
self.canvasBox = BoundingBox(float(xmin or 0), float(ymin or 0), float(xmax or 1), float(ymax or 1))
if not self.visibleBox:
# initialize minx/miny, but w/h must be set first to center properly
self.visibleBox = Box(0, 0, self.plotviewBox.w/self.xScaler, self.plotviewBox.h/self.yScaler)
self.visibleBox.xmin = self.canvasBox.xcenter - self.visibleBox.w/2
self.visibleBox.ymin = self.canvasBox.ycenter - self.visibleBox.h/2
else:
self.visibleBox.w = self.plotviewBox.w/self.xScaler
self.visibleBox.h = self.plotviewBox.h/self.yScaler
if not self.cursorBox:
self.cursorBox = Box(self.visibleBox.xmin, self.visibleBox.ymin, self.canvasCharWidth, self.canvasCharHeight)
self.plotlegends()
def plotlegends(self):
# display labels
for i, (legend, attr) in enumerate(self.legends.items()):
self.addCommand(str(i+1), 'toggle-%s'%(i+1), 'hideAttr(%s, %s not in hiddenAttrs)' % (attr, attr), 'toggle display of "%s"' % legend)
if attr in self.hiddenAttrs:
attr = colors.color_graph_hidden
self.plotlegend(i, '%s:%s'%(i+1,legend), attr, width=self.legendwidth+4)
def checkCursor(self):
'override Sheet.checkCursor'
if self.visibleBox and self.cursorBox:
if self.cursorBox.h < self.canvasCharHeight:
self.cursorBox.h = self.canvasCharHeight*3/4
if self.cursorBox.w < self.canvasCharWidth:
self.cursorBox.w = self.canvasCharWidth*3/4
return False
@property
def xScaler(self):
xratio = self.plotviewBox.w/(self.canvasBox.w*self.xzoomlevel)
if self.aspectRatio:
yratio = self.plotviewBox.h/(self.canvasBox.h*self.yzoomlevel)
return self.aspectRatio*min(xratio, yratio)
else:
return xratio
@property
def yScaler(self):
yratio = self.plotviewBox.h/(self.canvasBox.h*self.yzoomlevel)
if self.aspectRatio:
xratio = self.plotviewBox.w/(self.canvasBox.w*self.xzoomlevel)
return min(xratio, yratio)
else:
return yratio
def scaleX(self, x):
'returns plotter x coordinate'
return round(self.plotviewBox.xmin+(x-self.visibleBox.xmin)*self.xScaler)
def scaleY(self, y):
'returns plotter y coordinate'
return round(self.plotviewBox.ymin+(y-self.visibleBox.ymin)*self.yScaler)
def canvasW(self, plotter_width):
'plotter X units to canvas units'
return plotter_width/self.xScaler
def canvasH(self, plotter_height):
'plotter Y units to canvas units'
return plotter_height/self.yScaler
def refresh(self):
'triggers render() on next draw()'
self.needsRefresh = True
def render(self, h, w):
'resets plotter, cancels previous render threads, spawns a new render'
self.needsRefresh = False
cancelThread(*(t for t in self.currentThreads if t.name == 'plotAll_async'))
self.labels.clear()
self.resetCanvasDimensions(h, w)
self.render_async()
@asyncthread
def render_async(self):
self.render_sync()
def render_sync(self):
'plots points and lines and text onto the Plotter'
self.resetBounds()
bb = self.visibleBox
xmin, ymin, xmax, ymax = bb.xmin, bb.ymin, bb.xmax, bb.ymax
xfactor, yfactor = self.xScaler, self.yScaler
plotxmin, plotymin = self.plotviewBox.xmin, self.plotviewBox.ymin
for vertexes, attr, row in Progress(self.polylines, 'rendering'):
if len(vertexes) == 1: # single point
x1, y1 = vertexes[0]
x1, y1 = float(x1), float(y1)
if xmin <= x1 <= xmax and ymin <= y1 <= ymax:
x = plotxmin+(x1-xmin)*xfactor
y = plotymin+(y1-ymin)*yfactor
self.plotpixel(round(x), round(y), attr, row)
continue
prev_x, prev_y = vertexes[0]
for x, y in vertexes[1:]:
r = clipline(prev_x, prev_y, x, y, xmin, ymin, xmax, ymax)
if r:
x1, y1, x2, y2 = r
x1 = plotxmin+float(x1-xmin)*xfactor
y1 = plotymin+float(y1-ymin)*yfactor
x2 = plotxmin+float(x2-xmin)*xfactor
y2 = plotymin+float(y2-ymin)*yfactor
self.plotline(x1, y1, x2, y2, attr, row)
prev_x, prev_y = x, y
for x, y, text, attr, row in Progress(self.gridlabels, 'labeling'):
self.plotlabel(self.scaleX(x), self.scaleY(y), text, attr, row)
@asyncthread
def deleteSourceRows(self, rows):
rows = list(rows)
self.source.copyRows(rows)
self.source.deleteBy(lambda r,rows=rows: r in rows)
self.reload()
Plotter.addCommand('v', 'visibility', 'options.show_graph_labels = not options.show_graph_labels', 'toggle show_graph_labels option')
Canvas.addCommand(None, 'go-left', 'sheet.cursorBox.xmin -= cursorBox.w', 'move cursor left by its width')
Canvas.addCommand(None, 'go-right', 'sheet.cursorBox.xmin += cursorBox.w', 'move cursor right by its width' )
Canvas.addCommand(None, 'go-up', 'sheet.cursorBox.ymin -= cursorBox.h', 'move cursor up by its height')
Canvas.addCommand(None, 'go-down', 'sheet.cursorBox.ymin += cursorBox.h', 'move cursor down by its height')
Canvas.addCommand(None, 'go-leftmost', 'sheet.cursorBox.xmin = visibleBox.xmin', 'move cursor to left edge of visible canvas')
Canvas.addCommand(None, 'go-rightmost', 'sheet.cursorBox.xmin = visibleBox.xmax-cursorBox.w', 'move cursor to right edge of visible canvas')
Canvas.addCommand(None, 'go-top', 'sheet.cursorBox.ymin = visibleBox.ymin', 'move cursor to top edge of visible canvas')
Canvas.addCommand(None, 'go-bottom', 'sheet.cursorBox.ymin = visibleBox.ymax', 'move cursor to bottom edge of visible canvas')
Canvas.addCommand(None, 'go-pagedown', 't=(visibleBox.ymax-visibleBox.ymin); sheet.cursorBox.ymin += t; sheet.visibleBox.ymin += t; refresh()', 'move cursor down to next visible page')
Canvas.addCommand(None, 'go-pageup', 't=(visibleBox.ymax-visibleBox.ymin); sheet.cursorBox.ymin -= t; sheet.visibleBox.ymin -= t; refresh()', 'move cursor up to previous visible page')
Canvas.addCommand('zh', 'go-left-small', 'sheet.cursorBox.xmin -= canvasCharWidth', 'move cursor left one character')
Canvas.addCommand('zl', 'go-right-small', 'sheet.cursorBox.xmin += canvasCharWidth', 'move cursor right one character')
Canvas.addCommand('zj', 'go-down-small', 'sheet.cursorBox.ymin += canvasCharHeight', 'move cursor down one character')
Canvas.addCommand('zk', 'go-up-small', 'sheet.cursorBox.ymin -= canvasCharHeight', 'move cursor up one character')
Canvas.addCommand('gH', 'resize-cursor-halfwide', 'sheet.cursorBox.w /= 2', 'halve cursor width')
Canvas.addCommand('gL', 'resize-cursor-doublewide', 'sheet.cursorBox.w *= 2', 'double cursor width')
Canvas.addCommand('gJ','resize-cursor-halfheight', 'sheet.cursorBox.h /= 2', 'halve cursor height')
Canvas.addCommand('gK', 'resize-cursor-doubleheight', 'sheet.cursorBox.h *= 2', 'double cursor height')
Canvas.addCommand('H', 'resize-cursor-thinner', 'sheet.cursorBox.w -= canvasCharWidth', 'decrease cursor width by one character')
Canvas.addCommand('L', 'resize-cursor-wider', 'sheet.cursorBox.w += canvasCharWidth', 'increase cursor width by one character')
Canvas.addCommand('J', 'resize-cursor-taller', 'sheet.cursorBox.h += canvasCharHeight', 'increase cursor height by one character')
Canvas.addCommand('K', 'resize-cursor-shorter', 'sheet.cursorBox.h -= canvasCharHeight', 'decrease cursor height by one character')
Canvas.addCommand('zz', 'zoom-cursor', 'zoomTo(cursorBox)', 'set visible bounds to cursor')
Canvas.addCommand('-', 'zoomout-cursor', 'tmp=cursorBox.center; incrZoom(options.zoom_incr); fixPoint(plotviewBox.center, tmp)', 'zoom out from cursor center')
Canvas.addCommand('+', 'zoomin-cursor', 'tmp=cursorBox.center; incrZoom(1.0/options.zoom_incr); fixPoint(plotviewBox.center, tmp)', 'zoom into cursor center')
Canvas.addCommand('_', 'zoom-all', 'sheet.canvasBox = None; sheet.visibleBox = None; sheet.xzoomlevel=sheet.yzoomlevel=1.0; refresh()', 'zoom to fit full extent')
Canvas.addCommand('z_', 'set-aspect', 'sheet.aspectRatio = float(input("aspect ratio=", value=aspectRatio)); refresh()', 'set aspect ratio')
# set cursor box with left click
Canvas.addCommand('BUTTON1_PRESSED', 'start-cursor', 'sheet.cursorBox = Box(*canvasMouse.xy)', 'start cursor box with left mouse button press')
Canvas.addCommand('BUTTON1_RELEASED', 'end-cursor', 'setCursorSize(canvasMouse)', 'end cursor box with left mouse button release')
Canvas.addCommand('BUTTON3_PRESSED', 'start-move', 'sheet.anchorPoint = canvasMouse', 'mark grid point to move')
Canvas.addCommand('BUTTON3_RELEASED', 'end-move', 'fixPoint(plotterMouse, anchorPoint)', 'mark canvas anchor point')
Canvas.addCommand('BUTTON4_PRESSED', 'zoomin-mouse', 'tmp=canvasMouse; incrZoom(1.0/options.zoom_incr); fixPoint(plotterMouse, tmp)', 'zoom in with scroll wheel')
Canvas.addCommand('REPORT_MOUSE_POSITION', 'zoomout-mouse', 'tmp=canvasMouse; incrZoom(options.zoom_incr); fixPoint(plotterMouse, tmp)', 'zoom out with scroll wheel')
Canvas.bindkey('2097152', 'zoomout-mouse')
Canvas.addCommand('s', 'select-cursor', 'source.select(list(rowsWithin(plotterCursorBox)))', 'select rows on source sheet contained within canvas cursor')
Canvas.addCommand('t', 'stoggle-cursor', 'source.toggle(list(rowsWithin(plotterCursorBox)))', 'toggle selection of rows on source sheet contained within canvas cursor')
Canvas.addCommand('u', 'unselect-cursor', 'source.unselect(list(rowsWithin(plotterCursorBox)))', 'unselect rows on source sheet contained within canvas cursor')
Canvas.addCommand(ENTER, 'dive-cursor', 'vs=copy(source); vs.rows=list(rowsWithin(plotterCursorBox)); vd.push(vs)', 'open sheet of source rows contained within canvas cursor')
Canvas.addCommand('d', 'delete-cursor', 'deleteSourceRows(rowsWithin(plotterCursorBox))', 'delete rows on source sheet contained within canvas cursor')
Canvas.addCommand('gs', 'select-visible', 'source.select(list(rowsWithin(plotterVisibleBox)))', 'select rows on source sheet visible on screen')
Canvas.addCommand('gt', 'stoggle-visible', 'source.toggle(list(rowsWithin(plotterVisibleBox)))', 'toggle selection of rows on source sheet visible on screen')
Canvas.addCommand('gu', 'unselect-visible', 'source.unselect(list(rowsWithin(plotterVisibleBox)))', 'unselect rows on source sheet visible on screen')
Canvas.addCommand('g'+ENTER, 'dive-visible', 'vs=copy(source); vs.rows=list(rowsWithin(plotterVisibleBox)); vd.push(vs)', 'open sheet of source rows visible on screen')
Canvas.addCommand('gd', 'delete-visible', 'deleteSourceRows(rowsWithin(plotterVisibleBox))', 'delete rows on source sheet visible on screen')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612075960.0
visidata-2.2.1/visidata/choose.py 0000660 0001750 0001750 00000005634 00000000000 017455 0 ustar 00kefala kefala 0000000 0000000 from copy import copy
from visidata import vd, option, options, VisiData, ListOfDictSheet, ENTER, CompleteKey, ReturnValue
option('fancy_chooser', False, 'a nicer selection interface for aggregators and jointype')
@VisiData.api
def chooseOne(vd, choices):
'Return one user-selected key from *choices*.'
return vd.choose(choices, 1)
@VisiData.api
def choose(vd, choices, n=None):
'Return a list of 1 to *n* "key" from elements of *choices* (see chooseMany).'
ret = vd.chooseMany(choices) or vd.fail('no choice made')
if n and len(ret) > n:
vd.fail('can only choose %s' % n)
return ret[0] if n==1 else ret
class ChoiceSheet(ListOfDictSheet):
rowtype = 'choices' # rowdef = dict
precious = False
def makeChoice(self, rows):
# selected rows by their keys, separated by spaces
raise ReturnValue([r['key'] for r in rows])
@VisiData.api
def chooseFancy(vd, choices):
vs = ChoiceSheet('choices', source=copy(choices))
options.set('disp_splitwin_pct', -75, vs)
vs.reload()
vs.setKeys([vs.column('key')])
vd.push(vs)
chosen = vd.runresult()
vd.remove(vs)
return chosen
@VisiData.api
def chooseMany(vd, choices):
'Return a list of 1 or more keys from *choices*, which is a list of dicts. Each element dict must have a unique "key", which must be typed directly by the user in non-fancy mode (therefore no spaces). All other items in the dicts are also shown in fancy chooser mode. Use previous choices from the replay input if available. Add chosen keys (space-separated) to the cmdlog as input for the current command.'''
if vd.cmdlog:
v = vd.getLastArgs()
if v is not None:
# check that each key in v is in choices?
vd.setLastArgs(v)
return v.split()
if options.fancy_chooser:
chosen = vd.chooseFancy(choices)
else:
chosen = []
choice_keys = [c['key'] for c in choices]
prompt='choose any of %d options (Ctrl+X for menu)' % len(choice_keys)
try:
def throw_fancy(v, i):
ret = vd.chooseFancy(choices)
if ret:
raise ReturnValue(ret)
return v, i
chosenstr = vd.input(prompt+': ', completer=CompleteKey(choice_keys), bindings={'^X': throw_fancy})
for c in chosenstr.split():
poss = [p for p in choice_keys if str(p).startswith(c)]
if not poss:
vd.warning('invalid choice "%s"' % c)
else:
chosen.extend(poss)
except ReturnValue as e:
chosen = e.args[0]
if vd.cmdlog:
vd.status(str(chosen))
vd.setLastArgs(' '.join(chosen))
return chosen
ChoiceSheet.addCommand(ENTER, 'choose-rows', 'makeChoice([cursorRow])')
ChoiceSheet.addCommand('g'+ENTER, 'choose-rows-selected', 'makeChoice(onlySelectedRows)')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612761678.0
visidata-2.2.1/visidata/clipboard.py 0000660 0001750 0001750 00000021154 00000000000 020127 0 ustar 00kefala kefala 0000000 0000000 from copy import copy, deepcopy
import shutil
import subprocess
import sys
import tempfile
import functools
from visidata import VisiData, vd, asyncthread, option, options
from visidata import Sheet, saveSheets, Path, Column
option('clipboard_copy_cmd', '', 'command to copy stdin to system clipboard', sheettype=None)
option('clipboard_paste_cmd', '', 'command to get contents of system clipboard', sheettype=None)
VisiData.init('cliprows', list) # list of (source_sheet, source_row_idx, source_row)
VisiData.init('clipcells', list) # list of strings
def setslice(L, a, b, M):
L[a:b] = M
@Sheet.api
def copyRows(sheet, rows):
vd.cliprows = list((sheet, i, r) for i, r in enumerate(rows))
if not rows:
vd.warning('no %s selected; clipboard emptied' % sheet.rowtype)
else:
vd.status('copied %d %s to clipboard' % (len(rows), sheet.rowtype))
@Sheet.api
def copyCells(sheet, col, rows):
vd.clipcells = [col.getDisplayValue(r) for r in rows]
if not rows:
vd.warning('no %s selected; clipboard emptied' % sheet.rowtype)
return
vd.status('copied %d %s.%s to clipboard' % (len(rows), sheet.rowtype, col.name))
@Sheet.api
def syscopyRows(sheet, rows):
if not rows:
vd.fail('no %s selected' % sheet.rowtype)
filetype = vd.input("copy %d %s to system clipboard as filetype: " % (len(rows), sheet.rowtype), value=options.save_filetype)
saveToClipboard(sheet, rows, filetype)
vd.status('copied %d %s to system clipboard' % (len(rows), sheet.rowtype))
@Sheet.api
def syscopyCells(sheet, col, rows):
if not rows:
vd.fail('no %s selected' % sheet.rowtype)
clipboard().copy("\n".join(col.getDisplayValue(r) for r in rows))
vd.status('copied %s from %d %s to system clipboard' % (col.name, len(rows), sheet.rowtype))
@Sheet.api
def delete_row(sheet, rowidx):
if not sheet.defer:
oldrow = sheet.rows.pop(rowidx)
vd.addUndo(sheet.rows.insert, rowidx, oldrow)
else:
oldrow = sheet.rows[rowidx]
sheet.rowDeleted(oldrow)
vd.cliprows = [(sheet, rowidx, oldrow)]
@Sheet.api
def paste_after(sheet, rowidx):
vd.addUndo(sheet.rows.pop, rowidx+1)
sheet.rows[rowidx+1:rowidx+1] = list(deepcopy(r) for s,i,r in vd.cliprows)
@Sheet.api
def paste_before(sheet, rowidx):
sheet.rows[sheet.cursorRowIndex:sheet.cursorRowIndex] = list(deepcopy(r) for s,i,r in vd.cliprows)
vd.addUndo(sheet.rows.pop, rowidx)
# mapping of OS to list of possible (command name, command args) for copy and
# paste commands
__copy_commands = {
# TODO TEST WINDOWS AND MAC
'win32': [('clip', '')],
'darwin': [('pbcopy', 'w')],
# try these for all other platforms
None: [('xclip', '-selection clipboard -filter'),
('xsel', '--clipboard --input')]
}
__paste_commands = {
# TODO TEST WINDOWS AND MAC
'win32': [('clip', '')],
'darwin': [('pbpaste', '')],
# try these for all other platforms
None: [('xclip', '-selection clipboard -o'),
('xsel', '--clipboard')]
}
def detect_command(cmdlist):
'''Detect available clipboard util and return cmdline to copy data to the system clipboard.
cmddict is list of (platform, progname, argstr).'''
for cmd, args in cmdlist.get(sys.platform, cmdlist[None]):
path = shutil.which(cmd)
if path: # see if command exists on system
return ' '.join([path, args])
return ''
detect_copy_command = lambda: detect_command(__copy_commands)
detect_paste_command = lambda: detect_command(__paste_commands)
@functools.lru_cache()
def clipboard():
'Detect cmd and set option at first use, to allow option to be changed by user later.'
if not options.clipboard_copy_cmd:
options.clipboard_copy_cmd = detect_copy_command()
if not options.clipboard_paste_cmd:
options.clipboard_paste_cmd = detect_paste_command()
return _Clipboard()
class _Clipboard:
'Cross-platform helper to copy a cell or multiple rows to the system clipboard.'
def get_command(self, name):
if name not in {'copy', 'paste'}:
raise ValueError()
name = 'clipboard_{}_cmd'.format(name)
cmd = getattr(options, name) or vd.fail('options.{} not set'.format(name))
return cmd.split()
def paste(self):
return subprocess.check_output(self.get_command('paste')).decode('utf-8')
def copy(self, value):
'Copy a cell to the system clipboard.'
with tempfile.NamedTemporaryFile() as temp:
with open(temp.name, 'w', encoding=options.encoding) as fp:
fp.write(str(value))
p = subprocess.Popen(
self.get_command('copy'),
stdin=open(temp.name, 'r', encoding=options.encoding),
stdout=subprocess.DEVNULL)
p.communicate()
def save(self, vs, filetype):
'Copy rows to the system clipboard.'
# use NTF to generate filename and delete file on context exit
with tempfile.NamedTemporaryFile(suffix='.'+filetype) as temp:
vd.sync(saveSheets(Path(temp.name), vs))
p = subprocess.Popen(
self.get_command('copy'),
stdin=open(temp.name, 'r', encoding=options.encoding),
stdout=subprocess.DEVNULL,
close_fds=True)
p.communicate()
@VisiData.api
def pasteFromClipboard(vd, cols, rows):
text = vd.getLastArgs() or clipboard().paste().strip() or vd.error('system clipboard is empty')
vd.addUndoSetValues(cols, rows)
for line, r in zip(text.split('\n'), rows):
for v, c in zip(line.split('\t'), cols):
c.setValue(r, v)
@asyncthread
def saveToClipboard(sheet, rows, filetype=None):
'copy rows from sheet to system clipboard'
filetype = filetype or options.save_filetype
vs = copy(sheet)
vs.rows = rows
vd.status('copying rows to clipboard')
clipboard().save(vs, filetype)
Sheet.addCommand('y', 'copy-row', 'copyRows([cursorRow])', 'yank (copy) current row to clipboard')
Sheet.addCommand('d', 'delete-row', 'delete_row(cursorRowIndex)', 'delete (cut) current row and move it to clipboard')
Sheet.addCommand('p', 'paste-after', 'paste_after(cursorRowIndex)', 'paste clipboard rows after current row')
Sheet.addCommand('P', 'paste-before', 'paste_before(cursorRowIndex)', 'paste clipboard rows before current row')
Sheet.addCommand('gd', 'delete-selected', 'copyRows(onlySelectedRows); deleteSelected()', 'delete (cut) selected rows and move them to clipboard')
Sheet.addCommand('gy', 'copy-selected', 'copyRows(onlySelectedRows)', 'yank (copy) selected rows to clipboard')
Sheet.addCommand('zy', 'copy-cell', 'copyCells(cursorCol, [cursorRow])', 'yank (copy) current cell to clipboard')
Sheet.addCommand('zp', 'paste-cell', 'cursorCol.setValuesTyped([cursorRow], vd.clipcells[0]) if vd.clipcells else warning("no cells to paste")', 'set contents of current cell to last clipboard value')
Sheet.addCommand('zd', 'delete-cell', 'vd.clipcells = [cursorDisplay]; cursorCol.setValues([cursorRow], None)', 'delete (cut) current cell and move it to clipboard')
Sheet.addCommand('gzd', 'delete-cells', 'vd.clipcells = list(vd.sheet.cursorCol.getDisplayValue(r) for r in onlySelectedRows); cursorCol.setValues(onlySelectedRows, None)', 'delete (cut) contents of current column for selected rows and move them to clipboard')
Sheet.bindkey('BUTTON2_PRESSED', 'go-mouse')
Sheet.addCommand('BUTTON2_RELEASED', 'syspaste-cells', 'pasteFromClipboard(visibleCols[cursorVisibleColIndex:], rows[cursorRowIndex:])', 'paste into VisiData from system clipboard')
Sheet.bindkey('BUTTON2_CLICKED', 'go-mouse')
Sheet.addCommand('gzy', 'copy-cells', 'copyCells(cursorCol, onlySelectedRows)', 'yank (copy) contents of current column for selected rows to clipboard')
Sheet.addCommand('gzp', 'setcol-clipboard', 'for r, v in zip(onlySelectedRows, itertools.cycle(vd.clipcells)): cursorCol.setValuesTyped([r], v)', 'set cells of current column for selected rows to last clipboard value')
Sheet.addCommand('Y', 'syscopy-row', 'syscopyRows([cursorRow])', 'yank (copy) current row to system clipboard (using options.clipboard_copy_cmd)')
Sheet.addCommand('gY', 'syscopy-selected', 'syscopyRows(onlySelectedRows)', 'yank (copy) selected rows to system clipboard (using options.clipboard_copy_cmd)')
Sheet.addCommand('zY', 'syscopy-cell', 'syscopyCells(cursorCol, [cursorRow])', 'yank (copy) current cell to system clipboard (using options.clipboard_copy_cmd)')
Sheet.addCommand('gzY', 'syscopy-cells', 'syscopyCells(cursorCol, onlySelectedRows)', 'yank (copy) contents of current column from selected rows to system clipboard (using options.clipboard_copy_cmd')
Sheet.bindkey('KEY_DC', 'delete-cell'),
Sheet.bindkey('gKEY_DC', 'delete-cells'),
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1606688365.0
visidata-2.2.1/visidata/cliptext.py 0000660 0001750 0001750 00000004733 00000000000 020030 0 ustar 00kefala kefala 0000000 0000000 import unicodedata
import sys
import functools
from visidata import options
__all__ = ['clipstr', 'clipdraw', 'dispwidth']
disp_column_fill = ' '
### Curses helpers
def dispwidth(ss):
'Return display width of string, according to unicodedata width and options.disp_ambig_width.'
disp_ambig_width = options.disp_ambig_width
w = 0
for cc in ss:
eaw = unicodedata.east_asian_width(cc)
if eaw == 'A': # ambiguous
w += disp_ambig_width
elif eaw in 'WF': # wide/full
w += 2
elif not unicodedata.combining(cc):
w += 1
return w
@functools.lru_cache(maxsize=100000)
def clipstr(s, dispw):
'''Return clipped string and width in terminal display characters.
Note: width may differ from len(s) if East Asian chars are 'fullwidth'.'''
w = 0
ret = ''
trunch = options.disp_truncator
for c in s:
if c != ' ' and unicodedata.category(c) in ('Cc', 'Zs', 'Zl'): # control char, space, line sep
c = options.disp_oddspace
if c:
c = c[0] # multi-char disp_oddspace just uses the first char
ret += c
w += dispwidth(c)
if w > dispw-len(trunch)+1:
ret = ret[:-2] + trunch # replace final char with ellipsis
w += len(trunch)
break
return ret, w
def clipdraw(scr, y, x, s, attr, w=None, rtl=False):
'Draw string `s` at (y,x)-(y,x+w) with curses attr, clipping with ellipsis char. if rtl, draw inside (x-w, x). Returns width drawn (max of w).'
if not scr:
return 0
_, windowWidth = scr.getmaxyx()
dispw = 0
try:
if w is None:
w = len(s)
w = min(w, (x-1) if rtl else (windowWidth-x-1))
if w <= 0: # no room anyway
return 0
# convert to string just before drawing
clipped, dispw = clipstr(str(s), w)
if rtl:
# clearing whole area (w) has negative display effects; clearing just dispw area is useless
# scr.addstr(y, x-dispw-1, disp_column_fill*dispw, attr)
scr.addstr(y, x-dispw-1, clipped, attr)
else:
scr.addstr(y, x, disp_column_fill*w, attr) # clear whole area before displaying
scr.addstr(y, x, clipped, attr)
except Exception as e:
pass
# raise type(e)('%s [clip_draw y=%s x=%s dispw=%s w=%s clippedlen=%s]' % (e, y, x, dispw, w, len(clipped))
# ).with_traceback(sys.exc_info()[2])
return dispw
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612075960.0
visidata-2.2.1/visidata/cmdlog.py 0000660 0001750 0001750 00000040325 00000000000 017436 0 ustar 00kefala kefala 0000000 0000000 import threading
from visidata import *
import visidata
option('replay_wait', 0.0, 'time to wait between replayed commands, in seconds', sheettype=None)
theme('disp_replay_play', '▶', 'status indicator for active replay')
theme('disp_replay_pause', '‖', 'status indicator for paused replay')
theme('color_status_replay', 'green', 'color of replay status indicator')
option('replay_movement', False, 'insert movements during replay', sheettype=None)
option('visidata_dir', '~/.visidata/', 'directory to load and store additional files', sheettype=None)
# prefixes which should not be logged
nonLogged = '''forget exec-longname undo redo quit
show error errors statuses options threads jump
replay cancel save-cmdlog macro
go- search scroll prev next page start end zoom resize visibility
mouse suspend redraw no-op help syscopy sysopen profile toggle'''.split()
option('rowkey_prefix', 'キ', 'string prefix for rowkey in the cmdlog', sheettype=None)
option('cmdlog_histfile', '', 'file to autorecord each cmdlog action to', sheettype=None)
vd.activeCommand = UNLOADED
def open_vd(p):
return CommandLog(p.name, source=p, precious=True)
def open_vdj(p):
return CommandLogJsonl(p.name, source=p, precious=True)
VisiData.save_vd = VisiData.save_tsv
VisiData.save_vdj = VisiData.save_jsonl
def checkVersion(desired_version):
if desired_version != visidata.__version_info__:
vd.fail("version %s required" % desired_version)
def fnSuffix(prefix):
i = 0
fn = prefix + '.vd'
while Path(fn).exists():
i += 1
fn = f'{prefix}-{i}.vd'
return fn
def inputLongname(sheet):
longnames = set(k for (k, obj), v in vd.commands.iter(sheet))
return vd.input("command name: ", completer=CompleteKey(sorted(longnames)), type='longname')
def indexMatch(L, func):
'returns the smallest i for which func(L[i]) is true'
for i, x in enumerate(L):
if func(x):
return i
def keystr(k):
return options.rowkey_prefix+','.join(map(str, k))
def isLoggableCommand(longname):
for n in nonLogged:
if longname.startswith(n):
return False
return True
def isLoggableSheet(sheet):
return sheet is not vd.cmdlog and not isinstance(sheet, (OptionsSheet, ErrorSheet))
@Sheet.api
def moveToRow(vs, rowstr):
'Move cursor to row given by *rowstr*, which can be either the row number or keystr.'
rowidx = vs.getRowIndexFromStr(rowstr)
if rowidx is None:
return False
if vs.options.replay_movement:
while vs.cursorRowIndex != rowidx:
vs.cursorRowIndex += 1 if (rowidx - vs.cursorRowIndex) > 0 else -1
while not vd.delay(0.5):
pass
else:
vs.cursorRowIndex = rowidx
return True
@Sheet.api
def getRowIndexFromStr(vs, rowstr):
index = indexMatch(vs.rows, lambda r,vs=vs,rowstr=rowstr: keystr(vs.rowkey(r)) == rowstr)
if index is not None:
return index
try:
return int(rowstr)
except ValueError:
return None
@Sheet.api
def moveToCol(vs, colstr):
'Move cursor to column given by *colstr*, which can be either the column number or column name.'
try:
vcolidx = int(colstr)
except ValueError:
vcolidx = indexMatch(vs.visibleCols, lambda c,name=colstr: name == c.name)
if vcolidx is None:
return False
if vs.options.replay_movement:
while vs.cursorVisibleColIndex != vcolidx:
vs.cursorVisibleColIndex += 1 if (vcolidx - vs.cursorVisibleColIndex) > 0 else -1
while not vd.delay(0.5):
pass
else:
vs.cursorVisibleColIndex = vcolidx
return True
# rowdef: namedlist (like TsvSheet)
class _CommandLog:
'Log of commands for current session.'
rowtype = 'logged commands'
precious = False
_rowtype = namedlist('CommandLogRow', 'sheet col row longname input keystrokes comment undofuncs'.split())
columns = [
ColumnAttr('sheet'),
ColumnAttr('col'),
ColumnAttr('row'),
ColumnAttr('longname'),
ColumnAttr('input'),
ColumnAttr('keystrokes'),
ColumnAttr('comment'),
ColumnAttr('undo', 'undofuncs', type=vlen, width=0)
]
filetype = 'vd'
def newRow(self, **fields):
return self._rowtype(**fields)
def beforeExecHook(self, sheet, cmd, args, keystrokes):
if not isLoggableCommand(cmd.longname):
return
if vd.activeCommand:
self.afterExecSheet(sheet, False, '')
colname, rowname, sheetname = '', '', None
if sheet and not (cmd.longname.startswith('open-') and not cmd.longname in ('open-row', 'open-cell')):
sheetname = sheet
contains = lambda s, *substrs: any((a in s) for a in substrs)
if contains(cmd.execstr, 'cursorTypedValue', 'cursorDisplay', 'cursorValue', 'cursorCell', 'cursorRow') and sheet.nRows > 0:
k = sheet.rowkey(sheet.cursorRow)
rowname = keystr(k) if k else sheet.cursorRowIndex
if contains(cmd.execstr, 'cursorTypedValue', 'cursorDisplay', 'cursorValue', 'cursorCell', 'cursorCol', 'cursorVisibleCol'):
colname = sheet.cursorCol.name or sheet.visibleCols.index(sheet.cursorCol)
if contains(cmd.execstr, 'plotterCursorBox'):
bb = sheet.cursorBox
colname = '%s %s' % (sheet.formatX(bb.xmin), sheet.formatX(bb.xmax))
rowname = '%s %s' % (sheet.formatY(bb.ymin), sheet.formatY(bb.ymax))
elif contains(cmd.execstr, 'plotterVisibleBox'):
bb = sheet.visibleBox
colname = '%s %s' % (sheet.formatX(bb.xmin), sheet.formatX(bb.xmax))
rowname = '%s %s' % (sheet.formatY(bb.ymin), sheet.formatY(bb.ymax))
if contains(cmd.execstr, 'pasteFromClipboard'):
args = clipboard().paste().strip()
comment = vd.currentReplayRow.comment if vd.currentReplayRow else cmd.helpstr
vd.activeCommand = self.newRow(sheet=sheetname,
col=str(colname),
row=str(rowname),
keystrokes=keystrokes,
input=args,
longname=cmd.longname,
comment=comment,
undofuncs=[])
def afterExecSheet(self, sheet, escaped, err):
'Records vd.activeCommand'
if not vd.activeCommand: # nothing to record
return
if err:
vd.activeCommand[-1] += ' [%s]' % err
# remove user-aborted commands and simple movements
if not escaped and isLoggableCommand(vd.activeCommand.longname):
if isLoggableSheet(sheet): # don't record actions on global cmdlog or other internal sheets
self.addRow(vd.activeCommand) # add to global cmdlog
sheet.cmdlog_sheet.addRow(vd.activeCommand) # add to sheet-specific cmdlog
if options.cmdlog_histfile:
if not getattr(vd, 'sessionlog', None):
vd.sessionlog = vd.loadInternalSheet(CommandLog, Path(date().strftime(options.cmdlog_histfile)))
append_tsv_row(vd.sessionlog, vd.activeCommand)
vd.activeCommand = None
def openHook(self, vs, src):
r = self.newRow(keystrokes='o', input=src, longname='open-file')
vs.cmdlog_sheet.addRow(r)
self.addRow(r)
class CommandLog(_CommandLog, VisiDataMetaSheet):
pass
class CommandLogJsonl(_CommandLog, JsonLinesSheet):
def newRow(self):
return JsonLinesSheet.newRow(self)
def iterload(self):
for r in JsonLinesSheet.iterload(self):
if isinstance(r, TypedWrapper):
yield r
else:
yield AttrDict(r)
### replay
vd.paused = False
vd.currentReplay = None # CommandLog replaying currently
vd.currentReplayRow = None # must be global, to allow replay
vd.semaphore = threading.Semaphore(0)
@VisiData.api
def replay_pause(vd):
if not vd.currentReplay:
vd.fail('no replay to pause')
else:
if vd.paused:
vd.replay_advance()
vd.paused = not vd.paused
vd.status('paused' if vd.paused else 'resumed')
@VisiData.api
def replay_advance(vd):
vd.currentReplay or vd.fail("no replay to advance")
vd.semaphore.release()
@VisiData.api
def replay_cancel(vd):
vd.currentReplay or vd.fail("no replay to cancel")
vd.currentReplayRow = None
vd.currentReplay = None
vd.semaphore.release()
@VisiData.api
def moveToReplayContext(vd, r, vs):
'set the sheet/row/col to the values in the replay row. return sheet'
if r.row:
vs.moveToRow(r.row) or vd.error('no "%s" row' % r.row)
if r.col:
vs.moveToCol(r.col) or vd.error('no "%s" column' % r.col)
@VisiData.api
def delay(vd, factor=1):
'returns True if delay satisfied'
acquired = vd.semaphore.acquire(timeout=options.replay_wait*factor if not vd.paused else None)
return acquired or not vd.paused
@VisiData.property
def activeSheet(vd):
'Return top sheet on sheets stack, or cmdlog sheets stack empty.'
return vd.sheets[0] if vd.sheets else vd.cmdlog
@VisiData.api
def replayOne(vd, r):
'Replay the command in one given row.'
vd.currentReplayRow = r
longname = getattr(r, 'longname', None)
if r.sheet and longname not in ['set-option', 'unset-option']:
vs = vd.getSheet(r.sheet) or vd.error('no sheet named %s' % r.sheet)
else:
vs = None
if longname in ['set-option', 'unset-option']:
try:
context = vs if r.sheet and vs else vd
option_scope = r.sheet or r.col or 'global'
if option_scope == 'override': option_scope = 'global' # override is deprecated, is now global
if longname == 'set-option':
context.options.set(r.row, r.input, option_scope)
else:
context.options.unset(r.row, option_scope)
escaped = False
except Exception as e:
vd.exceptionCaught(e)
escaped = True
else:
if vs:
vd.push(vs)
else:
vs = vd.activeSheet
vd.moveToReplayContext(r, vs)
if r.comment:
vd.status(r.comment)
vd.keystrokes = r.keystrokes
# <=v1.2 used keystrokes in longname column; getCommand fetches both
escaped = vs.execCommand(longname if longname else r.keystrokes, keystrokes=r.keystrokes)
vd.currentReplayRow = None
if escaped: # escape during replay aborts replay
vd.warning('replay aborted during %s' % (longname or r.keystrokes))
return escaped
@VisiData.api
def replay_sync(vd, cmdlog, live=False):
'Replay all commands in log.'
cmdlog.cursorRowIndex = 0
vd.currentReplay = cmdlog
with Progress(total=len(cmdlog.rows)) as prog:
while cmdlog.cursorRowIndex < len(cmdlog.rows):
if vd.currentReplay is None:
vd.status('replay canceled')
return
vd.statuses.clear()
try:
if vd.replayOne(cmdlog.cursorRow):
vd.replay_cancel()
return True
except Exception as e:
vd.replay_cancel()
vd.exceptionCaught(e)
vd.status('replay canceled')
return True
cmdlog.cursorRowIndex += 1
prog.addProgress(1)
vd.activeSheet.ensureLoaded()
vd.sync()
while not vd.delay():
pass
vd.status('replay complete')
vd.currentReplay = None
@VisiData.api
@asyncthread
def replay(vd, cmdlog):
'Inject commands into live execution with interface.'
vd.replay_sync(cmdlog, live=True)
@VisiData.api
def getLastArgs(vd):
'Get user input for the currently playing command.'
if vd.currentReplayRow:
return vd.currentReplayRow.input
return None
@VisiData.api
def setLastArgs(vd, args):
'Set user input on last command, if not already set.'
# only set if not already set (second input usually confirmation)
if (vd.activeCommand is not None) and (vd.activeCommand is not UNLOADED):
if not vd.activeCommand.input:
vd.activeCommand.input = args
@VisiData.property
def replayStatus(vd):
x = options.disp_replay_pause if vd.paused else options.disp_replay_play
return ' │ %s %s/%s' % (x, vd.currentReplay.cursorRowIndex, len(vd.currentReplay.rows))
@BaseSheet.property
def cmdlog(sheet):
rows = sheet.cmdlog_sheet.rows
if isinstance(sheet.source, BaseSheet):
rows = sheet.source.cmdlog.rows + rows
return CommandLog(sheet.name+'_cmdlog', source=sheet, rows=rows)
@BaseSheet.lazy_property
def cmdlog_sheet(sheet):
return CommandLog(sheet.name+'_cmdlog', source=sheet, rows=[])
@BaseSheet.property
def shortcut(self):
try:
return str(vd.allSheets.index(self)+1)
except ValueError:
pass
try:
return self.cmdlog_sheet.rows[0].keystrokes
except Exception:
pass
return ''
@VisiData.lazy_property
def cmdlog(vd):
vs = CommandLog('cmdlog', rows=[])
vd.beforeExecHooks.append(vs.beforeExecHook)
return vs
@VisiData.property
def modifyCommand(vd):
if vd.activeCommand is not None and isLoggableCommand(vd.activeCommand.longname):
return vd.activeCommand
if not vd.cmdlog.rows:
return None
return vd.cmdlog.rows[-1]
globalCommand('gD', 'cmdlog-all', 'vd.push(vd.cmdlog)', 'open global CommandLog for all commands executed in current session')
globalCommand('D', 'cmdlog-sheet', 'vd.push(sheet.cmdlog)', "open current sheet's CommandLog with all other loose ends removed; includes commands from parent sheets")
globalCommand('zD', 'cmdlog-sheet-only', 'vd.push(sheet.cmdlog_sheet)', 'open current sheet\'s CommandLog with parent sheets commands\' removed')
globalCommand('^D', 'save-cmdlog', 'saveSheets(inputPath("save cmdlog to: ", value=fnSuffix(name)), vd.cmdlog, confirm_overwrite=options.confirm_overwrite)', 'save CommandLog to filename.vd file')
globalCommand('^U', 'replay-pause', 'vd.replay_pause()', 'pause/resume replay')
globalCommand('^N', 'replay-advance', 'vd.replay_advance()', 'execute next row in replaying sheet')
globalCommand('^K', 'replay-stop', 'vd.replay_cancel()', 'cancel current replay')
globalCommand(None, 'show-status', 'status(input("status: "))', 'show given message on status line')
globalCommand('^V', 'show-version', 'status(__version_info__);', 'show version and copyright information on status line')
globalCommand('z^V', 'check-version', 'checkVersion(input("require version: ", value=__version_info__))', 'check VisiData version against given version')
globalCommand(' ', 'exec-longname', 'execCommand(inputLongname(sheet))', 'execute command by its longname')
CommandLog.addCommand('x', 'replay-row', 'vd.replayOne(cursorRow); status("replayed one row")', 'replay command in current row')
CommandLog.addCommand('gx', 'replay-all', 'vd.replay(sheet)', 'replay contents of entire CommandLog')
CommandLog.addCommand('^C', 'replay-stop', 'sheet.cursorRowIndex = sheet.nRows', 'abort replay')
CommandLogJsonl.addCommand('x', 'replay-row', 'vd.replayOne(cursorRow); status("replayed one row")', 'replay command in current row')
CommandLogJsonl.addCommand('gx', 'replay-all', 'vd.replay(sheet)', 'replay contents of entire CommandLog')
CommandLogJsonl.addCommand('^C', 'replay-stop', 'sheet.cursorRowIndex = sheet.nRows', 'abort replay')
BaseSheet.addCommand('', 'repeat-last', 'execCommand(cmdlog_sheet.rows[-1].longname)', 'run most recent command with an empty, queried input')
BaseSheet.addCommand('', 'repeat-input', 'r = copy(cmdlog_sheet.rows[-1]); r.sheet=r.row=r.col=""; vd.replayOne(r)', 'run previous command, along with any previous input to that command')
CommandLog.class_options.json_sort_keys = False
CommandLogJsonl.class_options.json_sort_keys = False
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1606688365.0
visidata-2.2.1/visidata/color.py 0000660 0001750 0001750 00000007105 00000000000 017306 0 ustar 00kefala kefala 0000000 0000000 import curses
import functools
from copy import copy
from visidata import options, Extensible, drawcache, drawcache_property
from collections import namedtuple
__all__ = ['ColorAttr', 'colors', 'update_attr', 'ColorMaker']
ColorAttr = namedtuple('ColorAttr', ('color', 'attributes', 'precedence', 'attr'))
def update_attr(oldattr, updattr, updprec=None):
if isinstance(updattr, ColorAttr):
if updprec is None:
updprec = updattr.precedence
updcolor = updattr.color
updattr = updattr.attributes
else:
updcolor = updattr & curses.A_COLOR
updattr = updattr & ~curses.A_COLOR
if updprec is None:
updprec = 0
# starting values, work backwards
newcolor = oldattr.color
newattr = oldattr.attributes | updattr
newprec = oldattr.precedence
if not newcolor or updprec > newprec:
if updcolor:
newcolor = updcolor
newprec = updprec
return ColorAttr(newcolor, newattr, newprec, newcolor | newattr)
class ColorMaker:
def __init__(self):
self.attrs = {}
self.color_attrs = {}
@drawcache_property
def colorcache(self):
return {}
def setup(self):
if options.use_default_colors:
curses.use_default_colors()
default_bg = -1
else:
default_bg = curses.COLOR_BLACK
self.color_attrs['black'] = curses.color_pair(0)
for c in range(0, options.force_256_colors and 256 or curses.COLORS):
try:
curses.init_pair(c+1, c, default_bg)
self.color_attrs[str(c)] = curses.color_pair(c+1)
except curses.error as e:
pass # curses.init_pair gives a curses error on Windows
for c in 'red green yellow blue magenta cyan white'.split():
colornum = getattr(curses, 'COLOR_' + c.upper())
self.color_attrs[c] = curses.color_pair(colornum+1)
for a in 'normal blink bold dim reverse standout underline'.split():
self.attrs[a] = getattr(curses, 'A_' + a.upper())
def keys(self):
return list(self.attrs.keys()) + list(self.color_attrs.keys())
def __getitem__(self, colornamestr):
return self._colornames_to_cattr(colornamestr).attr
def __getattr__(self, optname):
'colors.color_foo returns colors[options.color_foo]'
return self.get_color(optname).attr
@drawcache
def resolve_colors(self, colorstack):
'Returns the ColorAttr for the colorstack, a list of color option names sorted highest-precedence color first.'
cattr = ColorAttr(0,0,0,0)
for coloropt in colorstack:
c = self.get_color(coloropt)
cattr = update_attr(cattr, c)
return cattr
def _colornames_to_cattr(self, colornamestr, precedence=0):
color, attr = 0, 0
for colorname in colornamestr.split(' '):
if colorname in self.color_attrs:
if not color:
color = self.color_attrs[colorname.lower()]
elif colorname in self.attrs:
attr = self.attrs[colorname.lower()]
return ColorAttr(color, attr, precedence, color | attr)
def get_color(self, optname, precedence=0):
'colors.color_foo returns colors[options.color_foo]'
r = self.colorcache.get(optname, None)
if r is None:
coloropt = options._get(optname)
colornamestr = coloropt.value if coloropt else optname
r = self.colorcache[optname] = self._colornames_to_cattr(colornamestr, precedence)
return r
colors = ColorMaker()
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1606688365.0
visidata-2.2.1/visidata/colorsheet.py 0000660 0001750 0001750 00000001554 00000000000 020341 0 ustar 00kefala kefala 0000000 0000000 import curses
from visidata import globalCommand, colors, Sheet, Column, RowColorizer, wrapply
class ColorSheet(Sheet):
rowtype = 'colors' # rowdef: color number as assigned in the colors object
columns = [
Column('color', type=int),
Column('R', getter=lambda col,row: curses.color_content(curses.pair_number(colors[row])-1)[0]),
Column('G', getter=lambda col,row: curses.color_content(curses.pair_number(colors[row])-1)[1]),
Column('B', getter=lambda col,row: curses.color_content(curses.pair_number(colors[row])-1)[2]),
]
colorizers = [
RowColorizer(7, None, lambda s,c,r,v: r)
]
def reload(self):
self.rows = sorted(colors.keys(), key=lambda n: wrapply(int, n))
globalCommand(None, 'colors', 'vd.push(ColorSheet("vdcolors"))', 'open Color Sheet with an overview of curses colors and codes')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612761678.0
visidata-2.2.1/visidata/column.py 0000660 0001750 0001750 00000045727 00000000000 017501 0 ustar 00kefala kefala 0000000 0000000 from copy import copy
import collections
import string
import itertools
import threading
import re
import time
from visidata import option, options, anytype, stacktrace, vd
from visidata import asyncthread, dispwidth
from visidata import wrapply, TypedWrapper, TypedExceptionWrapper
from visidata import Extensible, AttrDict, undoAttrFunc
class InProgress(Exception):
@property
def stacktrace(self):
return ['calculation in progress']
INPROGRESS = TypedExceptionWrapper(None, exception=InProgress()) # sentinel
option('col_cache_size', 0, 'max number of cache entries in each cached column')
option('clean_names', False, 'clean column/sheet names to be valid Python identifiers', replay=True)
__all__ = [
'clean_to_id',
'clean_name',
'maybe_clean',
'Column',
'setitem',
'getattrdeep',
'setattrdeep',
'getitemdef',
'ColumnAttr', 'AttrColumn',
'ColumnItem', 'ItemColumn',
'SettableColumn',
'SubColumnFunc',
'SubColumnItem',
'SubColumnAttr',
'ColumnExpr', 'ExprColumn',
'DisplayWrapper',
]
class DisplayWrapper:
def __init__(self, value=None, *, display=None, note=None, notecolor=None, error=None):
self.value = value # actual value (any type)
self.display = display # displayed string
self.note = note # single unicode character displayed in cell far right
self.notecolor = notecolor # configurable color name (like 'color_warning')
self.error = error # list of strings for stacktrace
def __bool__(self):
return bool(self.value)
def __eq__(self, other):
return self.value == other
def maybe_clean(s, vs):
if (vs or vd).options.clean_names:
s = clean_name(s)
return s
def clean_name(s):
s = re.sub(r'[^\w\d_]', '_', s) # replace non-alphanum chars with _
s = re.sub(r'_+', '_', s) # replace runs of _ with a single _
return s
def clean_to_id(s): # [Nas Banov] https://stackoverflow.com/a/3305731
return re.sub(r'\W|^(?=\d)', '_', str(s)).strip('_')
def _default_colnames():
'A B C .. Z AA AB .. ZZ AAA .. to infinity'
i=0
while True:
i += 1
for x in itertools.product(string.ascii_uppercase, repeat=i):
yield ''.join(x)
default_colnames = _default_colnames()
class Column(Extensible):
'''Base class for all column types.
- *name*: name of this column.
- *type*: ``anytype str int float date`` or other type-like conversion function.
- *cache*: cache behavior
- ``False`` (default): getValue never caches; calcValue is always called.
- ``True``: getValue maintains a cache of ``options.col_cache_size``.
- ``"async"``: ``getValue`` launches thread for every uncached result, returns invalid value until cache entry available.
- *width*: == 0 if hidden, None if auto-compute next time.
- *height*: max height, None/0 to auto-compute for each row.
- *fmtstr*: format string as applied by column type.
- *getter*: default calcValue calls ``getter(col, row)``.
- *setter*: default putValue calls ``setter(col, row, val)``.
- *kwargs*: other attributes to be set on this column.
'''
def __init__(self, name=None, *, type=anytype, cache=False, **kwargs):
self.sheet = None # owning Sheet, set in .recalc() via Sheet.addColumn
if name is None:
name = next(default_colnames)
self.name = str(name) # display visible name
self.fmtstr = '' # by default, use str()
self._type = type # anytype/str/int/float/date/func
self.getter = lambda col, row: row
self.setter = lambda col, row, value: vd.fail(col.name+' column cannot be changed')
self._width = None # == 0 if hidden, None if auto-compute next time
self.hoffset = 0 # starting horizontal (char) offset of displayed column value
self.voffset = 0 # starting vertical (line) offset of displayed column value
self.height = 1 # max height, None/0 to auto-compute for each row
self.keycol = 0 # keycol index (or 0 if not key column)
self.expr = None # Column-type-dependent parameter
self.setCache(cache)
for k, v in kwargs.items():
setattr(self, k, v) # instead of __dict__.update(kwargs) to invoke property.setters
def __copy__(self):
cls = self.__class__
ret = cls.__new__(cls)
ret.__dict__.update(self.__dict__)
ret.keycol = 0 # column copies lose their key status
if self._cachedValues is not None:
ret._cachedValues = collections.OrderedDict() # an unrelated cache for copied columns
return ret
def __deepcopy__(self, memo):
return self.__copy__() # no separate deepcopy
def __getstate__(self):
d = {k:getattr(self, k) for k in 'name width height expr keycol fmtstr voffset hoffset aggstr'.split()}
d['type'] = self.type.__name__
return d
def __setstate__(self, d):
for attr, v in d.items():
setattr(self, attr, v)
def recalc(self, sheet=None):
'Reset column cache, attach column to *sheet*, and reify column name.'
if self._cachedValues:
self._cachedValues.clear()
if sheet:
self.sheet = sheet
self.name = self._name
@property
def name(self):
'Name of this column.'
return self._name or ''
@name.setter
def name(self, name):
if name is None:
name = ''
if isinstance(name, str):
name = name.strip()
else:
name = str(name)
self._name = maybe_clean(name, self.sheet)
@property
def typestr(self):
'Type of this column as string.'
return self._type.__name__
@typestr.setter
def typestr(self, v):
self.type = vd.getGlobals()[v or 'anytype']
@property
def type(self):
'Type of this column.'
return self._type
@type.setter
def type(self, t):
if self._type != t:
vd.addUndo(setattr, self, '_type', self.type)
if not t:
self._type = anytype
elif isinstance(t, str):
self.typestr = t
else:
self._type = t
@property
def width(self):
'Width of this column in characters. 0 or negative means hidden. None means not-yet-autocomputed.'
return self._width
@width.setter
def width(self, w):
if self.width != w:
if self.width == 0 or w == 0: # hide/unhide
vd.addUndo(setattr, self, '_width', self.width)
self._width = w
@property
def fmtstr(self):
'Format string to use to display this column.'
return self._fmtstr or vd.getType(self.type).fmtstr
@fmtstr.setter
def fmtstr(self, v):
self._fmtstr = v
def formatValue(self, typedval):
'Return displayable string of *typedval* according to ``Column.fmtstr``.'
if typedval is None:
return None
if isinstance(typedval, (list, tuple)):
return '[%s]' % len(typedval)
if isinstance(typedval, dict):
return '{%s}' % len(typedval)
if isinstance(typedval, bytes):
typedval = typedval.decode(options.encoding, options.encoding_errors)
return vd.getType(self.type).formatter(self.fmtstr, typedval)
format=formatValue
def hide(self, hide=True):
if hide:
self.setWidth(0)
else:
self.setWidth(abs(self.width or self.getMaxWidth(self.sheet.visibleRows)))
@property
def hidden(self):
'Return True if width of this column is 0 or negative.'
if self.width is None:
return False
return self.width <= 0
def calcValue(self, row):
'Calculate and return value for *row* in this column.'
return (self.getter)(self, row)
def getTypedValue(self, row):
'Return the properly-typed value for the given row at this column, or a TypedWrapper object in case of null or error.'
return wrapply(self.type, wrapply(self.getValue, row))
def setCache(self, cache):
'''Set cache behavior for this column to *cache*:
- ``False`` (default): getValue never caches; calcValue is always called.
- ``True``: getValue maintains a cache of ``options.col_cache_size``.
- ``"async"``: ``getValue`` launches thread for every uncached result, maintains cache of infinite size. Returns invalid value until cache entry available.'''
self.cache = cache
self._cachedValues = collections.OrderedDict() if self.cache else None
@asyncthread
def _calcIntoCacheAsync(self, row):
# causes isues when moved into _calcIntoCache gen case
self._cachedValues[self.sheet.rowid(row)] = INPROGRESS
self._calcIntoCache(row)
def _calcIntoCache(self, row):
ret = wrapply(self.calcValue, row)
if not isinstance(ret, TypedExceptionWrapper) or ret.val is not INPROGRESS:
self._cachedValues[self.sheet.rowid(row)] = ret
return ret
def getValue(self, row):
'Return value for *row* in this column, calculating if not cached.'
if self.sheet.defer:
try:
row, rowmods = self.sheet._deferredMods[self.sheet.rowid(row)]
return rowmods[self]
except KeyError:
pass
if self._cachedValues is None:
return self.calcValue(row)
k = self.sheet.rowid(row)
if k in self._cachedValues:
return self._cachedValues[k]
if self.cache == 'async':
ret = self._calcIntoCacheAsync(row)
else:
ret = self._calcIntoCache(row)
cachesize = options.col_cache_size
if cachesize > 0 and len(self._cachedValues) > cachesize:
self._cachedValues.popitem(last=False)
return ret
def getCell(self, row):
'Return DisplayWrapper for displayable cell value.'
cellval = wrapply(self.getValue, row)
typedval = wrapply(self.type, cellval)
if isinstance(typedval, TypedWrapper):
if isinstance(cellval, TypedExceptionWrapper): # calc failed
exc = cellval.exception
if cellval.forwarded:
dispval = str(cellval) # traceback.format_exception_only(type(exc), exc)[-1].strip()
else:
dispval = options.disp_error_val
return DisplayWrapper(cellval.val, error=exc.stacktrace,
display=dispval,
note=options.note_getter_exc,
notecolor='color_error')
elif typedval.val is None: # early out for strict None
return DisplayWrapper(None, display='', # force empty display for None
note=options.disp_note_none,
notecolor='color_note_type')
elif isinstance(typedval, TypedExceptionWrapper): # calc succeeded, type failed
return DisplayWrapper(typedval.val, display=str(cellval),
error=typedval.stacktrace,
note=options.note_type_exc,
notecolor='color_warning')
else:
return DisplayWrapper(typedval.val, display=str(typedval.val),
error='unknown',
note=options.note_type_exc,
notecolor='color_warning')
elif isinstance(typedval, threading.Thread):
return DisplayWrapper(None,
display=options.disp_pending,
note=options.note_pending,
notecolor='color_note_pending')
dw = DisplayWrapper(cellval)
try:
dw.display = self.format(typedval) or ''
# annotate cells with raw value type in anytype columns, except for strings
if self.type is anytype and type(cellval) is not str:
typedesc = vd.typemap.get(type(cellval), None)
if typedesc:
dw.note = typedesc.icon
dw.notecolor = 'color_note_type'
except Exception as e: # formatting failure
e.stacktrace = stacktrace()
dw.error = e.stacktrace
try:
dw.display = str(cellval)
except Exception as e:
dw.display = str(e)
dw.note = options.note_format_exc
dw.notecolor = 'color_warning'
return dw
def getDisplayValue(self, row):
'Return string displayed in this column for given *row*.'
return self.getCell(row).display
def putValue(self, row, val):
'Change value for *row* in this column to *val* immediately. Does not check the type. Overrideable; by default calls ``.setter(row, val)``.'
return self.setter(self, row, val)
def setValue(self, row, val):
'Change value for *row* in this column to *val*. Call ``putValue`` immediately if parent ``sheet.defer`` is False, otherwise cache until later ``putChanges``. Caller must add undo function.'
if self.sheet.defer:
self.cellChanged(row, val)
else:
self.putValue(row, val)
def setValueSafe(self, row, value):
'setValue and ignore exceptions.'
try:
return self.setValue(row, value)
except Exception as e:
vd.exceptionCaught(e)
@asyncthread
def setValues(self, rows, *values):
'Set values in this column for *rows* to *values*, recycling values as needed to fill *rows*.'
vd.addUndoSetValues([self], rows)
for r, v in zip(rows, itertools.cycle(values)):
self.setValueSafe(r, v)
self.recalc()
return vd.status('set %d cells to %d values' % (len(rows), len(values)))
def setValuesTyped(self, rows, *values):
'Set values on this column for *rows* to *values*, coerced to column type, recycling values as needed to fill *rows*. Abort on type exception.'
vd.addUndoSetValues([self], rows)
for r, v in zip(rows, itertools.cycle(self.type(val) for val in values)):
self.setValueSafe(r, v)
self.recalc()
return vd.status('set %d cells to %d values' % (len(rows), len(values)))
def getMaxWidth(self, rows):
'Return the maximum length of any cell in column or its header.'
w = 0
nlen = dispwidth(self.name)
if len(rows) > 0:
w = max(max(dispwidth(self.getDisplayValue(r)) for r in rows), nlen)+2
return max(w, nlen)
# ---- Column makers
def setitem(r, i, v): # function needed for use in lambda
r[i] = v
return True
def getattrdeep(obj, attr, *default):
'Return dotted attr (like "a.b.c") from obj, or default if any of the components are missing.'
attrs = attr.split('.')
if default:
getattr_default = lambda o,a,d=default[0]: getattr(o, a, d)
else:
getattr_default = lambda o,a: getattr(o, a)
for a in attrs[:-1]:
obj = getattr_default(obj, a)
return getattr_default(obj, attrs[-1])
def setattrdeep(obj, attr, val):
'Set dotted attr (like "a.b.c") on obj to val.'
attrs = attr.split('.')
for a in attrs[:-1]:
obj = getattr(obj, a)
setattr(obj, attrs[-1], val)
def AttrColumn(name='', attr=None, **kwargs):
'Column using getattr/setattr with *attr*.'
return Column(name,
expr=attr if attr is not None else name,
getter=lambda col,row: getattrdeep(row, col.expr),
setter=lambda col,row,val: setattrdeep(row, col.expr, val),
**kwargs)
def getitemdef(o, k, default=None):
try:
return default if o is None else o[k]
except Exception:
return default
def ItemColumn(name=None, key=None, **kwargs):
'Column using getitem/setitem with *key*.'
return Column(name,
expr=key if key is not None else name,
getter=lambda col,row: getitemdef(row, col.expr),
setter=lambda col,row,val: setitem(row, col.expr, val),
**kwargs)
class SubColumnFunc(Column):
'Column compositor; preprocess row with *subfunc*(row, *expr*) before passing to *origcol*.getValue and *origcol*.setValue.'
def __init__(self, name='', origcol=None, expr=None, subfunc=getitemdef, **kwargs):
super().__init__(name, type=origcol.type, width=origcol.width, expr=expr, **kwargs)
self.origcol = origcol
self.subfunc = subfunc
def calcValue(self, row):
subrow = self.subfunc(row, self.expr)
if subrow is not None:
# call getValue to use deferred values from source sheet
return self.origcol.getValue(subrow)
def putValue(self, row, value):
subrow = self.subfunc(row, self.expr)
if subrow is None:
vd.fail('no source row')
self.origcol.setValue(subrow, value)
def recalc(self, sheet=None):
Column.recalc(self, sheet)
self.origcol.recalc() # reset cache but don't change sheet
def SubColumnAttr(attrname, c, **kwargs):
if 'name' not in kwargs:
kwargs['name'] = c.name
return SubColumnFunc(origcol=c, subfunc=getattrdeep, expr=attrname, **kwargs)
def SubColumnItem(idx, c, **kwargs):
if 'name' not in kwargs:
kwargs['name'] = c.name
return SubColumnFunc(origcol=c, subfunc=getitemdef, expr=idx, **kwargs)
class ExprColumn(Column):
'Column using *expr* to derive the value from each row.'
def __init__(self, name, expr=None, **kwargs):
super().__init__(name, **kwargs)
self.expr = expr or name
self.ncalcs = 0
self.totaltime = 0
self.maxtime = 0
def calcValue(self, row):
t0 = time.perf_counter()
r = self.sheet.evalExpr(self.compiledExpr, row, col=self)
t1 = time.perf_counter()
self.ncalcs += 1
self.maxtime = max(self.maxtime, t1-t0)
self.totaltime += (t1-t0)
return r
def putValue(self, row, val):
a = self.getDisplayValue(row)
b = self.format(self.type(val))
if a != b:
vd.warning('%s calced %s not %s' % (self.name, a, b))
@property
def expr(self):
return self._expr
@expr.setter
def expr(self, expr):
self.compiledExpr = compile(expr, '', 'eval') if expr else None
self._expr = expr
class SettableColumn(Column):
'Column using rowid to store and retrieve values internally.'
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self._store = {}
def putValue(self, row, value):
self._store[self.sheet.rowid(row)] = value
def calcValue(self, row):
return self._store.get(self.sheet.rowid(row), None)
# synonyms
ColumnItem = ItemColumn
ColumnAttr = AttrColumn
ColumnExpr = ExprColumn
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608541899.0
visidata-2.2.1/visidata/customdate.py 0000660 0001750 0001750 00000001736 00000000000 020344 0 ustar 00kefala kefala 0000000 0000000 import time
from visidata import vd, date, Sheet, ColumnsSheet
@Sheet.api
def customdate(sheet, fmtstr):
'Return date class with strptime parse format fixed to *fmtstr*.'
class _customdate(date):
def __new__(cls, *args, **kwargs):
if len(args) == 1 and isinstance(args[0], str):
return super().__new__(cls, *time.strptime(args[0], fmtstr)[:6])
return super().__new__(cls, *args, **kwargs)
vd.addType(_customdate, '@', '', formatter=lambda fmt,val: val.strftime(fmt or sheet.options.disp_date_fmt))
_customdate.__name__ = 'customdate(%s)' % fmtstr
return _customdate
Sheet.addCommand('z@', 'type-customdate', 'cursorCol.type=cursorCol.type=customdate(input("date format: ", type="fmtstr"))', 'set type of current column to custom date format')
ColumnsSheet.addCommand('gz@', 'type-customdate-selected', 'onlySelectedRows.type=customdate(input("date format: ", type="fmtstr"))', 'set type of selected columns to date')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612075960.0
visidata-2.2.1/visidata/deprecated.py 0000660 0001750 0001750 00000007031 00000000000 020266 0 ustar 00kefala kefala 0000000 0000000 from visidata import VisiData, vd
import visidata
alias = visidata.BaseSheet.bindkey
def deprecated(ver, instead=''):
def decorator(func):
def wrapper(*args, **kwargs):
import traceback
for line in reversed(traceback.extract_stack(limit=6)[:-1]):
vd.warning(f' file {line.filename} at line {line.lineno} in {line.name}')
vd.warning(f'Deprecated call traceback (most recent last):')
msg = f'{func.__name__} deprecated since v{ver}'
if instead:
msg += f'; use {instead}'
vd.warning(msg)
return func(*args, **kwargs)
return wrapper
return decorator
@deprecated('1.6', 'vd instead of vd()')
@VisiData.api
def __call__(vd):
'Deprecated; use plain "vd"'
return vd
@deprecated('1.6')
def copyToClipboard(value):
return visidata.clipboard().copy(value)
@deprecated('1.6')
def replayableOption(optname, default, helpstr):
vd.option(optname, default, helpstr, replay=True)
@deprecated('1.6')
def SubrowColumn(*args, **kwargs):
return visidata.SubColumnFunc(*args, **kwargs)
@deprecated('1.6')
def DeferredSetColumn(*args, **kwargs):
return visidata.Column(*args, defer=True, **kwargs)
@deprecated('2.0')
def bindkey_override(keystrokes, longname):
vd.bindkeys.set(keystrokes, longname)
bindkey = visidata.BaseSheet.bindkey
unbindkey = visidata.BaseSheet.unbindkey
@deprecated('2.0')
@visidata.Sheet.api
def exec_keystrokes(self, keystrokes, vdglobals=None):
return self.execCommand(self.getCommand(keystrokes), vdglobals, keystrokes=keystrokes)
visidata.Sheet.exec_command = deprecated('2.0')(visidata.Sheet.execCommand)
@deprecated('2.0', 'def open_ instead')
@VisiData.api
def filetype(vd, ext, constructor):
'Add constructor to handle the given file type/extension.'
globals().setdefault('open_'+ext, lambda p,ext=ext: constructor(p.name, source=p, filetype=ext))
@deprecated('2.0', 'Sheet(namepart1, namepart2, ...)')
@VisiData.global_api
def joinSheetnames(vd, *sheetnames):
'Concatenate sheet names in a standard way'
return visidata.options.name_joiner.join(str(x) for x in sheetnames)
@deprecated('2.0', 'PyobjSheet')
@VisiData.global_api
def load_pyobj(*names, **kwargs):
return visidata.PyobjSheet(*names, **kwargs)
@deprecated('2.0', 'PyobjSheet')
@VisiData.global_api
def push_pyobj(name, pyobj):
vs = visidata.PyobjSheet(name, source=pyobj)
if vs:
return vd.push(vs)
else:
vd.error("cannot push '%s' as pyobj" % type(pyobj).__name__)
@deprecated('2.1', 'vd.isNumeric instead')
def isNumeric(col):
return vd.isNumeric(col)
visidata.addGlobals({'load_pyobj': load_pyobj, 'isNumeric': isNumeric})
# The longnames on the left are deprecated for 2.0
alias('edit-cells', 'setcol-input')
alias('fill-nulls', 'setcol-fill')
alias('paste-cells', 'setcol-clipboard')
alias('frequency-rows', 'frequency-summary')
alias('dup-cell', 'dive-cell')
alias('dup-row', 'dive-row')
alias('next-search', 'search-next')
alias('prev-search', 'search-prev')
alias('search-prev', 'searchr-next')
alias('prev-sheet', 'jump-prev')
alias('prev-value', 'go-prev-value')
alias('next-value', 'go-next-value')
alias('prev-selected', 'go-prev-selected')
alias('next-selected', 'go-next-selected')
alias('prev-null', 'go-prev-null')
alias('next-null', 'go-next-null')
alias('page-right', 'go-right-page')
alias('page-left', 'go-left-page')
alias('dive-cell', 'open-cell')
alias('dive-row', 'open-row')
alias('add-sheet', 'open-new')
alias('save-sheets-selected', 'save-selected')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612075960.0
visidata-2.2.1/visidata/describe.py 0000660 0001750 0001750 00000010326 00000000000 017747 0 ustar 00kefala kefala 0000000 0000000 from statistics import mode, median, mean, stdev
from visidata import *
option('describe_aggrs', 'mean stdev', 'numeric aggregators to calculate on Describe sheet')
max_threads = 2
@Column.api
def isError(col, row):
'Return True if the computed or typed value for *row* in this column is an error.'
try:
v = col.getValue(row)
if v is not None:
col.type(v)
return False
except Exception as e:
return True
class DescribeColumn(Column):
def __init__(self, name, **kwargs):
super().__init__(name, getter=lambda col,srccol: col.sheet.describeData[srccol].get(col.expr, ''), expr=name, **kwargs)
# rowdef: Column from source sheet
class DescribeSheet(ColumnsSheet):
# rowtype = 'columns'
precious = True
columns = [
ColumnAttr('sheet', 'sheet', width=0),
ColumnAttr('column', 'name'),
ColumnAttr('type', 'typestr', width=0),
DescribeColumn('errors', type=vlen),
DescribeColumn('nulls', type=vlen),
DescribeColumn('distinct',type=vlen),
DescribeColumn('mode', type=str),
DescribeColumn('min', type=str),
DescribeColumn('max', type=str),
DescribeColumn('sum'),
DescribeColumn('median', type=str),
]
colorizers = [
RowColorizer(7, 'color_key_col', lambda s,c,r,v: r and r in r.sheet.keyCols),
]
nKeys = 2
@asyncthread
def reload(self):
super().reload()
self.rows = [c for c in self.rows if not c.hidden]
self.describeData = { col: {} for col in self.rows }
self.columns = []
for c in type(self).columns:
self.addColumn(c)
self.setKeys(self.columns[:self.nKeys])
for aggrname in options.describe_aggrs.split():
self.addColumn(DescribeColumn(aggrname, type=float))
for srccol in Progress(self.rows, 'categorizing'):
if not srccol.hidden:
self.reloadColumn(srccol)
def reloadColumn(self, srccol):
d = self.describeData[srccol]
isNull = srccol.sheet.isNullFunc()
vals = list()
d['errors'] = list()
d['nulls'] = list()
d['distinct'] = set()
for sr in Progress(srccol.sheet.rows, 'calculating'):
try:
v = srccol.getValue(sr)
if isNull(v):
d['nulls'].append(sr)
else:
v = srccol.type(v)
vals.append(v)
d['distinct'].add(v)
except Exception as e:
d['errors'].append(sr)
d['mode'] = self.calcStatistic(d, mode, vals)
if vd.isNumeric(srccol):
for func in [min, max, sum, median]: # use type
d[func.__name__] = self.calcStatistic(d, func, vals)
for aggrname in options.describe_aggrs.split():
func = globals()[aggrname]
d[func.__name__] = self.calcStatistic(d, func, vals)
def calcStatistic(self, d, func, *args, **kwargs):
r = wrapply(func, *args, **kwargs)
d[func.__name__] = r
return r
def openCell(self, col, row):
'open copy of source sheet with rows described in current cell'
val = col.getValue(row)
if isinstance(val, list):
vs=copy(row.sheet)
vs.rows=val
vs.name+="_%s_%s"%(row.name,col.name)
return vs
vd.warning(val)
Sheet.addCommand('I', 'describe-sheet', 'vd.push(DescribeSheet(sheet.name+"_describe", source=[sheet]))', 'open Describe Sheet with descriptive statistics for all visible columns')
globalCommand('gI', 'describe-all', 'vd.push(DescribeSheet("describe_all", source=vd.sheets))', 'open Describe Sheet with description statistics for all visible columns from all sheets')
DescribeSheet.addCommand('zs', 'select-cell', 'cursorRow.sheet.select(cursorValue)', 'select rows on source sheet which are being described in current cell')
DescribeSheet.addCommand('zu', 'unselect-cell', 'cursorRow.sheet.unselect(cursorValue)', 'unselect rows on source sheet which are being described in current cell')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608178530.0
visidata-2.2.1/visidata/editor.py 0000660 0001750 0001750 00000003174 00000000000 017460 0 ustar 00kefala kefala 0000000 0000000 import os
import signal
import subprocess
import tempfile
import curses
import visidata
class SuspendCurses:
'Context manager to leave windowed mode on enter and restore it on exit.'
def __enter__(self):
curses.endwin()
def __exit__(self, exc_type, exc_val, tb):
newscr = curses.initscr()
newscr.refresh()
curses.doupdate()
@visidata.VisiData.global_api
def launchEditor(vd, *args):
'Launch $EDITOR with *args* as arguments.'
editor = os.environ.get('EDITOR') or vd.fail('$EDITOR not set')
args = [editor] + list(args)
with SuspendCurses():
return subprocess.call(args)
@visidata.VisiData.global_api
def launchExternalEditor(vd, v, linenum=0):
'Launch $EDITOR to edit string *v* starting on line *linenum*.'
import tempfile
with tempfile.NamedTemporaryFile() as temp:
with open(temp.name, 'w') as fp:
fp.write(v)
return launchExternalEditorPath(visidata.Path(temp.name), linenum)
def launchExternalEditorPath(path, linenum=0):
'Launch $EDITOR to edit *path* starting on line *linenum*.'
if linenum:
launchEditor(path, '+%s' % linenum)
else:
launchEditor(path)
with open(path, 'r') as fp:
try:
return fp.read().rstrip('\n') # trim inevitable trailing newlines
except Exception as e:
visidata.vd.exceptionCaught(e)
return ''
def suspend():
import signal
with SuspendCurses():
os.kill(os.getpid(), signal.SIGSTOP)
visidata.globalCommand('^Z', 'suspend', 'suspend()', 'suspend VisiData process')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1606688365.0
visidata-2.2.1/visidata/errors.py 0000660 0001750 0001750 00000001456 00000000000 017507 0 ustar 00kefala kefala 0000000 0000000 import traceback
from visidata import vd, VisiData, options
__all__ = ['stacktrace', 'ExpectedException']
class ExpectedException(Exception):
'an expected exception'
pass
def stacktrace(e=None):
if not e:
return traceback.format_exc().strip().splitlines()
return traceback.format_exception_only(type(e), e)
@VisiData.global_api
def exceptionCaught(vd, exc=None, **kwargs):
'Maintain list of most recent errors and return most recent one.'
if isinstance(exc, ExpectedException): # already reported, don't log
return
vd.lastErrors.append(stacktrace())
if kwargs.get('status', True):
vd.status(vd.lastErrors[-1][-1], priority=2) # last line of latest error
if options.debug:
raise
# see textsheet.py for ErrorSheet and associated commands
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608706358.0
visidata-2.2.1/visidata/expr.py 0000660 0001750 0001750 00000005317 00000000000 017151 0 ustar 00kefala kefala 0000000 0000000 from visidata import Progress, Sheet, Column, asyncthread, vd, ExprColumn
class CompleteExpr:
def __init__(self, sheet=None):
self.sheet = sheet
def __call__(self, val, state):
i = len(val)-1
while val[i:].isidentifier() and i >= 0:
i -= 1
if i < 0:
base = ''
partial = val
elif val[i] == '.': # no completion of attributes
return None
else:
base = val[:i+1]
partial = val[i+1:]
varnames = []
varnames.extend(sorted((base+col.name) for col in self.sheet.columns if col.name.startswith(partial)))
varnames.extend(sorted((base+x) for x in globals() if x.startswith(partial)))
# Remove duplicate tabbing suggestions
varnames_dict = {var:None for var in varnames}
varnames = list(varnames_dict.keys())
return varnames[state%len(varnames)]
@Column.api
@asyncthread
def setValuesFromExpr(self, rows, expr):
'Set values in this column for *rows* to the result of the Python expression *expr* applied to each row.'
compiledExpr = compile(expr, '', 'eval')
vd.addUndoSetValues([self], rows)
for row in Progress(rows, 'setting'):
# Note: expressions that are only calculated once, do not need to pass column identity
# they can reference their "previous selves" once without causing a recursive problem
self.setValueSafe(row, self.sheet.evalExpr(compiledExpr, row))
self.recalc()
vd.status('set %d values = %s' % (len(rows), expr))
@Sheet.api
def inputExpr(self, prompt, *args, **kwargs):
return vd.input(prompt, "expr", *args, completer=CompleteExpr(self), **kwargs)
Sheet.addCommand('=', 'addcol-expr', 'addColumnAtCursor(ExprColumn(inputExpr("new column expr="), curcol=cursorCol))', 'create new column from Python expression, with column names as variables')
Sheet.addCommand('g=', 'setcol-expr', 'cursorCol.setValuesFromExpr(someSelectedRows, inputExpr("set selected="))', 'set current column for selected rows to result of Python expression')
Sheet.addCommand('z=', 'setcell-expr', 'cursorCol.setValues([cursorRow], evalExpr(inputExpr("set expr="), cursorRow,))', 'evaluate Python expression on current row and set current cell with result of Python expression')
Sheet.addCommand('gz=', 'setcol-iter', 'cursorCol.setValues(someSelectedRows, *list(itertools.islice(eval(input("set column= ", "expr", completer=CompleteExpr())), len(someSelectedRows))))', 'set current column for selected rows to the items in result of Python sequence expression')
Sheet.addCommand(None, 'show-expr', 'status(evalExpr(inputExpr("show expr="), cursorRow))', 'evaluate Python expression on current row and show result on status line')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608178530.0
visidata-2.2.1/visidata/extensible.py 0000660 0001750 0001750 00000005552 00000000000 020336 0 ustar 00kefala kefala 0000000 0000000 from functools import wraps, lru_cache
__all__ = ['Extensible', 'cache']
class Extensible:
_cache_clearers = [] # list of func() to call in clearCaches()
@classmethod
def init(cls, membername, initfunc=lambda: None, copy=False):
'Append equivalent of ``self. = initfunc()`` to ``.__init__``.'
oldinit = cls.__init__
def newinit(self, *args, **kwargs):
oldinit(self, *args, **kwargs)
if not hasattr(self, membername): # can be overridden by a subclass
setattr(self, membername, initfunc())
cls.__init__ = newinit
oldcopy = cls.__copy__
def newcopy(self, *args, **kwargs):
ret = oldcopy(self, *args, **kwargs)
setattr(ret, membername, getattr(self, membername) if copy else initfunc())
return ret
cls.__copy__ = newcopy
@classmethod
def api(cls, func):
oldfunc = getattr(cls, func.__name__, None)
if oldfunc:
func = wraps(oldfunc)(func)
setattr(cls, func.__name__, func)
return func
@classmethod
def class_api(cls, func):
name = func.__get__(None, dict).__func__.__name__
oldfunc = getattr(cls, name, None)
if oldfunc:
func = wraps(oldfunc)(func)
setattr(cls, name, func)
return func
@classmethod
def property(cls, func):
@property
@wraps(func)
def dofunc(self):
return func(self)
setattr(cls, func.__name__, dofunc)
return dofunc
@classmethod
def lazy_property(cls, func):
'Return ``func()`` on first access and cache result; return cached result thereafter.'
@property
@wraps(func)
def get_if_not(self):
name = '_' + func.__name__
if not hasattr(self, name):
setattr(self, name, func(self))
return getattr(self, name)
setattr(cls, func.__name__, get_if_not)
return get_if_not
@classmethod
def cached_property(cls, func):
'Return ``func()`` on first access, and cache result; return cached result until ``clearCaches()``.'
@property
@wraps(func)
@lru_cache(maxsize=None)
def get_if_not(self):
return func(self)
setattr(cls, func.__name__, get_if_not)
Extensible._cache_clearers.append(get_if_not.fget.cache_clear)
return get_if_not
@classmethod
def clear_all_caches(cls):
for func in Extensible._cache_clearers:
func()
def cache(func):
'Return func(...) on first access, and cache result; return cached result until clearCaches().'
@wraps(func)
@lru_cache(maxsize=None)
def call_if_not(self, *args, **kwargs):
return func(self, *args, **kwargs)
Extensible._cache_clearers.append(call_if_not.cache_clear)
return call_if_not
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612763790.0
visidata-2.2.1/visidata/fill.py 0000660 0001750 0001750 00000001776 00000000000 017126 0 ustar 00kefala kefala 0000000 0000000 from visidata import *
@VisiData.api
@asyncthread
def fillNullValues(vd, col, rows):
'Fill null cells in col with the previous non-null value'
lastval = None
oldvals = [] # for undo
isNull = col.sheet.isNullFunc()
n = 0
rowsToFill = [id(r) for r in rows]
for r in Progress(col.sheet.rows, 'filling'): # loop over all rows
try:
val = col.getValue(r)
except Exception as e:
val = e
if isNull(val):
if lastval and (id(r) in rowsToFill):
oldvals.append((col,r,val))
col.setValue(r, lastval)
n += 1
else:
lastval = val
def _undo():
for c, r, v in oldvals:
c.setValue(r, v)
vd.addUndo(_undo)
col.recalc()
vd.status("filled %d values" % n)
Sheet.addCommand('f', 'setcol-fill', 'fillNullValues(cursorCol, selectedRows)', 'fills null cells in selected rows of current column with contents of non-null cells up the current column')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608604413.0
visidata-2.2.1/visidata/freeze.py 0000660 0001750 0001750 00000004166 00000000000 017454 0 ustar 00kefala kefala 0000000 0000000 from visidata import *
@Column.api
def resetCache(col):
col._cachedValues = collections.OrderedDict()
vd.status("reset cache for " + col.name)
@Sheet.api
def StaticColumn(sheet, col):
frozencol = SettableColumn(col.name+'_frozen', width=col.width, type=col.type, fmtstr=col._fmtstr)
frozencol.recalc(sheet)
@asyncthread
def calcRows_async(frozencol, rows, col):
# no need to undo, addColumn undo is enough
for r in Progress(rows, 'calculating'):
try:
frozencol.putValue(r, col.getTypedValue(r))
except Exception as e:
frozencol.putValue(r, e)
calcRows_async(frozencol, sheet.rows, col)
return frozencol
class StaticSheet(Sheet):
'A copy of the source sheet with all cells frozen.'
def __init__(self, source):
super().__init__(source.name + "'", source=source)
self.columns = []
for i, col in enumerate(self.source.visibleCols):
colcopy = ColumnItem(col.name, i, width=col.width, type=col.type, fmtstr=col._fmtstr)
self.addColumn(colcopy)
if col in self.source.keyCols:
self.setKeys([colcopy])
@asyncthread
def reload(self):
self.rows = []
for r in Progress(self.source.rows, 'calculating'):
row = []
self.addRow(row)
for col in self.source.visibleCols:
val = col.getTypedValue(r)
if isinstance(val, TypedExceptionWrapper):
row.append(None)
else:
row.append(val)
Sheet.addCommand("'", 'freeze-col', 'sheet.addColumnAtCursor(StaticColumn(cursorCol))', 'add a frozen copy of current column with all cells evaluated')
Sheet.addCommand("g'", 'freeze-sheet', 'vd.push(StaticSheet(sheet)); status("pushed frozen copy of "+name)', 'open a frozen copy of current sheet with all visible columns evaluated')
Sheet.addCommand("z'", 'cache-col', 'cursorCol.resetCache()', 'add/reset cache for current column')
Sheet.addCommand("gz'", 'cache-cols', 'for c in visibleCols: c.resetCache()', 'add/reset cache for all visible columns')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608541899.0
visidata-2.2.1/visidata/freqtbl.py 0000660 0001750 0001750 00000010210 00000000000 017616 0 ustar 00kefala kefala 0000000 0000000 import math
import collections
from visidata import *
theme('disp_histogram', '*', 'histogram element character')
option('disp_histolen', 50, 'width of histogram column')
option('histogram_bins', 0, 'number of bins for histogram of numeric columns')
option('numeric_binning', False, 'bin numeric columns into ranges', replay=True)
def valueNames(discrete_vals, numeric_vals):
ret = [ '+'.join(str(x) for x in discrete_vals) ]
if numeric_vals != (0, 0):
ret.append('%s-%s' % numeric_vals)
return '+'.join(ret)
class FreqTableSheet(PivotSheet):
'Generate frequency-table sheet on currently selected column.'
rowtype = 'bins' # rowdef FreqRow(keys, sourcerows)
def __init__(self, sheet, *groupByCols):
fqcolname = '%s_%s_freq' % (sheet.name, '-'.join(col.name for col in groupByCols))
super().__init__(fqcolname, groupByCols, [], source=sheet)
self.largest = 1
def selectRow(self, row):
self.source.select(row.sourcerows) # select all entries in the bin on the source sheet
return super().selectRow(row) # then select the bin itself on this sheet
def unselectRow(self, row):
self.source.unselect(row.sourcerows)
return super().unselectRow(row)
def updateLargest(self, grouprow):
self.largest = max(self.largest, len(grouprow.sourcerows))
@asyncthread
def reload(self):
'Generate frequency table then reverse-sort by length.'
super().initCols()
# add default bonus columns
for c in [
ColumnAttr('count', 'sourcerows', type=vlen),
Column('percent', type=float, getter=lambda col,row: len(row.sourcerows)*100/col.sheet.source.nRows),
Column('histogram', type=str, getter=lambda col,row: options.disp_histogram*(options.disp_histolen*len(row.sourcerows)//col.sheet.largest), width=options.disp_histolen+2),
]:
self.addColumn(c)
# two more threads
vd.sync(self.addAggregateCols(),
self.groupRows(self.updateLargest))
if self.nCols > len(self.groupByCols)+3: # hide percent/histogram if aggregations added
self.column('percent').hide()
self.column('histogram').hide()
if not [c for c in self.groupByCols if vd.isNumeric(c)]:
self.orderBy(self.column('count'), reverse=True)
def openRow(self, row):
'open copy of source sheet with rows that are grouped in current row'
if row.sourcerows:
vs = copy(self.source)
vs.name += "_"+valueNames(row.discrete_keys, row.numeric_key)
vs.rows=copy(row.sourcerows)
return vs
vd.warning("no source rows")
def openCell(self, col, row):
return Sheet.openCell(self, col, row)
class FreqTableSheetSummary(FreqTableSheet):
'Append a PivotGroupRow to FreqTable with only selectedRows.'
@asyncthread
def reload(self):
FreqTableSheet.reload.__wrapped__(self)
self.addRow(PivotGroupRow(['Selected'], (0,0), self.source.selectedRows, {}))
Sheet.addCommand('F', 'freq-col', 'vd.push(FreqTableSheet(sheet, cursorCol))', 'open Frequency Table grouped on current column, with aggregations of other columns')
Sheet.addCommand('gF', 'freq-keys', 'vd.push(FreqTableSheet(sheet, *keyCols))', 'open Frequency Table grouped by all key columns on source sheet, with aggregations of other columns')
Sheet.addCommand('zF', 'freq-summary', 'vd.push(FreqTableSheetSummary(sheet, Column("Total", sheet=sheet, getter=lambda col, row: "Total")))', 'open one-line summary for all rows and selected rows')
ColumnsSheet.addCommand(ENTER, 'freq-row', 'vd.push(FreqTableSheet(source[0], cursorRow))', 'open a Frequency Table sheet grouped on column referenced in current row')
FreqTableSheet.addCommand('gu', 'unselect-rows', 'unselect(selectedRows)', 'unselect all source rows grouped in current row')
FreqTableSheet.addCommand('g'+ENTER, 'dive-rows', 'vs = copy(source); vs.name += "_several"; vs.rows=list(itertools.chain.from_iterable(row.sourcerows for row in selectedRows)); vd.push(vs)', 'open copy of source sheet with rows that are grouped in selected rows')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608541899.0
visidata-2.2.1/visidata/graph.py 0000660 0001750 0001750 00000016652 00000000000 017300 0 ustar 00kefala kefala 0000000 0000000 from visidata import *
option('color_graph_axis', 'bold', 'color for graph axis labels')
def numericCols(cols):
return [c for c in cols if vd.isNumeric(c)]
class InvertedCanvas(Canvas):
def zoomTo(self, bbox):
super().zoomTo(bbox)
self.fixPoint(Point(self.plotviewBox.xmin, self.plotviewBox.ymax), bbox.xymin)
def plotpixel(self, x, y, attr, row=None):
y = self.plotviewBox.ymax-y
self.pixels[y][x][attr].append(row)
def scaleY(self, canvasY):
'returns plotter y coordinate, with y-axis inverted'
plotterY = super().scaleY(canvasY)
return (self.plotviewBox.ymax-plotterY+4)
def canvasH(self, plotterY):
return (self.plotviewBox.ymax-plotterY)/self.yScaler
@property
def canvasMouse(self):
p = super().canvasMouse
p.y = self.visibleBox.ymin + (self.plotviewBox.ymax-self.plotterMouse.y)/self.yScaler
return p
# provides axis labels, legend
class GraphSheet(InvertedCanvas):
def __init__(self, *names, **kwargs):
super().__init__(*names, **kwargs)
self.xcols or vd.fail('at least one key col necessary for x-axis')
self.ycols or vd.fail('%s is non-numeric' % '/'.join(yc.name for yc in kwargs.get('ycols')))
@asyncthread
def reload(self):
nerrors = 0
nplotted = 0
self.reset()
vd.status('loading data points')
catcols = [c for c in self.xcols if not vd.isNumeric(c)]
numcols = numericCols(self.xcols)
for ycol in self.ycols:
for rownum, row in enumerate(Progress(self.sourceRows, 'plotting')): # rows being plotted from source
try:
k = tuple(c.getValue(row) for c in catcols) if catcols else (ycol.name,)
# convert deliberately to float (to e.g. linearize date)
graph_x = float(numcols[0].type(numcols[0].getValue(row))) if numcols else rownum
graph_y = ycol.type(ycol.getValue(row))
attr = self.plotColor(k)
self.point(graph_x, graph_y, attr, row)
nplotted += 1
except Exception:
nerrors += 1
if options.debug:
raise
vd.status('loaded %d points (%d errors)' % (nplotted, nerrors))
self.xzoomlevel=self.yzoomlevel=1.0
self.resetBounds()
self.refresh()
def resetBounds(self):
super().resetBounds()
self.createLabels()
def moveToRow(self, rowstr):
ymin, ymax = map(float, map(self.parseY, rowstr.split()))
self.cursorBox.ymin = ymin
self.cursorBox.h = ymax-ymin
return True
def moveToCol(self, colstr):
xmin, xmax = map(float, map(self.parseX, colstr.split()))
self.cursorBox.xmin = xmin
self.cursorBox.w = xmax-xmin
return True
def formatX(self, amt):
return ','.join(xcol.format(xcol.type(amt)) for xcol in self.xcols if vd.isNumeric(xcol))
def formatY(self, amt):
srccol = self.ycols[0]
return srccol.format(srccol.type(amt))
def parseX(self, txt):
return self.xcols[0].type(txt)
def parseY(self, txt):
return self.ycols[0].type(txt)
def add_y_axis_label(self, frac):
txt = self.formatY(self.visibleBox.ymin + frac*self.visibleBox.h)
# plot y-axis labels on the far left of the canvas, but within the plotview height-wise
attr = colors.color_graph_axis
self.plotlabel(0, self.plotviewBox.ymin + (1.0-frac)*self.plotviewBox.h, txt, attr)
def add_x_axis_label(self, frac):
txt = self.formatX(self.visibleBox.xmin + frac*self.visibleBox.w)
# plot x-axis labels below the plotviewBox.ymax, but within the plotview width-wise
attr = colors.color_graph_axis
xmin = self.plotviewBox.xmin + frac*self.plotviewBox.w
if frac == 1.0:
# shift rightmost label to be readable
xmin -= max(len(txt)*2 - self.rightMarginPixels+1, 0)
self.plotlabel(xmin, self.plotviewBox.ymax+4, txt, attr)
def createLabels(self):
self.gridlabels = []
# y-axis
self.add_y_axis_label(1.00)
self.add_y_axis_label(0.75)
self.add_y_axis_label(0.50)
self.add_y_axis_label(0.25)
self.add_y_axis_label(0.00)
# x-axis
self.add_x_axis_label(1.00)
self.add_x_axis_label(0.75)
self.add_x_axis_label(0.50)
self.add_x_axis_label(0.25)
self.add_x_axis_label(0.00)
# TODO: if 0 line is within visible bounds, explicitly draw the axis
# TODO: grid lines corresponding to axis labels
xname = ','.join(xcol.name for xcol in self.xcols if vd.isNumeric(xcol)) or 'row#'
xname, _ = clipstr(xname, self.leftMarginPixels//2-2)
self.plotlabel(0, self.plotviewBox.ymax+4, xname+'»', colors.color_graph_axis)
Sheet.addCommand('.', 'plot-column', 'vd.push(GraphSheet(sheet.name, "graph", source=sheet, sourceRows=rows, xcols=keyCols, ycols=numericCols([cursorCol])))', 'plot current numeric column vs key columns; numeric key column is used for x-axis, while categorical key columns determine color')
Sheet.addCommand('g.', 'plot-numerics', 'vd.push(GraphSheet(sheet.name, "graph", source=sheet, sourceRows=rows, xcols=keyCols, ycols=numericCols(nonKeyVisibleCols)))', 'plot a graph of all visible numeric columns vs key columns')
# swap directions of up/down
InvertedCanvas.addCommand(None, 'go-up', 'sheet.cursorBox.ymin += cursorBox.h', 'move cursor up by its height')
InvertedCanvas.addCommand(None, 'go-down', 'sheet.cursorBox.ymin -= cursorBox.h', 'move cursor down by its height')
InvertedCanvas.addCommand(None, 'go-top', 'sheet.cursorBox.ymin = visibleBox.ymax', 'move cursor to top edge of visible canvas')
InvertedCanvas.addCommand(None, 'go-bottom', 'sheet.cursorBox.ymin = visibleBox.ymin', 'move cursor to bottom edge of visible canvas')
InvertedCanvas.addCommand(None, 'go-pagedown', 't=(visibleBox.ymax-visibleBox.ymin); sheet.cursorBox.ymin -= t; sheet.visibleBox.ymin -= t; sheet.refresh()', 'move cursor down to next visible page')
InvertedCanvas.addCommand(None, 'go-pageup', 't=(visibleBox.ymax-visibleBox.ymin); sheet.cursorBox.ymin += t; sheet.visibleBox.ymin += t; sheet.refresh()', 'move cursor up to previous visible page')
InvertedCanvas.addCommand(None, 'go-down-small', 'sheet.cursorBox.ymin -= canvasCharHeight', 'move cursor down one character')
InvertedCanvas.addCommand(None, 'go-up-small', 'sheet.cursorBox.ymin += canvasCharHeight', 'move cursor up one character')
InvertedCanvas.addCommand(None, 'resize-cursor-shorter', 'sheet.cursorBox.h -= canvasCharHeight', 'decrease cursor height by one character')
InvertedCanvas.addCommand(None, 'resize-cursor-taller', 'sheet.cursorBox.h += canvasCharHeight', 'increase cursor height by one character')
@GraphSheet.api
def set_y(sheet, s):
ymin, ymax = map(float, map(sheet.parseY, s.split()))
sheet.zoomTo(BoundingBox(sheet.visibleBox.xmin, ymin, sheet.visibleBox.xmax, ymax))
sheet.refresh()
@GraphSheet.api
def set_x(sheet, s):
xmin, xmax = map(float, map(sheet.parseX, s.split()))
sheet.zoomTo(BoundingBox(xmin, sheet.visibleBox.ymin, xmax, sheet.visibleBox.ymax))
sheet.refresh()
Canvas.addCommand('y', 'resize-y-input', 'sheet.set_y(input("set ymin ymax="))', 'set ymin/ymax on graph axes')
Canvas.addCommand('x', 'resize-x-input', 'sheet.set_x(input("set xmin xmax="))', 'set xmin/xmax on graph axes')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1609740788.0
visidata-2.2.1/visidata/help.py 0000660 0001750 0001750 00000005732 00000000000 017124 0 ustar 00kefala kefala 0000000 0000000 from visidata import *
class HelpSheet(MetaSheet):
'Show all commands available to the source sheet.'
rowtype = 'commands'
precious = False
_ordering = [('sheet', False), ('longname', False)]
columns = [
ColumnAttr('sheet'),
ColumnAttr('longname'),
Column('keystrokes', getter=lambda col,row: col.sheet.revbinds.get(row.longname)),
Column('description', getter=lambda col,row: col.sheet.cmddict[(row.sheet, row.longname)].helpstr),
ColumnAttr('execstr', width=0),
Column('logged', width=0, getter=lambda col,row: isLoggableCommand(row.longname)),
]
nKeys = 2
def iterload(self):
from pkg_resources import resource_filename
cmdlist = VisiDataMetaSheet('cmdlist', source=None)
self.cmddict = {}
itcmds = vd.commands.iterall()
for (k, o), v in itcmds:
yield v
v.sheet = o
self.cmddict[(v.sheet, v.longname)] = v
for cmdrow in cmdlist.rows:
k = (cmdrow.sheet, cmdrow.longname)
if k in self.cmddict:
self.cmddict[k].helpstr = cmdrow.helpstr
self.revbinds = {} # [longname] -> keystrokes
itbindings = vd.bindkeys.iterall()
for (keystrokes, _), longname in itbindings:
if (keystrokes not in self.revbinds) and not vd.isLongname(keystrokes):
self.revbinds[longname] = keystrokes
@VisiData.api
@asyncthread
def help_search(vd, sheet, regex):
vs = HelpSheet(source=None)
vs.rows = [] # do not trigger push reload
vd.push(vs) # push first, then reload
vd.sync(vs.reload())
# find rows matching regex on original HelpSheet
rowidxs = list(vd.searchRegex(vs, regex=regex, columns="visibleCols"))
# add only matching rows
allrows = vs.rows
vs.rows = []
for rowidx in rowidxs:
vs.addRow(allrows[rowidx])
@VisiData.global_api
def openManPage(vd):
from pkg_resources import resource_filename
import os
with SuspendCurses():
if os.system(' '.join(['man', resource_filename(__name__, 'man/vd.1')])) != 0:
vd.push(TextSheet('man_vd', source=Path(resource_filename(__name__, 'man/vd.txt'))))
# in VisiData, ^H refers to the man page
globalCommand('^H', 'sysopen-help', 'openManPage()', 'view vd man page')
BaseSheet.addCommand('z^H', 'help-commands', 'vd.push(HelpSheet(name + "_commands", source=sheet, revbinds={}))', 'view sheet of command longnames and keybindings for current sheet')
BaseSheet.addCommand('gz^H', 'help-commands-all', 'vd.push(HelpSheet("all_commands", source=None, revbinds={}))', 'view sheet of command longnames and keybindings for all sheet types')
globalCommand(None, 'help-search', 'help_search(sheet, input("help: "))', 'search through command longnames with search terms')
BaseSheet.bindkey('KEY_F(1)', 'sysopen-help')
BaseSheet.bindkey('KEY_BACKSPACE', 'sysopen-help')
BaseSheet.bindkey('zKEY_F(1)', 'help-commands')
BaseSheet.bindkey('zKEY_BACKSPACE', 'help-commands')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608178530.0
visidata-2.2.1/visidata/incr.py 0000660 0001750 0001750 00000002472 00000000000 017125 0 ustar 00kefala kefala 0000000 0000000 from visidata import VisiData, Sheet, vd, options, option
option('incr_base', 1.0, 'start value for column increments', replay=True)
@VisiData.api
def numrange(vd, n, step=1):
'Generate n values, starting from options.incr_base and increasing by step for each number.'
base = type(step)(options.incr_base)
yield from (base+x*step for x in range(n))
@VisiData.api
def num(vd, *args):
'Return parsed string as number, preferring int to float.'
try:
return int(*args)
except Exception:
return float(*args)
Sheet.addCommand('i', 'addcol-incr', 'c=SettableColumn(type=int); addColumnAtCursor(c); c.setValues(rows, *numrange(nRows))', 'add column with incremental values')
Sheet.addCommand('gi', 'setcol-incr', 'cursorCol.setValues(selectedRows, *numrange(sheet.nSelectedRows))', 'set current column for selected rows to incremental values')
Sheet.addCommand('zi', 'addcol-incr-step', 'n=num(input("interval step: ")); c=SettableColumn(type=type(n)); addColumnAtCursor(c); c.setValues(rows, *numrange(nRows, step=n))', 'add column with incremental values times given step')
Sheet.addCommand('gzi', 'setcol-incr-step', 'n=num(input("interval step: ")); cursorCol.setValues(selectedRows, *numrange(nSelectedRows, n))', 'set current column for selected rows to incremental values times given step')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608541899.0
visidata-2.2.1/visidata/join.py 0000660 0001750 0001750 00000024210 00000000000 017123 0 ustar 00kefala kefala 0000000 0000000 import collections
import itertools
import functools
from copy import copy
from visidata import *
def createJoinedSheet(sheets, jointype=''):
sheets[1:] or vd.fail("join requires more than 1 sheet")
if jointype == 'append':
keyedcols = collections.defaultdict(list, {col.name:[col] for col in sheets[0].visibleCols})
for s in sheets[1:]:
for col in s.visibleCols:
key = col.name if col.name in keyedcols else col.sheet.visibleCols.index(col)
keyedcols[key].append(col)
return ConcatSheet('&'.join(vs.name for vs in sheets), sourceCols=list(keyedcols.values()))
elif jointype == 'extend':
vs = copy(sheets[0])
vs.name = '+'.join(vs.name for vs in sheets)
vs.reload = functools.partial(ExtendedSheet_reload, vs, sheets)
return vs
else:
return JoinSheet('+'.join(vs.name for vs in sheets), sources=sheets, jointype=jointype)
jointypes = [{'key': k, 'desc': v} for k, v in {
'inner': 'only rows which match keys on all sheets',
'outer': 'all rows from first selected sheet',
'full': 'all rows from all sheets (union)',
'diff': 'only rows NOT in all sheets',
'append': 'columns all sheets; extend with rows from all sheets',
'extend': 'only rows from first sheet; extend with columns from all sheets',
'merge': 'merge differences from other sheets into first sheet',
}.items()]
def joinkey(sheet, row):
return tuple(c.getDisplayValue(row) for c in sheet.keyCols)
def groupRowsByKey(sheets, rowsBySheetKey, rowsByKey):
with Progress(gerund='grouping', total=sum(len(vs.rows) for vs in sheets)*2) as prog:
for vs in sheets:
# tally rows by keys for each sheet
rowsBySheetKey[vs] = collections.defaultdict(list)
for r in vs.rows:
prog.addProgress(1)
key = joinkey(vs, r)
rowsBySheetKey[vs][key].append(r)
for vs in sheets:
for r in vs.rows:
prog.addProgress(1)
key = joinkey(vs, r)
if key not in rowsByKey: # gather for this key has not been done yet
# multiplicative for non-unique keys
rowsByKey[key] = []
for crow in itertools.product(*[rowsBySheetKey[vs2].get(key, [None]) for vs2 in sheets]):
rowsByKey[key].append(list(crow))
class JoinKeyColumn(Column):
def __init__(self, name='', keycols=None, **kwargs):
super().__init__(name, type=keycols[0].type, width=keycols[0].width, **kwargs)
self.keycols = keycols
def calcValue(self, row):
vals = set()
for i, c in enumerate(self.keycols):
if row[i] is not None:
vals.add(c.getValue(row[i]))
if len(vals) == 1:
return vals.pop()
else:
raise Exception(f'inconsistent keys--reload join')
def putValue(self, row, value):
for i, c in enumerate(self.keycols):
if row[i] is not None:
c.setValues([row[i]], value)
def recalc(self, sheet=None):
Column.recalc(self, sheet)
for c in self.keycols:
c.recalc()
class MergeColumn(Column):
def calcValue(self, row):
for i, c in enumerate(self.cols):
if c:
v = c.getTypedValue(row[i])
if v and not isinstance(v, TypedWrapper):
return v
def putValue(self, row, value):
for r, c in zip(row, self.cols[::-1]):
if c:
c.setValue(r, value)
#### slicing and dicing
# rowdef: [sheet1_row, sheet2_row, ...]
# if a sheet does not have this key, sheet#_row is None
class JoinSheet(Sheet):
'Column-wise join/merge. `jointype` constructor arg should be one of jointypes.'
colorizers = [
CellColorizer(0, 'color_diff', lambda s,c,r,v: c and r and isinstance(c, MergeColumn) and c.cols[0] and v.value != c.cols[0].getValue(r[0]))
]
@asyncthread
def reload(self):
sheets = self.sources
# first item in joined row is the key tuple from the first sheet.
# first columns are the key columns from the first sheet, using its row (0)
self.columns = []
keyDict = collections.defaultdict(list)
for s in sheets:
for keyCol in s.keyCols:
keyDict[keyCol.name].append(keyCol)
for i, cols in enumerate(keyDict.values()):
self.addColumn(JoinKeyColumn(name=cols[0].name, keycols=cols)) # ColumnItem(c.name, i, sheet=sheets[0], type=c.type, width=c.width)))
self.setKeys(self.columns)
allcols = collections.defaultdict(lambda n=len(sheets): [None]*n)
for sheetnum, vs in enumerate(sheets):
for c in vs.nonKeyVisibleCols:
allcols[c.name][sheetnum] = c
if self.jointype == 'merge':
for colname, cols in allcols.items():
self.addColumn(MergeColumn(colname, cols=cols))
else:
ctr = collections.Counter(c.name for vs in sheets for c in vs.nonKeyVisibleCols)
for sheetnum, vs in enumerate(sheets):
# subsequent elements are the rows from each source, in order of the source sheets
for c in vs.nonKeyVisibleCols:
newname = c.name if ctr[c.name] == 1 else '%s_%s' % (vs.name, c.name)
self.addColumn(SubColumnItem(sheetnum, c, name=newname))
rowsBySheetKey = {}
rowsByKey = {}
groupRowsByKey(sheets, rowsBySheetKey, rowsByKey)
self.rows = []
with Progress(gerund='joining', total=len(rowsByKey)) as prog:
for k, combinedRows in rowsByKey.items():
prog.addProgress(1)
if self.jointype in ['full', 'merge']: # keep all rows from all sheets
for combinedRow in combinedRows:
self.addRow(combinedRow)
elif self.jointype == 'inner': # only rows with matching key on all sheets
for combinedRow in combinedRows:
if all(combinedRow):
self.addRow(combinedRow)
elif self.jointype == 'outer': # all rows from first sheet
for combinedRow in combinedRows:
if combinedRow[0]:
self.addRow(combinedRow)
elif self.jointype == 'diff': # only rows without matching key on all sheets
for combinedRow in combinedRows:
if not all(combinedRow):
self.addRow(combinedRow)
## for ExtendedSheet_reload below
class ExtendedColumn(Column):
def calcValue(self, row):
key = joinkey(self.sheet.joinSources[0], row)
srcsheet = self.sheet.joinSources[self.sheetnum]
srcrow = self.sheet.rowsBySheetKey[srcsheet][key]
if srcrow[0]:
return self.sourceCol.calcValue(srcrow[0])
@asyncthread
def ExtendedSheet_reload(self, sheets):
self.joinSources = sheets
# first item in joined row is the key tuple from the first sheet.
# first columns are the key columns from the first sheet, using its row (0)
self.columns = []
for i, c in enumerate(sheets[0].keyCols):
self.addColumn(copy(c))
self.setKeys(self.columns)
for i, c in enumerate(sheets[0].nonKeyVisibleCols):
self.addColumn(copy(c))
for sheetnum, vs in enumerate(sheets[1:]):
# subsequent elements are the rows from each source, in order of the source sheets
# ctr = collections.Counter(c.name for c in vs.nonKeyVisibleCols)
for c in vs.nonKeyVisibleCols:
newname = '%s_%s' % (vs.name, c.name)
newcol = ExtendedColumn(newname, sheetnum=sheetnum+1, sourceCol=c)
self.addColumn(newcol)
self.rowsBySheetKey = {} # [srcSheet][key] -> list(rowobjs from sheets[0])
rowsByKey = {} # [key] -> [rows0, rows1, ...]
groupRowsByKey(sheets, self.rowsBySheetKey, rowsByKey)
self.rows = []
with Progress(gerund='joining', total=len(rowsByKey)) as prog:
for k, combinedRows in rowsByKey.items():
prog.addProgress(1)
for combinedRow in combinedRows:
if combinedRow[0]:
self.addRow(combinedRow[0])
## for ConcatSheet
class ConcatColumn(Column):
def __init__(self, name, cols, **kwargs):
super().__init__(name, **kwargs)
self.cols = cols
def getColBySheet(self, s):
for c in self.cols:
if c.sheet is s:
return c
def calcValue(self, row):
srcSheet, srcRow = row
srcCol = self.getColBySheet(srcSheet)
if srcCol:
return srcCol.calcValue(srcRow)
def setValue(self, row, v):
srcSheet, srcRow = row
srcCol = self.getColBySheet(srcSheet)
if srcCol:
srcCol.setValue(srcRow, v)
else:
vd.fail('column not on source sheet')
# rowdef: (srcSheet, srcRow)
class ConcatSheet(Sheet):
'combination of multiple sheets by row concatenation. sourceCols=list(cols). '
@asyncthread
def reload(self):
self.rows = []
sourceSheets = []
for cols in self.sourceCols:
for c in cols:
if c.sheet not in sourceSheets:
sourceSheets.append(c.sheet)
self.columns = []
self.addColumn(ColumnItem('origin_sheet', 0, width=0))
for cols in self.sourceCols:
self.addColumn(ConcatColumn(cols[0].name, cols, type=cols[0].type))
for sheet in sourceSheets:
for r in Progress(sheet.rows):
self.addRow((sheet, r))
IndexSheet.addCommand('&', 'join-sheets', 'vd.push(createJoinedSheet(selectedRows or fail("no sheets selected to join"), jointype=chooseOne(jointypes)))', 'merge selected sheets with visible columns from all, keeping rows according to jointype')
Sheet.addCommand('&', 'join-sheets-top2', 'vd.push(createJoinedSheet(vd.sheets[:2], jointype=chooseOne(jointypes)))', 'concatenate top two sheets in Sheets Stack')
Sheet.addCommand('g&', 'join-sheets-all', 'vd.push(createJoinedSheet(vd.sheets, jointype=chooseOne(jointypes)))', 'concatenate all sheets in Sheets Stack')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608178530.0
visidata-2.2.1/visidata/layout.py 0000660 0001750 0001750 00000003666 00000000000 017515 0 ustar 00kefala kefala 0000000 0000000 from visidata import vd, Column, Sheet, options, Fanout
@Column.api
def setWidth(self, w):
if self.width != w:
if self.width == 0 or w == 0: # hide/unhide
vd.addUndo(setattr, self, '_width', self.width)
self._width = w
@Column.api
def toggleWidth(self, width):
'Change column width to either given `width` or default value.'
if self.width != width:
self.width = width
else:
self.width = int(options.default_width)
@Column.api
def toggleVisibility(self):
if self.height == 1:
self.height = options.default_height
else:
self.height = 1
def unhide_cols(cols, rows):
'sets appropriate width if column was either hidden (0) or unseen (None)'
for c in cols:
c.setWidth(abs(c.width or 0) or c.getMaxWidth(rows))
Sheet.addCommand('_', 'resize-col-max', 'cursorCol.toggleWidth(cursorCol.getMaxWidth(visibleRows))', 'toggle width of current column between full and default width'),
Sheet.addCommand('z_', 'resize-col-input', 'width = int(input("set width= ", value=cursorCol.width)); cursorCol.setWidth(width)', 'adjust width of current column to N')
Sheet.addCommand('g_', 'resize-cols-max', 'for c in visibleCols: c.setWidth(c.getMaxWidth(visibleRows))', 'toggle widths of all visible clumns between full and default width'),
Sheet.addCommand('gz_', 'resize-cols-input', 'width = int(input("set width= ", value=cursorCol.width)); Fanout(visibleCols).setWidth(width)', 'adjust widths of all visible columns to N')
Sheet.addCommand('-', 'hide-col', 'cursorCol.hide()', 'hide current column')
Sheet.addCommand('z-', 'resize-col-half', 'cursorCol.setWidth(cursorCol.width//2)', 'reduce width of current column by half'),
Sheet.addCommand('gv', 'unhide-cols', 'unhide_cols(columns, visibleRows)', 'unhide all columns')
Sheet.addCommand('v', 'visibility-sheet', 'for c in visibleCols: c.toggleVisibility()')
Sheet.addCommand('zv', 'visibility-col', 'cursorCol.toggleVisibility()')
././@PaxHeader 0000000 0000000 0000000 00000000034 00000000000 011452 x ustar 00 0000000 0000000 28 mtime=1612763830.9859815
visidata-2.2.1/visidata/loaders/ 0000770 0001750 0001750 00000000000 00000000000 017243 5 ustar 00kefala kefala 0000000 0000000 ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1606688371.0
visidata-2.2.1/visidata/loaders/__init__.py 0000660 0001750 0001750 00000000000 00000000000 021343 0 ustar 00kefala kefala 0000000 0000000 ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612075960.0
visidata-2.2.1/visidata/loaders/_pandas.py 0000660 0001750 0001750 00000033655 00000000000 021237 0 ustar 00kefala kefala 0000000 0000000 from functools import partial
from visidata import *
def open_pandas(p):
return PandasSheet(p.name, source=p)
def open_dta(p):
return PandasSheet(p.name, source=p, filetype='stata')
open_stata = open_pandas
for ft in 'feather gbq orc parquet pickle sas stata'.split():
globals().setdefault('open_'+ft, lambda p,ft=ft: PandasSheet(p.name, source=p, filetype=ft))
class DataFrameAdapter:
def __init__(self, df):
import pandas as pd
if not isinstance(df, pd.DataFrame):
vd.fail('%s is not a dataframe' % type(df).__name__)
self.df = df
def __len__(self):
if 'df' not in self.__dict__:
return 0
return len(self.df)
def __getitem__(self, k):
if isinstance(k, slice):
return DataFrameAdapter(self.df.iloc[k])
return self.df.iloc[k]
def __getattr__(self, k):
if 'df' not in self.__dict__:
raise AttributeError(f"'{self.__class__.__name__}' has no attribute '{k}'")
return getattr(self.df, k)
# source=DataFrame
class PandasSheet(Sheet):
'''Sheet sourced from a pandas.DataFrame
Warning:
The index of the pandas.DataFrame input must be unique.
Otherwise the selection functionality, which relies on
looking up selected rows via the index, will break.
This can be done by calling reset_index().
Note:
Columns starting with "__vd_" are reserved for internal usage
by the VisiData loader.
'''
def dtype_to_type(self, dtype):
import numpy as np
# Find the underlying numpy dtype for any pandas extension dtypes
dtype = getattr(dtype, 'numpy_dtype', dtype)
try:
if np.issubdtype(dtype, np.integer):
return int
if np.issubdtype(dtype, np.floating):
return float
if np.issubdtype(dtype, np.datetime64):
return date
except TypeError:
# For categoricals and other pandas-defined dtypes
pass
return anytype
def read_tsv(self, path, **kwargs):
'Partial function for reading TSV files using pd.read_csv'
import pandas as pd
return pd.read_csv(path, sep='\t', **kwargs)
@property
def df(self):
if isinstance(getattr(self, 'rows', None), DataFrameAdapter):
return self.rows.df
@df.setter
def df(self, val):
if isinstance(getattr(self, 'rows', None), DataFrameAdapter):
self.rows.df = val
else:
self.rows = DataFrameAdapter(val)
def getValue(self, col, row):
'''Look up column values in the underlying DataFrame.'''
return col.sheet.df.loc[row.name, col.name]
def setValue(self, col, row, val):
'''
Update a column's value in the underlying DataFrame, loosening the
column's type as needed. Take care to avoid assigning to a view or
a copy as noted here:
https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#why-does-assignment-fail-when-using-chained-indexing
'''
try:
col.sheet.df.loc[row.name, col.name] = val
except ValueError as err:
vd.warning(f'Type of {val} does not match column {col.name}. Changing type.')
col.type = anytype
col.sheet.df.loc[row.name, col.name] = val
def reload(self):
import pandas as pd
if isinstance(self.source, pd.DataFrame):
df = self.source
elif isinstance(self.source, Path):
filetype = getattr(self, 'filetype', self.source.ext)
if filetype == 'tsv':
readfunc = self.read_tsv
elif filetype == 'jsonl':
readfunc = partial(pd.read_json, lines=True)
else:
readfunc = getattr(pd, 'read_'+filetype) or vd.error('no pandas.read_'+filetype)
df = readfunc(str(self.source), **options.getall('pandas_'+filetype+'_'))
else:
try:
df = pd.DataFrame(self.source)
except ValueError as err:
vd.fail('error building pandas DataFrame from source data: %s' % err)
# reset the index here
if type(df.index) is not pd.RangeIndex:
df = df.reset_index()
# VisiData assumes string column names but pandas does not. Forcing string
# columns at load-time avoids various errors later.
df.columns = df.columns.astype(str)
self.columns = []
for col in (c for c in df.columns if not c.startswith("__vd_")):
self.addColumn(Column(
col,
type=self.dtype_to_type(df[col]),
getter=self.getValue,
setter=self.setValue
))
if self.columns[0].name == 'index': # if the df contains an index column
self.column('index').hide()
self.rows = DataFrameAdapter(df)
self._selectedMask = pd.Series(False, index=df.index)
if df.index.nunique() != df.shape[0]:
vd.warning("Non-unique index, row selection API may not work or may be incorrect")
@asyncthread
def sort(self):
'''Sort rows according to the current self._ordering.'''
by_cols = []
ascending = []
for col, reverse in self._ordering[::-1]:
by_cols.append(col.name)
ascending.append(not reverse)
self.rows.sort_values(by=by_cols, ascending=ascending, inplace=True)
def _checkSelectedIndex(self):
import pandas as pd
if self._selectedMask.index is not self.df.index:
# DataFrame was modified inplace, so the selection is no longer valid
vd.status('pd.DataFrame.index updated, clearing {} selected rows'
.format(self._selectedMask.sum()))
self._selectedMask = pd.Series(False, index=self.df.index)
def rowid(self, row):
return getattr(row, 'name', None) or ''
# Base selection API. Refer to GH #266: using id() will not identify
# pandas rows since iterating on rows / selecting rows will return
# different copies. Instead, re-implement the selection API by
# keeping a boolean pd.Series indicating the selected rows.
def isSelected(self, row):
if row is None:
return False
self._checkSelectedIndex()
return self._selectedMask.loc[row.name]
def selectRow(self, row):
'Select given row'
self._checkSelectedIndex()
self._selectedMask.loc[row.name] = True
def unselectRow(self, row):
self._checkSelectedIndex()
is_selected = self._selectedMask.loc[row.name]
self._selectedMask.loc[row.name] = False
return is_selected
@property
def nSelectedRows(self):
self._checkSelectedIndex()
return self._selectedMask.sum()
@property
def selectedRows(self):
self._checkSelectedIndex()
return DataFrameAdapter(self.df.loc[self._selectedMask])
# Vectorized implementation of multi-row selections
@asyncthread
def select(self, rows, status=True, progress=True):
self.addUndoSelection()
for row in (Progress(rows, 'selecting') if progress else rows):
self.selectRow(row)
@asyncthread
def unselect(self, rows, status=True, progress=True):
self.addUndoSelection()
for row in (Progress(rows, 'unselecting') if progress else rows):
self.unselectRow(row)
def clearSelected(self):
import pandas as pd
self._selectedMask = pd.Series(False, index=self.df.index)
def selectByIndex(self, start=None, end=None):
self._checkSelectedIndex()
self._selectedMask.iloc[start:end] = True
def unselectByIndex(self, start=None, end=None):
self._checkSelectedIndex()
self._selectedMask.iloc[start:end] = False
def toggleByIndex(self, start=None, end=None):
self._checkSelectedIndex()
self.addUndoSelection()
self._selectedMask.iloc[start:end] = ~self._selectedMask.iloc[start:end]
def _selectByILoc(self, mask, selected=True):
self._checkSelectedIndex()
self._selectedMask.iloc[mask] = selected
@asyncthread
def selectByRegex(self, regex, columns, unselect=False):
'''
Find rows matching regex in the provided columns. By default, add
matching rows to the selection. If unselect is True, remove from the
active selection instead.
'''
import pandas as pd
case_sensitive = 'I' not in vd.options.regex_flags
masks = pd.DataFrame([
self.df[col.name].astype(str).str.contains(pat=regex, case=case_sensitive, regex=True)
for col in columns
])
if unselect:
self._selectedMask = self._selectedMask & ~masks.any()
else:
self._selectedMask = self._selectedMask | masks.any()
def addUndoSelection(self):
vd.addUndo(undoAttrCopyFunc([self], '_selectedMask'))
@property
def nRows(self):
if self.df is None:
return 0
return len(self.df)
def newRows(self, n):
'''
Return n rows of empty data. Let pandas decide on the most
appropriate missing value (NaN, NA, etc) based on the underlying
DataFrame's dtypes.
'''
import pandas as pd
return pd.DataFrame({
col: [None] * n for col in self.df.columns
}).astype(self.df.dtypes.to_dict(), errors='ignore')
def _addRows(self, rows, idx):
import pandas as pd
if idx is None:
self.df = self.df.append(pd.DataFrame(rows))
else:
self.df = pd.concat((self.df.iloc[0:idx], pd.DataFrame(rows), self.df.iloc[idx:]))
self.df.index = pd.RangeIndex(self.nRows)
self._checkSelectedIndex()
def _deleteRows(self, which):
import pandas as pd
self.df.drop(which, inplace=True)
self.df.index = pd.RangeIndex(self.nRows)
self._checkSelectedIndex()
def addNewRows(self, n, idx=None):
self._addRows(self.newRows(n), idx)
idx = idx or self.nRows - 1
vd.addUndo(self._deleteRows, range(idx, idx + n))
def addRow(self, row, idx=None):
self._addRows([row], idx)
vd.addUndo(self._deleteRows, idx or self.nRows - 1)
def delete_row(self, rowidx):
import pandas as pd
oldrow = self.df.iloc[rowidx:rowidx+1]
# Use to_dict() here to work around an edge case when applying undos.
# As an action is undone, its entry gets removed from the cmdlog sheet.
# If we use `oldrow` directly, we get errors comparing DataFrame objects
# when there are multiple deletion commands for the same row index.
# There may be a better way to handle that case.
vd.addUndo(self._addRows, oldrow.to_dict(), rowidx)
self._deleteRows(rowidx)
vd.cliprows = [(self, rowidx, oldrow)]
def deleteBy(self, by):
'''Delete rows for which func(row) is true. Returns number of deleted rows.'''
import pandas as pd
oldidx = self.cursorRowIndex
nRows = self.nRows
vd.addUndo(setattr, self, 'df', self.df.copy())
self.df = self.df[~by]
self.df.index = pd.RangeIndex(self.nRows)
ndeleted = nRows - self.nRows
vd.status('deleted %s %s' % (ndeleted, self.rowtype))
return ndeleted
def deleteSelected(self):
'''Delete all selected rows.'''
self.deleteBy(self._selectedMask)
def view_pandas(df):
run(PandasSheet('', source=df))
# Override with vectorized implementations
PandasSheet.addCommand(None, 'stoggle-rows', 'toggleByIndex()', 'toggle selection of all rows')
PandasSheet.addCommand(None, 'select-rows', 'selectByIndex()', 'select all rows')
PandasSheet.addCommand(None, 'unselect-rows', 'unselectByIndex()', 'unselect all rows')
PandasSheet.addCommand(None, 'stoggle-before', 'toggleByIndex(end=cursorRowIndex)', 'toggle selection of rows from top to cursor')
PandasSheet.addCommand(None, 'select-before', 'selectByIndex(end=cursorRowIndex)', 'select all rows from top to cursor')
PandasSheet.addCommand(None, 'unselect-before', 'unselectByIndex(end=cursorRowIndex)', 'unselect all rows from top to cursor')
PandasSheet.addCommand(None, 'stoggle-after', 'toggleByIndex(start=cursorRowIndex)', 'toggle selection of rows from cursor to bottom')
PandasSheet.addCommand(None, 'select-after', 'selectByIndex(start=cursorRowIndex)', 'select all rows from cursor to bottom')
PandasSheet.addCommand(None, 'unselect-after', 'unselectByIndex(start=cursorRowIndex)', 'unselect all rows from cursor to bottom')
PandasSheet.addCommand(None, 'random-rows', 'nrows=int(input("random number to select: ", value=nRows)); vs=copy(sheet); vs.name=name+"_sample"; vs.rows=DataFrameAdapter(sheet.df.sample(nrows or nRows)); vd.push(vs)', 'open duplicate sheet with a random population subset of N rows'),
# Handle the regex selection family of commands through a single method,
# since the core logic is shared
PandasSheet.addCommand('|', 'select-col-regex', 'selectByRegex(regex=input("select regex: ", type="regex", defaultLast=True), columns=[cursorCol])', 'select rows matching regex in current column')
PandasSheet.addCommand('\\', 'unselect-col-regex', 'selectByRegex(regex=input("select regex: ", type="regex", defaultLast=True), columns=[cursorCol], unselect=True)', 'unselect rows matching regex in current column')
PandasSheet.addCommand('g|', 'select-cols-regex', 'selectByRegex(regex=input("select regex: ", type="regex", defaultLast=True), columns=visibleCols)', 'select rows matching regex in any visible column')
PandasSheet.addCommand('g\\', 'unselect-cols-regex', 'selectByRegex(regex=input("select regex: ", type="regex", defaultLast=True), columns=visibleCols, unselect=True)', 'unselect rows matching regex in any visible column')
# Override with a pandas/dataframe-aware implementation
PandasSheet.addCommand('"', 'dup-selected', 'vs=PandasSheet(sheet.name, "selectedref", source=selectedRows.df); vd.push(vs)', 'open duplicate sheet with only selected rows'),
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608541899.0
visidata-2.2.1/visidata/loaders/archive.py 0000660 0001750 0001750 00000005727 00000000000 021252 0 ustar 00kefala kefala 0000000 0000000 import codecs
import tarfile
import zipfile
from visidata import *
def open_zip(p):
return ZipSheet(p.name, source=p)
def open_tar(p):
return TarSheet(p.name, source=p)
open_tgz = open_tar
open_txz = open_tar
open_tbz2 = open_tar
class ZipSheet(Sheet):
'Wrapper for `zipfile` library.'
rowtype = 'files' # rowdef ZipInfo
columns = [
ColumnAttr('filename'),
ColumnAttr('file_size', type=int),
Column('date_time', type=date,
getter=lambda col, row: datetime.datetime(*row.date_time)),
ColumnAttr('compress_size', type=int)
]
def openZipFile(self, fp, *args, **kwargs):
'''Use VisiData input to handle password-protected zip files.'''
try:
return fp.open(*args, **kwargs)
except RuntimeError as err:
if 'password required' in err.args[0]:
pwd = vd.input(f'{args[0].filename} is encrypted, enter password: ', display=False)
return fp.open(*args, **kwargs, pwd=pwd.encode('utf-8'))
vd.error(err)
def openRow(self, fi):
decodedfp = codecs.iterdecode(self.openZipFile(self.zfp, fi),
encoding=options.encoding,
errors=options.encoding_errors)
return vd.openSource(Path(fi.filename, fp=decodedfp, filesize=fi.file_size), filetype=options.filetype)
@asyncthread
def extract(self, *rows, path=None):
self.zfp.extractall(members=[r.filename for r in rows], path=path)
@property
def zfp(self):
return zipfile.ZipFile(str(self.source), 'r')
def iterload(self):
with self.zfp as zf:
for zi in Progress(zf.infolist()):
yield zi
class TarSheet(Sheet):
'Wrapper for `tarfile` library.'
rowtype = 'files' # rowdef TarInfo
columns = [
ColumnAttr('name'),
ColumnAttr('size', type=int),
ColumnAttr('mtime', type=date),
ColumnAttr('type', type=int),
ColumnAttr('mode', type=int),
ColumnAttr('uname'),
ColumnAttr('gname')
]
def openRow(self, fi):
tfp = tarfile.open(name=str(self.source))
decodedfp = codecs.iterdecode(tfp.extractfile(fi),
encoding=options.encoding,
errors=options.encoding_errors)
return vd.openSource(Path(fi.name, fp=decodedfp, filesize=fi.size))
def iterload(self):
with tarfile.open(name=str(self.source)) as tf:
for ti in Progress(tf.getmembers()):
yield ti
ZipSheet.addCommand('x', 'extract-file', 'extract(cursorRow)')
ZipSheet.addCommand('gx', 'extract-selected', 'extract(*onlySelectedRows)')
ZipSheet.addCommand('zx', 'extract-file-to', 'extract(cursorRow, path=inputPath("extract to: "))')
ZipSheet.addCommand('gzx', 'extract-selected-to', 'extract(*onlySelectedRows, path=inputPath("extract %d files to: " % nSelected))')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608178530.0
visidata-2.2.1/visidata/loaders/csv.py 0000660 0001750 0001750 00000003771 00000000000 020421 0 ustar 00kefala kefala 0000000 0000000
from visidata import *
import csv
option('csv_dialect', 'excel', 'dialect passed to csv.reader', replay=True)
option('csv_delimiter', ',', 'delimiter passed to csv.reader', replay=True)
option('csv_quotechar', '"', 'quotechar passed to csv.reader', replay=True)
option('csv_skipinitialspace', True, 'skipinitialspace passed to csv.reader', replay=True)
option('csv_escapechar', None, 'escapechar passed to csv.reader', replay=True)
option('csv_lineterminator', '\r\n', 'lineterminator passed to csv.writer', replay=True)
option('safety_first', False, 'sanitize input/output to handle edge cases, with a performance cost', replay=True)
csv.field_size_limit(2**31-1) # Windows has max 32-bit
options_num_first_rows = 10
def open_csv(p):
return CsvSheet(p.name, source=p)
def removeNulls(fp):
for line in fp:
yield line.replace('\0', '')
class CsvSheet(SequenceSheet):
_rowtype = list # rowdef: list of values
def iterload(self):
'Convert from CSV, first handling header row specially.'
with self.source.open_text() as fp:
if options.safety_first:
rdr = csv.reader(removeNulls(fp), **options.getall('csv_'))
else:
rdr = csv.reader(fp, **options.getall('csv_'))
while True:
try:
yield next(rdr)
except csv.Error as e:
e.stacktrace=stacktrace()
yield [TypedExceptionWrapper(None, exception=e)]
except StopIteration:
return
@VisiData.api
def save_csv(vd, p, sheet):
'Save as single CSV file, handling column names as first line.'
with p.open_text(mode='w') as fp:
cw = csv.writer(fp, **options.getall('csv_'))
colnames = [col.name for col in sheet.visibleCols]
if ''.join(colnames):
cw.writerow(colnames)
with Progress(gerund='saving'):
for dispvals in sheet.iterdispvals(format=True):
cw.writerow(dispvals.values())
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608178530.0
visidata-2.2.1/visidata/loaders/eml.py 0000660 0001750 0001750 00000003562 00000000000 020401 0 ustar 00kefala kefala 0000000 0000000 from visidata import *
def open_eml(p):
return EmailSheet(p.name, source=p)
class EmailSheet(TableSheet):
rowtype = 'parts' # rowdef: sub-Messages
columns = [
Column('filename', getter=lambda c,r: r.get_filename()),
Column('content_type', getter=lambda c,r: r.get_content_type()),
Column('payload', type=vlen, getter=lambda c,r: r.get_payload(decode=False)),
]
def iterload(self):
import email
parser = email.parser.Parser()
with self.source.open_text() as fp:
yield from parser.parse(fp).walk()
@EmailSheet.api
def extract_part(sheet, givenpath, part):
with givenpath.open_bytes(mode='w') as fp:
fp.write(part.get_payload(decode=True))
@EmailSheet.api
def extract_parts(sheet, givenpath, *parts):
'Save all *parts* to Path *givenpath*.'
if givenpath.exists() and sheet.options.confirm_overwrite:
confirm("%s already exists. overwrite? " % givenpath.given)
vd.status('saving %s parts to %s' % (len(parts), givenpath.given))
# forcibly specify save individual files into directory by ending path with /
if givenpath.is_dir() or givenpath.given.endswith('/') or len(parts) > 1:
# save as individual files in the givenpath directory
try:
os.makedirs(givenpath, exist_ok=True)
except FileExistsError:
pass
for part in parts:
vd.execAsync(sheet.extract_part, givenpath / part.get_filename(), part)
elif len(parts) == 1:
vd.execAsync(sheet.extract_part, givenpath, part)
else:
vd.fail('cannot save multiple parts to non-dir')
EmailSheet.addCommand('x', 'extract-part', 'extract_part(inputPath("save part as: ", value=cursorRow.get_filename()), cursorRow)')
EmailSheet.addCommand('gx', 'extract-part-selected', 'extract_parts(inputPath("save %d parts in: " % nSelectedRows), *selectedRows)')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608541899.0
visidata-2.2.1/visidata/loaders/fixed_width.py 0000660 0001750 0001750 00000005602 00000000000 022117 0 ustar 00kefala kefala 0000000 0000000
from visidata import *
option('fixed_rows', 1000, 'number of rows to check for fixed width columns')
option('fixed_maxcols', 0, 'max number of fixed-width columns to create (0 is no max)')
def open_fixed(p):
return FixedWidthColumnsSheet(p.name, source=p, headerlines=[])
class FixedWidthColumn(Column):
def __init__(self, name, i, j, **kwargs):
super().__init__(name, **kwargs)
self.i, self.j = i, j
def calcValue(self, row):
return row[0][self.i:self.j]
def putValue(self, row, value):
value = str(value)[:self.j-self.i]
j = self.j or len(row)
row[0] = row[0][:self.i] + '%-*s' % (j-self.i, value) + row[0][self.j:]
def columnize(rows):
'Generate (i,j) indexes for fixed-width columns found in rows'
## find all character columns that are not spaces ever
allNonspaces = set()
for r in rows:
for i, ch in enumerate(r):
if not ch.isspace():
allNonspaces.add(i)
colstart = 0
prev = 0
# collapse fields
for i in allNonspaces:
if i > prev+1:
yield colstart, i
colstart = i
prev = i
yield colstart, None # final column gets rest of line
class FixedWidthColumnsSheet(SequenceSheet):
rowtype = 'lines' # rowdef: [line] (wrapping in list makes it unique and modifiable)
def addRow(self, row, index=None):
Sheet.addRow(self, row, index=index)
def iterload(self):
itsource = iter(self.source)
# compute fixed width columns from first fixed_rows lines
maxcols = self.options.fixed_maxcols
self.columns = []
fixedRows = list([x] for x in self.optlines(itsource, 'fixed_rows'))
for i, j in columnize(list(r[0] for r in fixedRows)):
if maxcols and self.nCols >= maxcols-1:
self.addColumn(FixedWidthColumn('', i, None))
break
else:
self.addColumn(FixedWidthColumn('', i, j))
yield from fixedRows
self.setColNames(self.headerlines)
yield from ([line] for line in itsource)
def setCols(self, headerlines):
self.headerlines = headerlines
@VisiData.api
def save_fixed(vd, p, *vsheets):
with p.open_text(mode='w') as fp:
for sheet in vsheets:
if len(vsheets) > 1:
fp.write('%s\n\n' % vs.name)
# headers
for col in sheet.visibleCols:
fp.write('{0:{width}}'.format(col.name, width=col.width))
fp.write('\n')
# rows
with Progress(gerund='saving'):
for dispvals in sheet.iterdispvals(format=True):
for col, val in dispvals.items():
fp.write('{0:{align}{width}}'.format(val, width=col.width, align='>' if vd.isNumeric(col) else '<'))
fp.write('\n')
vd.status('%s save finished' % p)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608541899.0
visidata-2.2.1/visidata/loaders/frictionless.py 0000660 0001750 0001750 00000000653 00000000000 022326 0 ustar 00kefala kefala 0000000 0000000 from visidata import *
def open_frictionless(p):
return FrictionlessIndexSheet(p.name, source=p)
class FrictionlessIndexSheet(IndexSheet):
def iterload(self):
import datapackage
self.dp = datapackage.Package(self.source.open_text())
for r in Progress(self.dp.resources):
yield vd.openSource(self.source.with_name(r.descriptor['path']), filetype=r.descriptor.get('format', 'json'))
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612075960.0
visidata-2.2.1/visidata/loaders/geojson.py 0000660 0001750 0001750 00000010575 00000000000 021272 0 ustar 00kefala kefala 0000000 0000000 from functools import reduce
import json
from visidata import *
def open_geojson(p):
return GeoJSONSheet(p.name, source=p)
def getter_factory(prop):
def getter(col, row):
return row.get('properties', {}).get(prop)
return getter
class GeoJSONSheet(PythonSheet):
rowtype = 'shapes'
def iterload(self):
self.colnames = {}
with self.source.open_text() as fp:
ret = json.load(fp)
if ret['type'] == 'FeatureCollection':
features = ret['features']
elif ret['type'] == 'Feature':
features = [ret]
elif ret['type'] == 'GeometryCollection':
features = list(map(lambda g: { 'type': 'Feature', 'geometry': g }, ret['geometries']))
else: # Some form of geometry
features = [{ 'type': 'Feature', 'geometry': ret }]
for feature in Progress(features):
for prop in feature.get('properties', {}).keys():
if prop not in self.colnames:
c = Column(name=prop, getter=getter_factory(prop))
self.colnames[prop] = c
self.addColumn(c)
yield feature
class GeoJSONMap(InvertedCanvas):
aspectRatio = 1.0
filetype = 'geojson'
@asyncthread
def reload(self):
self.reset()
for row in Progress(self.sourceRows):
k = self.source.rowkey(row)
colour = self.plotColor(k)
bbox = self.parse_geometry(row, colour)
x1, y1, x2, y2 = bbox
textx, texty = (x1+x2)/2, (y1+y2)/2
disptext = self.textCol.getDisplayValue(row)
self.label(textx, texty, disptext, colour, row)
self.refresh()
def parse_geometry(self, row, colour, bbox=None):
if bbox is None: bbox = [180, 90, -180, -90]
typ = row['geometry']['type']
if typ == 'GeometryCollection':
for g in row['geometries']:
bbox = self.parse_geometry(row, colour, bbox)
return bbox
coords = row['geometry']['coordinates']
if typ in ('Point', 'LineString', 'Polygon'):
coords = [coords]
if typ in ('Point', 'MultiPoint'):
for x, y in coords:
self.point(x, y, colour, row)
bbox = reduce_coords(coords, bbox)
elif typ in ('LineString', 'MultiLineString'):
for line in coords:
self.polyline(line, colour, row)
bbox = reduce_coords(line, bbox)
elif typ in ('Polygon', 'MultiPolygon'):
for polygon in coords:
self.polygon(polygon[0], colour, row)
bbox = reduce_coords(polygon[0], bbox)
for hole in polygon[1:]:
self.polygon(hole, 0, row)
else:
vd.status('notimpl shapeType %s' % typ)
return bbox
def reduce_coords(coords, initial):
return reduce(
lambda a,n: [min(a[0],n[0]), min(a[1],n[1]), max(a[2],n[0]), max(a[3],n[1])],
coords, initial)
@GeoJSONMap.api
def save_geojson(vd, p, vs):
features = []
visibleCols = list(map(lambda c: c.name, vs.source.visibleCols))
for row in Progress(vs.sourceRows, 'saving'):
row = deepcopy(row)
row['properties'] = {k:v for k,v in row.get('properties', {}).items() if k in visibleCols}
features.append(row)
featcoll = {
'type': 'FeatureCollection',
'features': features,
}
try:
indent = int(vs.options.json_indent)
except Exception:
indent = vs.options.json_indent
with p.open_text(mode='w') as fp:
encoder = json.JSONEncoder(indent=indent, sort_keys=vs.options.json_sort_keys)
for chunk in encoder.iterencode(featcoll):
fp.write(chunk)
GeoJSONSheet.addCommand('.', 'plot-row', 'vd.push(GeoJSONMap(name+"_map", sourceRows=[cursorRow], textCol=cursorCol, source=sheet))', 'plot geospatial vector in current row')
GeoJSONSheet.addCommand('g.', 'plot-rows', 'vd.push(GeoJSONMap(name+"_map", sourceRows=rows, textCol=cursorCol, source=sheet))', 'plot all geospatial vectors in current sheet')
GeoJSONMap.addCommand('^S', 'save-sheet', 'vd.saveSheets(inputPath("save to: ", value=getDefaultSaveName(sheet)), sheet, confirm_overwrite=options.confirm_overwrite)', 'save current sheet to filename in format determined by extension (default .geojson)')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608541899.0
visidata-2.2.1/visidata/loaders/graphviz.py 0000660 0001750 0001750 00000004347 00000000000 021460 0 ustar 00kefala kefala 0000000 0000000 from visidata import vd, options, option, TypedWrapper, asyncthread, Progress
from visidata import wrapply, clean_to_id, VisiData, SIFormatter
option('graphviz_edge_labels', True, 'whether to include edge labels on graphviz diagrams')
def is_valid(v):
if v is None:
return False
if isinstance(v, TypedWrapper):
return False
return True
@VisiData.api
def save_dot(vd, p, vs):
unusedColors = 'orange green purple cyan red blue black'.split()
assignedColors = {}
srccol = vs.keyCols[0]
dstcol = vs.keyCols[1]
with p.open_text(mode='w') as fp:
print('graph { concentrate=true;', file=fp)
for row in Progress(vs.rows, 'saving'):
src = srccol.getTypedValue(row)
dst = dstcol.getTypedValue(row)
if not is_valid(src) or not is_valid(dst):
continue
downsrc = clean_to_id(str(src)) or src
downdst = clean_to_id(str(dst)) or dst
edgenotes = [c.getTypedValue(row) for c in vs.nonKeyVisibleCols if not vd.isNumeric(c)]
edgetype = '-'.join(str(x) for x in edgenotes if is_valid(x))
color = assignedColors.get(edgetype, None)
if not color:
color = unusedColors.pop() if unusedColors else 'black'
assignedColors[edgetype] = color
if options.graphviz_edge_labels:
nodelabels = [wrapply(SIFormatter, '%0.1f', c.getTypedValue(row)) for c in vs.nonKeyVisibleCols if vd.isNumeric(c)]
label = '/'.join(str(x) for x in nodelabels if is_valid(x))
else:
label = ''
print('\t%s[label="%s"];' % (downsrc, src), file=fp)
print('\t%s[label="%s"];' % (downdst, dst), file=fp)
print('\t%s -- %s[label="%s", color=%s];' % (downsrc, downdst, label, color), file=fp)
print('label="%s"' % vs.name, file=fp)
print('node[shape=plaintext];', file=fp)
print('subgraph cluster_legend {', file=fp)
print('label="Legend";', file=fp)
for i, (k, color) in enumerate(assignedColors.items()):
print('key%d[label="%s", fontcolor=%s];' % (i, k, color), file=fp)
print('}', file=fp) # legend subgraph
print('}', file=fp)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608541899.0
visidata-2.2.1/visidata/loaders/hdf5.py 0000660 0001750 0001750 00000004135 00000000000 020447 0 ustar 00kefala kefala 0000000 0000000 from visidata import *
def open_h5(p):
return Hdf5ObjSheet(p.name, source=p)
open_hdf5 = open_h5
class Hdf5ObjSheet(Sheet):
'Support sheets in HDF5 format.'
def iterload(self):
import h5py
source = self.source
if isinstance(self.source, Path):
source = h5py.File(str(self.source), 'r')
self.columns = []
if isinstance(source, h5py.Group):
self.rowtype = 'sheets'
self.columns = [
Column(source.name, type=str, getter=lambda col,row: row.source.name.split('/')[-1], keycol=1),
Column('type', type=str, getter=lambda col,row: type(row.source).__name__),
Column('nItems', type=int, getter=lambda col,row: len(row.source)),
]
self.recalc()
for k, v in source.items():
yield Hdf5ObjSheet(self.name, k, source=v)
elif isinstance(source, h5py.Dataset):
if len(source.shape) == 1:
for i, colname in enumerate(source.dtype.names or [0]):
self.addColumn(ColumnItem(colname, colname), index=i)
yield from source # copy
elif len(source.shape) == 2: # matrix
ncols = source.shape[1]
for i in range(ncols):
self.addColumn(ColumnItem('', i, width=8), index=i)
self.recalc()
yield from source # copy
else:
vd.status('too many dimensions in shape %s' % str(source.shape))
else:
vd.status('unknown h5 object type %s' % type(source))
def openRow(self, row):
import h5py
if isinstance(row, BaseSheet):
return row
if isinstance(row, h5py.HLObject):
return Hdf5ObjSheet(row)
import numpy
from .npy import NpySheet
if isinstance(row, numpy.ndarray):
return NpySheet(None, npy=row)
Hdf5ObjSheet.addCommand('A', 'dive-metadata', 'vd.push(SheetDict(cursorRow.name + "_attrs", source=cursorRow.attrs))', 'open metadata sheet for object referenced in current row')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612075960.0
visidata-2.2.1/visidata/loaders/html.py 0000660 0001750 0001750 00000011374 00000000000 020570 0 ustar 00kefala kefala 0000000 0000000 import html
from visidata import *
option('html_title', '
{sheet.name}
', 'table header when saving to html')
def open_html(p):
return HtmlTablesSheet(p.name, source=p)
open_htm = open_html
class HtmlTablesSheet(IndexSheet):
rowtype = 'sheets' # rowdef: HtmlTableSheet (sheet.html = lxml.html.HtmlElement)
columns = IndexSheet.columns + [
Column('tag', width=0, getter=lambda col,row: row.html.tag),
Column('id', getter=lambda col,row: row.html.attrib.get('id')),
Column('classes', getter=lambda col,row: row.html.attrib.get('class')),
]
def iterload(self):
import lxml.html
from lxml import etree
utf8_parser = etree.HTMLParser(encoding='utf-8')
with self.source.open_text() as fp:
html = lxml.html.etree.parse(fp, parser=utf8_parser)
self.setKeys([self.column('name')])
self.column('keys').hide()
self.column('source').hide()
for i, e in enumerate(html.iter('table')):
if e.tag == 'table':
yield HtmlTableSheet(e.attrib.get("id", "table_" + str(i)), source=e, html=e)
def is_header(elem):
scope = elem.attrib.get('scope', '')
if elem.tag == 'th':
if not scope or scope == 'col':
return True
return False
class HtmlTableSheet(Sheet):
rowtype = 'rows' # list of strings
columns = []
def iterload(self):
headers = []
maxlinks = {} # [colnum] -> nlinks:int
for rownum, r in enumerate(self.source.iter('tr')):
row = []
colnum = 0
# get starting column, which might be different if there were rowspan>1 already
if rownum < len(headers):
while colnum < len(headers[rownum]):
if headers[rownum][colnum] is None:
break
colnum += 1
for cell in r.getchildren():
colspan = int(cell.attrib.get('colspan', 1))
rowspan = int(cell.attrib.get('rowspan', 1))
cellval = ' '.join(x.strip() for x in cell.itertext()) # text only without markup
links = [x.get('href') for x in cell.iter('a')]
maxlinks[colnum] = max(maxlinks.get(colnum, 0), len(links))
if is_header(cell):
for k in range(rownum, rownum+rowspan):
while k >= len(headers): # extend headers list with lists for all header rows
headers.append([])
for j in range(colnum, colnum+colspan):
while j >= len(headers[k]):
headers[k].append(None)
headers[k][j] = cellval
cellval = '' # use empty non-None value for subsequent rows in the rowspan
else:
while colnum >= len(row):
row.append(None)
row[colnum] = (cellval, links)
colnum += colspan
if any(row):
yield row
self.columns = []
if headers:
it = itertools.zip_longest(*headers, fillvalue='')
else:
it = [list(x) for x in self.rows[0]]
self.rows = self.rows[1:]
for colnum, names in enumerate(it):
name = '_'.join(str(x) for x in names if x)
self.addColumn(Column(name, getter=lambda c,r,i=colnum: r[i][0]))
for linknum in range(maxlinks.get(colnum, 0)):
self.addColumn(Column(name+'_link'+str(linknum), width=20, getter=lambda c,r,i=colnum,j=linknum: r[i][1][j]))
@VisiData.api
def save_html(vd, p, *vsheets):
'Save vsheets as HTML tables in a single file'
with open(p, 'w', encoding='ascii', errors='xmlcharrefreplace') as fp:
for sheet in vsheets:
if options.html_title:
fp.write(options.html_title.format(sheet=sheet, vd=vd))
fp.write('
')
for col in sheet.visibleCols:
contents = html.escape(col.name)
fp.write('
{colname}
'.format(colname=contents))
fp.write('
\n')
# rows
with Progress(gerund='saving'):
for dispvals in sheet.iterdispvals(format=True):
fp.write('
')
for val in dispvals.values():
fp.write('
')
fp.write(html.escape(val))
fp.write('
')
fp.write('
\n')
fp.write('
')
vd.status('%s save finished' % p)
VisiData.save_htm = VisiData.save_html
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612075960.0
visidata-2.2.1/visidata/loaders/http.py 0000660 0001750 0001750 00000003764 00000000000 020607 0 ustar 00kefala kefala 0000000 0000000 from visidata import Path, RepeatFile, options, vd
content_filetypes = {
'tab-separated-values': 'tsv'
}
vd.option('http_max_next', 0, 'max next.url pages to follow in http response') #848
def openurl_http(path, filetype=None):
import requests
response = requests.get(path.given, stream=True)
response.raise_for_status()
if not filetype:
# try auto-detect from extension
ext = path.suffix[1:].lower()
openfunc = vd.getGlobals().get(f'open_{ext}')
if openfunc:
filetype = ext
else:
# if extension unknown, fallback to mime-type
contenttype = response.headers['content-type']
subtype = contenttype.split(';')[0].split('/')[-1]
filetype = content_filetypes.get(subtype, subtype)
# If no charset is provided by response headers, use the user-specified
# encoding option (which defaults to UTF-8) and hope for the best. The
# alternative is an error because iter_lines() will produce bytes. We're
# streaming so can't use response.apparent_encoding.
if not response.encoding:
response.encoding = options.encoding
# Automatically paginate if a 'next' URL is given
def _iter_lines(path=path, response=response, max_next=options.http_max_next):
path.responses = []
n = 0
while response:
path.responses.append(response)
yield from response.iter_lines(decode_unicode=True)
src = response.links.get('next', {}).get('url', None)
if not src:
break
n += 1
if n > max_next:
vd.warning(f'stopping at max {max_next} pages')
break
vd.status(f'fetching next page from {src}')
response = requests.get(src, stream=True)
# add resettable iterator over contents as an already-open fp
path.fp = RepeatFile(iter_lines=_iter_lines())
return vd.openSource(path, filetype=filetype)
openurl_https = openurl_http
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608178530.0
visidata-2.2.1/visidata/loaders/imap.py 0000660 0001750 0001750 00000004065 00000000000 020551 0 ustar 00kefala kefala 0000000 0000000 from visidata import vd, TableSheet, asyncthread, ColumnItem, Column, ColumnAttr, Progress
from urllib.parse import urlparse
def openurl_imap(p, **kwargs):
url = urlparse(p.given)
password = url.password or vd.error('no password given in url') # vd.input("imap password for %s" % user, display=False))
return ImapSheet(url.hostname, source=url, password=password)
class ImapSheet(TableSheet):
columns = [
ColumnItem('message-id'),
ColumnItem('folder'),
ColumnItem('Date'),
ColumnItem('From'),
ColumnItem('To'),
ColumnItem('Subject'),
ColumnAttr('defects'),
Column('payload', getter=lambda c,r: r.get_payload()),
Column('content_type', getter=lambda c,r: r.get_content_type()),
]
nKeys = 1
@asyncthread
def reload(self):
import imaplib
import email.parser
m = imaplib.IMAP4_SSL(host=self.source.hostname)
user = self.source.username
m.login(user, self.password)
typ, folders = m.list()
for r in Progress(folders, gerund="downloading"):
fname = r.decode('utf-8').split()[-1][1:-1]
try:
m.select(fname)
typ, data = m.search(None, 'ALL')
for num in data[0].split():
typ, msgbytes = m.fetch(num, '(RFC822)')
if typ != 'OK':
vd.warning(typ, msgbytes)
continue
msg = email.message_from_bytes(msgbytes[0][1])
msg['folder'] = fname
self.addRow(msg)
m.close()
except Exception:
vd.exceptionCaught()
m.logout()
def addRow(self, row, **kwargs):
if row.is_multipart():
for p in row.get_payload():
for hdr in 'message-id folder Date From To Subject'.split():
if hdr in row:
p[hdr] = row[hdr]
self.addRow(p, **kwargs)
else:
super().addRow(row, **kwargs)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608541899.0
visidata-2.2.1/visidata/loaders/json.py 0000660 0001750 0001750 00000010261 00000000000 020567 0 ustar 00kefala kefala 0000000 0000000 import json
from collections import OrderedDict
from visidata import *
option('json_indent', None, 'indent to use when saving json')
option('json_sort_keys', False, 'sort object keys when saving to json')
option('default_colname', '', 'column name to use for non-dict rows')
def open_jsonobj(p):
return JsonSheet(p.name, source=p)
def open_jsonl(p):
return JsonSheet(p.name, source=p)
open_ndjson = open_ldjson = open_json = open_jsonl
class JsonSheet(PythonSheet):
def iterload(self):
self.colnames = {} # [colname] -> Column
self.columns = []
with self.source.open_text() as fp:
for L in fp:
try:
if L.startswith('#'): # skip commented lines
continue
ret = json.loads(L)
if isinstance(ret, list):
yield from Progress(ret)
else:
yield ret
except ValueError as e:
if self.rows: # if any rows have been added already
e.stacktrace = stacktrace()
yield TypedExceptionWrapper(json.loads, L, exception=e) # an error on one line
else:
with self.source.open_text() as fp:
ret = json.load(fp)
if isinstance(ret, list):
yield from Progress(ret)
else:
yield ret
break
def addRow(self, row, index=None):
# Wrap non-dict rows in a dummy object with a predictable key name.
# This allows for more consistent handling of rows containing scalars
# or lists.
if not isinstance(row, dict):
v = {options.default_colname: row}
row = visidata.AlwaysDict(row, **v)
super().addRow(row, index=index)
for k in row:
if k not in self.colnames:
c = ColumnItem(k, type=deduceType(row[k]))
self.colnames[k] = c
self.addColumn(c)
return row
def newRow(self):
return {}
JsonLinesSheet=JsonSheet
## saving json and jsonl
class Cell:
def __init__(self, col, row):
self.col = col
self.row = row
@property
def value(cell):
o = wrapply(cell.col.getTypedValue, cell.row)
if isinstance(o, TypedExceptionWrapper):
return options.safe_error or str(o.exception)
elif isinstance(o, TypedWrapper):
return o.val
elif isinstance(o, date):
return cell.col.getDisplayValue(cell.row)
return o
class _vjsonEncoder(json.JSONEncoder):
def __init__(self, **kwargs):
super().__init__(sort_keys=options.json_sort_keys, **kwargs)
self.safe_error = options.safe_error
def default(self, obj):
return obj.value if isinstance(obj, Cell) else str(obj)
def _rowdict(cols, row):
ret = {}
for c in cols:
cell = Cell(c, row)
if cell.value is not None:
ret[c.name] = cell
return ret
@VisiData.api
def save_json(vd, p, *vsheets):
with p.open_text(mode='w') as fp:
if len(vsheets) == 1:
vs = vsheets[0]
it = [_rowdict(vs.visibleCols, row) for row in vs.iterrows()]
else:
it = {vs.name: [_rowdict(vs.visibleCols, row) for row in vs.iterrows()] for vs in vsheets}
try:
indent = int(options.json_indent)
except Exception:
indent = options.json_indent
jsonenc = _vjsonEncoder(indent=indent)
with Progress(gerund='saving'):
for chunk in jsonenc.iterencode(it):
fp.write(chunk)
@VisiData.api
def save_jsonl(vd, p, *vsheets):
with p.open_text(mode='w') as fp:
for vs in vsheets:
vcols = vs.visibleCols
jsonenc = _vjsonEncoder()
with Progress(gerund='saving'):
for row in vs.iterrows():
rowdict = _rowdict(vcols, row)
fp.write(jsonenc.encode(rowdict) + '\n')
VisiData.save_ndjson = VisiData.save_jsonl
VisiData.save_ldjson = VisiData.save_jsonl
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608541899.0
visidata-2.2.1/visidata/loaders/markdown.py 0000660 0001750 0001750 00000003140 00000000000 021436 0 ustar 00kefala kefala 0000000 0000000 from visidata import *
def markdown_escape(s, style='orgmode'):
if style == 'jira':
return s
ret = ''
for ch in s:
if ch in '\\`*_{}[]()>#+-.!':
ret += '\\'+ch
else:
ret += ch
return ret
def markdown_colhdr(col):
if vd.isNumeric(col):
return ('-' * (col.width-1)) + ':'
else:
return '-' * (col.width or options.default_width)
def write_md(p, *vsheets, md_style='orgmode'):
'pipe tables compatible with org-mode'
if md_style == 'jira':
delim = '||'
else:
delim = '|'
with p.open_text(mode='w') as fp:
for vs in vsheets:
if len(vsheets) > 1:
fp.write('# %s\n\n' % vs.name)
fp.write(delim + delim.join('%-*s' % (col.width or options.default_width, markdown_escape(col.name, md_style)) for col in vs.visibleCols) + '|\n')
if md_style == 'orgmode':
fp.write('|' + '|'.join(markdown_colhdr(col) for col in vs.visibleCols) + '|\n')
with Progress(gerund='saving'):
for dispvals in vs.iterdispvals(format=True):
s = '|'
for col, val in dispvals.items():
s += '%-*s|' % (col.width or options.default_width, markdown_escape(val, md_style))
s += '\n'
fp.write(s)
fp.write('\n')
vd.status('%s save finished' % p)
@VisiData.api
def save_md(vd, p, *sheets):
write_md(p, *sheets, md_style='orgmode')
@VisiData.api
def save_jira(vd, p, *sheets):
write_md(p, *sheets, md_style='jira')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608541899.0
visidata-2.2.1/visidata/loaders/mbtiles.py 0000660 0001750 0001750 00000010712 00000000000 021256 0 ustar 00kefala kefala 0000000 0000000 from visidata import *
import json
import gzip
import sqlite3
def open_pbf(p):
return PbfSheet(p.name, source=p)
def open_mbtiles(p):
return MbtilesSheet(p.name, source=p)
def getListDepth(L):
if not isinstance(L, list):
return 0
if len(L) == 0:
return 0
return getListDepth(L[0]) + 1
def getFeatures(tile_data):
for layername, layer in tile_data.items():
for feat in layer['features']:
yield layername, feat
def tilename(row):
return ",".join(str(x) for x in row)
class MbtilesSheet(Sheet):
columns = [
ColumnItem('zoom_level', 0),
ColumnItem('tile_column', 1),
ColumnItem('tile_row', 2),
]
def getTile(self, zoom_level, tile_col, tile_row):
import mapbox_vector_tile
con = sqlite3.connect(str(self.source))
tile_data = con.execute('''
SELECT tile_data FROM tiles
WHERE zoom_level = ?
AND tile_column = ?
AND tile_row = ?''', (zoom_level, tile_col, tile_row)).fetchone()[0]
return mapbox_vector_tile.decode(gzip.decompress(tile_data))
def iterload(self):
con = sqlite3.connect(str(self.source))
self.metadata = dict(con.execute('SELECT name, value FROM metadata').fetchall())
tiles = con.execute('SELECT zoom_level, tile_column, tile_row FROM tiles')
yield from Progress(tiles.fetchall())
def getPlot(self, *rows):
if len(rows) == 1:
name = self.name+'_'+tilename(rows[0])
else:
name = self.name+'_selected'
sourceRows = sum((list(getFeatures(self.getTile(*r))) for r in rows), [])
return PbfCanvas(name+"_map", source=PbfSheet(name, source=self), sourceRows=sourceRows)
def openRow(self, row):
'load table referenced in current row into memory'
return PbfSheet(tilename(row), source=self, sourceRow=row)
class PbfSheet(Sheet):
columns = [
ColumnItem('layer', 0),
Column('geometry_type', getter=lambda col,row: row[1]['geometry']['type']),
Column('geometry_coords', getter=lambda col,row: row[1]['geometry']['coordinates'], width=0),
Column('geometry_coords_depth', getter=lambda col,row: getListDepth(row[1]['geometry']['coordinates']), width=0),
]
nKeys = 1 # layer
def iterload(self):
props = set() # property names
for r in getFeatures(self.source.getTile(*self.sourceRow)):
yield r
props.update(r[1]['properties'].keys())
for key in props:
self.addColumn(Column(key, getter=lambda col,row,key=key: row[1]['properties'][key]))
class PbfCanvas(InvertedCanvas):
aspectRatio = 1.0
def iterpolylines(self, r):
layername, feat = r
geom = feat['geometry']
t = geom['type']
coords = geom['coordinates']
key = self.source.rowkey(r)
if t == 'LineString':
yield coords, self.plotColor(key), r
elif t == 'Point':
yield [coords], self.plotColor(key), r
elif t == 'Polygon':
for poly in coords:
yield poly+[poly[0]], self.plotColor(key), r
elif t == 'MultiLineString':
for line in coords:
yield line, self.plotColor(key), r
elif t == 'MultiPolygon':
for mpoly in coords:
for poly in mpoly:
yield poly+[poly[0]], self.plotColor(key), r
else:
vd.warning('unknown geometry type %s' % t)
@asyncthread
def reload(self):
self.reset()
for r in Progress(self.sourceRows):
for vertexes, attr, row in self.iterpolylines(r):
self.polyline(vertexes, attr, row)
if len(vertexes) == 1:
textx, texty = vertexes[0]
disptext = self.textCol.getDisplayValue(row)
if disptext:
self.label(textx, texty, disptext, attr, row)
self.refresh()
PbfSheet.addCommand('.', 'plot-row', 'vd.push(PbfCanvas(name+"_map", source=sheet, sourceRows=[cursorRow], textCol=cursorCol))', 'plot blocks in current row')
PbfSheet.addCommand('g.', 'plot-rows', 'vd.push(PbfCanvas(name+"_map", source=sheet, sourceRows=rows, textCol=cursorCol))', 'plot selected blocks')
MbtilesSheet.addCommand('.', 'plot-row', 'vd.push(getPlot(cursorRow))', 'plot tiles in current row')
MbtilesSheet.addCommand('g.', 'plot-selected', 'vd.push(getPlot(*selectedRows))', 'plot selected tiles'),
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612075960.0
visidata-2.2.1/visidata/loaders/mysql.py 0000660 0001750 0001750 00000007256 00000000000 020775 0 ustar 00kefala kefala 0000000 0000000 from visidata import *
from contextlib import contextmanager
__all__ = ['openurl_mysql', 'MyTable', 'MyTablesSheet']
def codeToType(type_code, colname):
import MySQLdb as mysql
types = mysql.constants.FIELD_TYPE
if type_code in (types.TINY, types.SHORT, types.LONG, types.LONGLONG, types.INT24,):
return int
if type_code in (types.FLOAT, types.DOUBLE, types.DECIMAL, types.NEWDECIMAL,):
return float
if type_code == mysql.STRING:
return str
return anytype
def openurl_mysql(url, filetype=None):
url = urlparse(url.given)
dbname = url.path[1:]
return MyTablesSheet(dbname+"_tables", sql=SQL(url), schema=dbname)
class SQL:
def __init__(self, url):
self.url = url
@contextmanager
def cur(self, qstr):
import MySQLdb as mysql
import MySQLdb.cursors as cursors
dbname = self.url.path[1:]
connection = mysql.connect(
user=self.url.username,
database=self.url.path[1:],
host=self.url.hostname,
port=self.url.port or 3306,
password=self.url.password,
use_unicode=True,
charset='utf8',
cursorclass=cursors.SSCursor) ## if SSCursor is not used mysql will first fetch ALL data, and only then visualize it
try:
cursor = connection.cursor() # one connection per request as SSCursor only allows to fetch data asynchronously from one query at a time
cursor.execute(qstr)
with cursor as c:
yield c
finally:
cursor.close()
connection.close()
@asyncthread
def query_async(self, qstr, callback=None):
with self.cur(qstr) as cur:
callback(cur)
def cursorToColumns(cur, sheet):
sheet.columns = []
for i, coldesc in enumerate(cur.description):
name, type, *_ = coldesc
sheet.addColumn(ColumnItem(name, i, type=codeToType(type, name)))
# rowdef: (table_name, ncols)
class MyTablesSheet(Sheet):
rowtype = 'tables'
def reload(self):
qstr = f'''
select
t.table_name,
column_count.ncols,
t.table_rows as est_nrows
from
information_schema.tables t,
(
select
table_name,
count(column_name) as ncols
from
information_schema.columns
where
table_schema = '{self.schema}'
group by
table_name
) as column_count
where
t.table_name = column_count.table_name
AND t.table_schema = '{self.schema}';
'''
with self.sql.cur(qstr) as cur:
self.rows = []
# try to get first row to make cur.description available
r = cur.fetchone()
if r:
self.addRow(r)
cursorToColumns(cur, self)
self.setKeys(self.columns[0:1]) # table_name is the key
for r in cur:
self.addRow(r)
def openRow(self, row):
return MyTable(self.name+"."+row[0], source=row[0], sql=self.sql)
# rowdef: tuple of values as returned by fetchone()
class MyTable(Sheet):
@asyncthread
def reload(self):
with self.sql.cur("SELECT * FROM " + self.source) as cur:
self.rows = []
r = cur.fetchone()
if r:
self.addRow(r)
cursorToColumns(cur, self)
for r in cur:
self.addRow(r)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612075960.0
visidata-2.2.1/visidata/loaders/npy.py 0000660 0001750 0001750 00000005102 00000000000 020422 0 ustar 00kefala kefala 0000000 0000000 from visidata import *
'Loaders for .npy and .npz. Save to .npy. Depends on the zip loader.'
def open_npy(p):
return NpySheet(p.name, source=p)
def open_npz(p):
return NpzSheet(p.name, source=p)
class NpySheet(Sheet):
def iterload(self):
import numpy
if not hasattr(self, 'npy'):
self.npy = numpy.load(str(self.source), encoding='bytes')
self.reloadCols()
yield from Progress(self.npy, total=len(self.npy))
def reloadCols(self):
self.columns = []
for i, (name, fmt, *shape) in enumerate(self.npy.dtype.descr):
if shape:
t = anytype
elif 'M' in fmt:
self.addColumn(Column(name, type=date, getter=lambda c,r,i=i: str(r[i])))
continue
elif 'i' in fmt:
t = int
elif 'f' in fmt:
t = float
else:
t = anytype
self.addColumn(ColumnItem(name, i, type=t))
class NpzSheet(ZipSheet):
# rowdef: tuple(tablename, table)
columns = [
ColumnItem('name', 0),
ColumnItem('length', 1, type=vlen),
]
def iterload(self):
import numpy
self.npz = numpy.load(str(self.source), encoding='bytes')
yield from Progress(self.npz.items())
def openRow(self, row):
import numpy
tablename, tbl = row
if isinstance(tbl, numpy.ndarray):
return NpySheet(tablename, npy=tbl)
return PyobjSheet(tablename, source=tbl)
@VisiData.api
def save_npy(vd, p, sheet):
import numpy as np
dtype = []
for col in Progress(sheet.visibleCols):
if col.type in (int, vlen):
dt = 'i8'
elif col.type in (float, currency, floatlocale):
dt = 'f8'
elif col.type is date:
dt = 'datetime64[s]'
else: # if col.type in (str, anytype):
width = col.getMaxWidth(sheet.rows)
dt = 'U'+str(width)
dtype.append((col.name, dt))
data = []
for typedvals in sheet.iterdispvals(format=False):
nprow = []
for col, val in typedvals.items():
if isinstance(val, TypedWrapper):
if col.type is anytype:
val = ''
else:
val = options.safe_error
elif col.type is date:
val = np.datetime64(val.isoformat())
nprow.append(val)
data.append(tuple(nprow))
arr = np.array(data, dtype=dtype)
with p.open_bytes(mode='w') as outf:
np.save(outf, arr, allow_pickle=False)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612075960.0
visidata-2.2.1/visidata/loaders/pandas_freqtbl.py 0000660 0001750 0001750 00000021021 00000000000 022577 0 ustar 00kefala kefala 0000000 0000000 import math
import collections
from visidata import *
class DataFrameRowSliceAdapter:
"""Tracks original dataframe and a boolean row mask
This is a workaround to (1) save memory (2) keep id(row)
consistent when iterating, as id() is used significantly
by visidata's selectRow implementation.
"""
def __init__(self, df, mask):
import pandas as pd
import numpy as np
if not isinstance(df, pd.DataFrame):
vd.fail('%s is not a dataframe' % type(df).__name__)
if not isinstance(mask, pd.Series):
vd.fail('mask %s is not a Series' % type(mask).__name__)
if df.shape[0] != mask.shape[0]:
vd.fail('dataframe and mask have different shapes (%s vs %s)' % (df.shape[0], mask.shape[0]))
self.df = df
self.mask_bool = mask # boolean mask
self.mask_iloc = np.where(mask.values)[0] # integer indexes corresponding to mask
self.mask_count = mask.sum()
def __len__(self):
return self.mask_count
def __getitem__(self, k):
if isinstance(k, slice):
import pandas as pd
new_mask = pd.Series(False, index=self.df.index)
new_mask.iloc[self.mask_iloc[k]] = True
return DataFrameRowSliceAdapter(self.df, new_mask)
return self.df.iloc[self.mask_iloc[k]]
def __iter__(self):
# With the internal selection API used by PandasSheet,
# this should no longer be needed and can be replaced by
# DataFrameAdapter(self.df[self.mask_iloc])
return DataFrameRowSliceIter(self.df, self.mask_iloc)
def __getattr__(self, k):
# This is trouble ..
return getattr(self.df[self.mask_bool], k)
class DataFrameRowSliceIter:
def __init__(self, df, mask_iloc, index=0):
self.df = df
self.mask_iloc = mask_iloc
self.index = index
def __next__(self):
# Accessing row of original dataframe, to ensure
# that no copies are made and id() of selected rows
# will match original dataframe's rows
if self.index >= self.mask_iloc.shape[0]:
raise StopIteration()
row = self.df.iloc[self.mask_iloc[self.index]]
self.index += 1
return row
class PandasFreqTableSheet(PivotSheet):
'Generate frequency-table sheet on currently selected column.'
rowtype = 'bins' # rowdef FreqRow(keys, sourcerows)
def __init__(self, sheet, *groupByCols):
fqcolname = '%s_%s_freq' % (sheet.name, '-'.join(col.name for col in groupByCols))
super().__init__(fqcolname, groupByCols, [], source=sheet)
self.largest = 1
def selectRow(self, row):
# Select all entries in the bin on the source sheet.
# Use the internally defined _selectByLoc to avoid
# looping which causes a significant performance hit.
self.source._selectByILoc(row.sourcerows.mask_iloc, selected=True)
# then select the bin itself on this sheet
return super().selectRow(row)
def unselectRow(self, row):
self.source._selectByILoc(row.sourcerows.mask_iloc, selected=False)
return super().unselectRow(row)
def updateLargest(self, grouprow):
self.largest = max(self.largest, len(grouprow.sourcerows))
@asyncthread
def reload(self):
'Generate frequency table then reverse-sort by length.'
import pandas as pd
# Note: visidata's base FrequencyTable bins numeric data in ranges
# (e.g. as a histogram). We currently don't provide support for this
# for PandasSheet, although we could implement it with a pd.Grouper
# that operates similarly to pd.cut.
super().initCols()
df = self.source.df.copy()
# Implementation (special case): for one row, this degenerates
# to .value_counts(); however this does not order in a stable manner.
# if len(self.groupByCols) == 1:
# this_column = df.loc[:, str(self.groupByCols[0].name)]
# value_counts = this_column.value_counts()
if len(self.groupByCols) >= 1:
# Implementation (1): add a dummy column to aggregate over in a pd.pivot_table.
# Is there a way to avoid having to mutate the dataframe? We can delete the
# column afterwards but we do incur the overhead of block consolidation.
_pivot_count_column = "__vd_pivot_count"
if _pivot_count_column not in df.columns:
df[_pivot_count_column] = 1
# Aggregate count over columns to group, and then apply a stable sort
value_counts = df.pivot_table(
index=[c.name for c in self.groupByCols],
values=_pivot_count_column,
aggfunc="count"
)[_pivot_count_column].sort_values(ascending=False, kind="mergesort")
# TODO: it seems that the ascending=False causes this to do a "reversed stable sort"?
# TODO: possibly register something to delete this column as soon as
# we exit visidata?
# del df["__vd_pivot_count"]
# Implementation (2) which does not require adding a dummy column:
# Compute cross-tabulation to get counts, and sort/remove zero-entries.
# Note that this is not space-efficient: the initial cross-tabulation will
# have space on the order of product of number of unique elements for each
# column, even though its possible the combinations present are sparse
# and most combinations have zero count.
# this_column = df.loc[:, str(self.groupByCols[0].name)]
# value_counts = pd.crosstab(this_column, [df.df[c.name] for c in self.groupByCols[1:]])
# value_counts = value_counts.stack(list(range(len(self.groupByCols) - 1)))
# value_counts = value_counts.loc[value_counts > 0].sort_values(ascending=False)
else:
vd.fail("Unable to do FrequencyTable, no columns to group on provided")
# add default bonus columns
for c in [
Column('count', type=int,
getter=lambda col,row: len(row.sourcerows)),
Column('percent', type=float,
getter=lambda col,row: len(row.sourcerows)*100/df.shape[0]),
Column('histogram', type=str,
getter=lambda col,row: options.disp_histogram*(options.disp_histolen*len(row.sourcerows)//value_counts.max()),
width=options.disp_histolen+2),
]:
self.addColumn(c)
for element in Progress(value_counts.index):
if len(self.groupByCols) == 1:
element = (element,)
elif len(element) != len(self.groupByCols):
vd.fail('different number of index cols and groupby cols (%s vs %s)' % (len(element), len(self.groupByCols)))
mask = df[self.groupByCols[0].name] == element[0]
for i in range(1, len(self.groupByCols)):
mask = mask & (df[self.groupByCols[i].name] == element[i])
self.addRow(PivotGroupRow(
element,
(0, 0),
DataFrameRowSliceAdapter(df, mask),
{}
))
def expand_source_rows(source, vd, cursorRow):
"""Support for expanding a row of frequency table to underlying rows"""
if cursorRow.sourcerows is None:
vd.error("no source rows")
vs = PandasSheet(source.name, valueNames(cursorRow.discrete_keys, cursorRow.numeric_key), source=cursorRow.sourcerows)
vd.push(vs)
PandasSheet.addCommand('F', 'freq-col', 'vd.push(PandasFreqTableSheet(sheet, cursorCol))', 'open Frequency Table grouped on current column, with aggregations of other columns')
PandasSheet.addCommand('gF', 'freq-keys', 'vd.push(PandasFreqTableSheet(sheet, *keyCols))', 'open Frequency Table grouped by all key columns on source sheet, with aggregations of other columns')
PandasFreqTableSheet.addCommand('t', 'stoggle-row', 'toggle([cursorRow]); cursorDown(1)', 'toggle selection of rows grouped in current row in source sheet')
PandasFreqTableSheet.addCommand('s', 'select-row', 'select([cursorRow]); cursorDown(1)', 'select rows grouped in current row in source sheet')
PandasFreqTableSheet.addCommand('u', 'unselect-row', 'unselect([cursorRow]); cursorDown(1)', 'unselect rows grouped in current row in source sheet')
PandasFreqTableSheet.addCommand(ENTER, 'open-row', 'expand_source_rows(source, vd, cursorRow)', 'open copy of source sheet with rows that are grouped in current row')
PandasFreqTableSheet.class_options.numeric_binning = False
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612684761.0
visidata-2.2.1/visidata/loaders/pcap.py 0000660 0001750 0001750 00000032720 00000000000 020545 0 ustar 00kefala kefala 0000000 0000000 import collections
import ipaddress
from visidata import *
option('pcap_internet', 'n', '(y/s/n) if save_dot includes all internet hosts separately (y), combined (s), or does not include the internet (n)')
protocols = collections.defaultdict(dict) # ['ethernet'] = {[6] -> 'IP'}
_flags = collections.defaultdict(dict) # ['tcp'] = {[4] -> 'FIN'}
url_oui = 'https://visidata.org/plugins/pcap/wireshark-oui.tsv'
url_iana = 'https://visidata.org/plugins/pcap/iana-ports.tsv'
oui = {} # [macprefix (like '01:02:dd:0')] -> 'manufacturer'
services = {} # [('tcp', 25)] -> 'smtp'
def open_pcap(p):
return PcapSheet(p.name, source=p)
open_cap = open_pcap
open_pcapng = open_pcap
open_ntar = open_pcap
def manuf(mac):
return oui.get(mac[:13]) or oui.get(mac[:10]) or oui.get(mac[:8])
def macaddr(addrbytes):
mac = ':'.join('%02x' % b for b in addrbytes)
return mac
def macmanuf(mac):
manuf = oui.get(mac[:13])
if manuf:
return manuf + mac[13:]
manuf = oui.get(mac[:10])
if manuf:
return manuf + mac[10:]
manuf = oui.get(mac[:8])
if manuf:
return manuf + mac[8:]
return mac
def norm_host(host):
if not host:
return None
srcmac = str(host.macaddr)
if srcmac == 'ff:ff:ff:ff:ff:ff': return None
srcip = str(host.ipaddr)
if srcip == '0.0.0.0' or srcip == '::': return None
if srcip == '255.255.255.255': return None
if host.ipaddr:
if host.ipaddr.is_global:
opt = options.pcap_internet
if opt == 'n':
return None
elif opt == 's':
return "internet"
if host.ipaddr.is_multicast:
# include in multicast (minus dns?)
return 'multicast'
names = [host.hostname, host.ipaddr, macmanuf(host.macaddr)]
return '\\n'.join(str(x) for x in names if x)
def FlagGetter(flagfield):
def flags_func(fl):
return ' '.join([flagname for f, flagname in _flags[flagfield].items() if fl & f])
return flags_func
def init_pcap():
if protocols: # already init'ed
return
global dpkt, dnslib
import dpkt
import dnslib
load_consts(protocols['ethernet'], dpkt.ethernet, 'ETH_TYPE_')
load_consts(protocols['ip'], dpkt.ip, 'IP_PROTO_')
load_consts(_flags['ip_tos'], dpkt.ip, 'IP_TOS_')
load_consts(protocols['icmp'], dpkt.icmp, 'ICMP_')
load_consts(_flags['tcp'], dpkt.tcp, 'TH_')
load_oui(url_oui)
load_iana(url_iana)
def read_pcap(f):
try:
return dpkt.pcapng.Reader(f.open_bytes())
except ValueError:
return dpkt.pcap.Reader(f.open_bytes())
@asyncthread
def load_oui(url):
vsoui = TsvSheet('vsoui', source=urlcache(url, days=30))
vsoui.reload.__wrapped__(vsoui)
for r in vsoui.rows:
if r.prefix.endswith('/36'): prefix = r.prefix[:13]
elif r.prefix.endswith('/28'): prefix = r.prefix[:10]
else: prefix = r.prefix[:8]
try:
oui[prefix.lower()] = r.shortname
except Exception as e:
vd.exceptionCaught(e)
@asyncthread
def load_iana(url):
ports_tsv = TsvSheet('ports_tsv', source=urlcache(url, days=30))
ports_tsv.reload.__wrapped__(ports_tsv)
for r in ports_tsv.rows:
try:
services[(r.transport, int(r.port))] = r.service
except Exception as e:
vd.exceptionCaught(e)
class Host:
dns = {} # [ipstr] -> dnsname
hosts = {} # [macaddr] -> { [ipaddr] -> Host }
@classmethod
def get_host(cls, pkt, field='src'):
mac = macaddr(getattr(pkt, field))
machosts = cls.hosts.get(mac, None)
if not machosts:
machosts = cls.hosts[mac] = {}
ipraw = getattrdeep(pkt, 'ip.'+field, None)
if ipraw is not None:
ip = ipaddress.ip_address(ipraw)
if ip not in machosts:
machosts[ip] = Host(mac, ip)
return machosts[ip]
else:
if machosts:
return list(machosts.values())[0]
return Host(mac, None)
@classmethod
def get_by_ip(cls, ip):
'Returns Host instance for the given ip address.'
ret = cls.hosts_by_ip.get(ip)
if ret is None:
ret = cls.hosts_by_ip[ip] = [Host(ip)]
return ret
def __init__(self, mac, ip):
self.ipaddr = ip
self.macaddr = mac
self.mac_manuf = None
def __str__(self):
return str(self.hostname or self.ipaddr or macmanuf(self.macaddr))
def __lt__(self, x):
if isinstance(x, Host):
return str(self.ipaddr) < str(x.ipaddr)
return True
@property
def hostname(self):
return Host.dns.get(str(self.ipaddr))
def load_consts(outdict, module, attrprefix):
for k in dir(module):
if k.startswith(attrprefix):
v = getattr(module, k)
outdict[v] = k[len(attrprefix):]
def getTuple(pkt):
if getattrdeep(pkt, 'ip.tcp', None):
tup = ('tcp', Host.get_host(pkt, 'src'), pkt.ip.tcp.sport, Host.get_host(pkt, 'dst'), pkt.ip.tcp.dport)
elif getattrdeep(pkt, 'ip.udp', None):
tup = ('udp', Host.get_host(pkt, 'src'), pkt.ip.udp.sport, Host.get_host(pkt, 'dst'), pkt.ip.udp.dport)
else:
return None
a,b,c,d,e = tup
if b > d:
return a,d,e,b,c # swap src/sport and dst/dport
else:
return tup
def getService(tup):
if not tup: return
transport, _, sport, _, dport = tup
if (transport, dport) in services:
return services.get((transport, dport))
if (transport, sport) in services:
return services.get((transport, sport))
def get_transport(pkt):
ret = 'ether'
if getattr(pkt, 'arp', None):
return 'arp'
if getattr(pkt, 'ip', None):
ret = 'ip'
if getattr(pkt.ip, 'tcp', None):
ret = 'tcp'
elif getattr(pkt.ip, 'udp', None):
ret = 'udp'
elif getattr(pkt.ip, 'icmp', None):
ret = 'icmp'
if getattr(pkt, 'ip6', None):
ret = 'ipv6'
if getattr(pkt.ip6, 'tcp', None):
ret = 'tcp'
elif getattr(pkt.ip6, 'udp', None):
ret = 'udp'
elif getattr(pkt.ip6, 'icmp6', None):
ret = 'icmpv6'
return ret
def get_port(pkt, field='sport'):
return getattrdeep(pkt, 'ip.tcp.'+field, None) or getattrdeep(pkt, 'ip.udp.'+field, None)
class EtherSheet(Sheet):
'Layer 2 (ethernet) packets'
rowtype = 'packets'
columns = [
ColumnAttr('timestamp', type=date, fmtstr="%H:%M:%S.%f"),
Column('ether_manuf', type=str, getter=lambda col,row: mac_manuf(macaddr(row.src))),
Column('ether_src', type=str, getter=lambda col,row: macaddr(row.src), width=6),
Column('ether_dst', type=str, getter=lambda col,row: macaddr(row.dst), width=6),
ColumnAttr('ether_data', 'data', type=vlen, width=0),
]
class IPSheet(Sheet):
rowtype = 'packets'
columns = [
ColumnAttr('timestamp', type=date, fmtstr="%H:%M:%S.%f"),
ColumnAttr('ip', type=str, width=0),
Column('ip_src', type=str, width=14, getter=lambda col,row: ipaddress.ip_address(row.ip.src)),
Column('ip_dst', type=str, width=14, getter=lambda col,row: ipaddress.ip_address(row.ip.dst)),
ColumnAttr('ip_hdrlen', 'ip.hl', type=int, width=0, helpstr="IPv4 Header Length"),
ColumnAttr('ip_proto', 'ip.p', type=lambda v: protocols['ip'].get(v), width=8, helpstr="IPv4 Protocol"),
ColumnAttr('ip_id', 'ip.id', type=int, width=10, helpstr="IPv4 Identification"),
ColumnAttr('ip_rf', 'ip.rf', type=int, width=10, helpstr="IPv4 Reserved Flag (Evil Bit)"),
ColumnAttr('ip_df', 'ip.df', type=int, width=10, helpstr="IPv4 Don't Fragment flag"),
ColumnAttr('ip_mf', 'ip.mf', type=int, width=10, helpstr="IPv4 More Fragments flag"),
ColumnAttr('ip_tos', 'ip.tos', width=10, type=FlagGetter('ip_tos'), helpstr="IPv4 Type of Service"),
ColumnAttr('ip_ttl', 'ip.ttl', type=int, width=10, helpstr="IPv4 Time To Live"),
ColumnAttr('ip_ver', 'ip.v', type=int, width=10, helpstr="IPv4 Version"),
]
def iterload(self):
for pkt in Progress(self.source.rows):
if getattr(pkt, 'ip', None):
yield pkt
class TCPSheet(IPSheet):
columns = IPSheet.columns + [
ColumnAttr('tcp_srcport', 'ip.tcp.sport', type=int, width=8, helpstr="TCP Source Port"),
ColumnAttr('tcp_dstport', 'ip.tcp.dport', type=int, width=8, helpstr="TCP Dest Port"),
ColumnAttr('tcp_opts', 'ip.tcp.opts', width=0),
ColumnAttr('tcp_flags', 'ip.tcp.flags', type=FlagGetter('tcp'), helpstr="TCP Flags"),
]
def iterload(self):
for pkt in Progress(self.source.rows):
if getattrdeep(pkt, 'ip.tcp', None):
yield pkt
class UDPSheet(IPSheet):
columns = IPSheet.columns + [
ColumnAttr('udp_srcport', 'ip.udp.sport', type=int, width=8, helpstr="UDP Source Port"),
ColumnAttr('udp_dstport', 'ip.udp.dport', type=int, width=8, helpstr="UDP Dest Port"),
ColumnAttr('ip.udp.data', type=vlen, width=0),
ColumnAttr('ip.udp.ulen', type=int, width=0),
]
def iterload(self):
for pkt in Progress(self.source.rows):
if getattrdeep(pkt, 'ip.udp', None):
yield pkt
class PcapSheet(Sheet):
rowtype = 'packets'
columns = [
ColumnAttr('timestamp', type=date, fmtstr="%H:%M:%S.%f"),
Column('transport', type=get_transport, width=5),
Column('srcmanuf', type=str, getter=lambda col,row: manuf(macaddr(row.src))),
Column('srchost', type=str, getter=lambda col,row: row.srchost),
Column('srcport', type=int, getter=lambda col,row: get_port(row, 'sport')),
Column('dstmanuf', type=str, getter=lambda col,row: manuf(macaddr(row.dst))),
Column('dsthost', type=str, getter=lambda col,row: row.dsthost),
Column('dstport', type=int, getter=lambda col,row: get_port(row, 'dport')),
ColumnAttr('ether_proto', 'type', type=lambda v: protocols['ethernet'].get(v), width=0),
ColumnAttr('tcp_flags', 'ip.tcp.flags', type=FlagGetter('tcp'), helpstr="TCP Flags"),
Column('service', type=str, getter=lambda col,row: getService(getTuple(row))),
ColumnAttr('data', type=vlen),
ColumnAttr('ip_len', 'ip.len', type=int),
ColumnAttr('tcp', 'ip.tcp', width=4, type=vlen),
ColumnAttr('udp', 'ip.udp', width=4, type=vlen),
ColumnAttr('icmp', 'ip.icmp', width=4, type=vlen),
ColumnAttr('dns', type=str, width=4),
]
def iterload(self):
init_pcap()
self.pcap = read_pcap(self.source)
self.rows = []
with Progress(total=filesize(self.source)) as prog:
for ts, buf in self.pcap:
eth = dpkt.ethernet.Ethernet(buf)
yield eth
prog.addProgress(len(buf))
eth.timestamp = ts
if not getattr(eth, 'ip', None):
eth.ip = getattr(eth, 'ip6', None)
eth.dns = try_apply(lambda eth: dnslib.DNSRecord.parse(eth.ip.udp.data), eth)
if eth.dns:
for rr in eth.dns.rr:
Host.dns[str(rr.rdata)] = str(rr.rname)
eth.srchost = Host.get_host(eth, 'src')
eth.dsthost = Host.get_host(eth, 'dst')
flowtype = collections.namedtuple('flow', 'packets transport src sport dst dport'.split())
class PcapFlowsSheet(Sheet):
rowtype = 'netflows' # rowdef: flowtype
_rowtype = flowtype
columns = [
ColumnAttr('transport', type=str),
Column('src', type=str, getter=lambda col,row: row.src),
ColumnAttr('sport', type=int),
Column('dst', type=str, getter=lambda col,row: row.dst),
ColumnAttr('dport', type=int),
Column('service', type=str, width=8, getter=lambda col,row: getService(getTuple(row.packets[0]))),
ColumnAttr('packets', type=vlen),
Column('connect_latency_ms', type=float, getter=lambda col,row: col.sheet.latency[getTuple(row.packets[0])]),
]
def iterload(self):
self.flows = {}
self.latency = {} # [flowtuple] -> float ms of latency
self.syntimes = {} # [flowtuple] -> timestamp of SYN
flags = FlagGetter('tcp')
for pkt in Progress(self.source.rows):
tup = getTuple(pkt)
if tup:
flowpkts = self.flows.get(tup)
if flowpkts is None:
flowpkts = self.flows[tup] = []
yield flowtype(flowpkts, *tup)
flowpkts.append(pkt)
if not getattr(pkt.ip, 'tcp', None):
continue
tcpfl = flags(pkt.ip.tcp.flags)
if 'SYN' in tcpfl:
if 'ACK' in tcpfl:
if tup in self.syntimes:
self.latency[tup] = (pkt.timestamp - self.syntimes[tup])*1000
else:
self.syntimes[tup] = pkt.timestamp
def openRow(self, row):
return PcapSheet("%s_packets"%flowname(row), rows=row.packets)
def flowname(flow):
return '%s_%s:%s-%s:%s' % (flow.transport, flow.src, flow.sport, flow.dst, flow.dport)
def try_apply(func, *args, **kwargs):
try:
return func(*args, **kwargs)
except Exception as e:
pass
PcapSheet.addCommand('W', 'flows', 'vd.push(PcapFlowsSheet(sheet.name+"_flows", source=sheet))')
PcapSheet.addCommand('2', 'l2-packet', 'vd.push(IPSheet("L2packets", source=sheet))')
PcapSheet.addCommand('3', 'l3-packet', 'vd.push(TCPSheet("L3packets", source=sheet))')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608541899.0
visidata-2.2.1/visidata/loaders/pdf.py 0000660 0001750 0001750 00000003020 00000000000 020362 0 ustar 00kefala kefala 0000000 0000000 import io
from visidata import *
vd.option('pdf_tables', False, 'parse PDF for tables instead of pages of text', replay=True)
def open_pdf(p):
if vd.options.pdf_tables:
return TabulaSheet(p.name, source=p)
return PdfMinerSheet(p.name, source=p)
class PdfMinerSheet(TableSheet):
rowtype='pages' # rowdef: [pdfminer.LTPage, pageid, text]
columns=[
ColumnItem('pdfpage', 0, width=0),
ColumnItem('pagenum', 1, type=int),
ColumnItem('contents', 2),
]
def iterload(self):
import pdfminer.high_level
from pdfminer.pdfinterp import PDFResourceManager, PDFPageInterpreter
from pdfminer.converter import TextConverter, PDFPageAggregator
from pdfminer.layout import LAParams
from pdfminer.pdfpage import PDFPage
with self.source.open_bytes() as fp:
for page in PDFPage.get_pages(fp):
with io.StringIO() as output_string:
newrsrcmgr = PDFResourceManager()
txtconv = TextConverter(newrsrcmgr, output_string, codec=options.encoding, laparams=LAParams())
interpreter = PDFPageInterpreter(newrsrcmgr, txtconv)
interpreter.process_page(page)
yield [page, page.pageid, output_string.getvalue()]
class TabulaSheet(IndexSheet):
def iterload(self):
import tabula
for i, t in enumerate(tabula.read_pdf(self.source, pages='all', multiple_tables=True)):
yield PandasSheet(self.source.name, i, source=t)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608178530.0
visidata-2.2.1/visidata/loaders/png.py 0000660 0001750 0001750 00000005326 00000000000 020410 0 ustar 00kefala kefala 0000000 0000000 from visidata import *
def open_png(p):
return PNGSheet(p.name, source=p)
@functools.lru_cache(256)
def rgb_to_attr(r,g,b,a):
if a == 0: return 0
if r > g and r > b: return colors['red']
if g > r and g > b: return colors['green']
if b > r and b > g: return colors['blue']
if a == 255: return colors['white']
return 0
class PNGSheet(Sheet):
rowtype = 'pixels' # rowdef: tuple(x, y, r, g, b, a)
columns = [ColumnItem(name, i, type=int) for i, name in enumerate('x y R G B A'.split())] + [
Column('attr', type=int, getter=lambda col,row: rgb_to_attr(*row[2:]))
]
nKeys = 2
def newRow(self):
return list((None, None, 0, 0, 0, 0))
def iterload(self):
import png
r = png.Reader(bytes=self.source.read_bytes())
self.width, self.height, pixels, md = r.asRGBA()
for y, row in enumerate(pixels):
for i in range(0, len(row)-1, 4):
r,g,b,a = row[i:i+4]
yield [i//4, y, r, g, b, a]
class PNGDrawing(Canvas):
aspectRatio = 1.0
rowtype = 'pixels'
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
def togglePixel(self, rows):
for row in rows:
x,y,r,g,b,a = row
self.pixels[y][x][rgb_to_attr(r,g,b,a)].remove(row)
row[5] = a = 0 if row[5] else 255
self.plotpixel(x, y, rgb_to_attr(r,g,b,a), row)
def setPixel(self, rows, attr):
for row in rows:
x,y,r,g,b,a = row
self.pixels[y][x][rgb_to_attr(r,g,b,a)].remove(row)
row[5] = a = attr
self.plotpixel(x, y, rgb_to_attr(r,g,b,a), row)
@asyncthread
def reload(self):
self.reset()
for row in self.sourceRows:
x, y, r, g, b, a = row
self.point(x, y, rgb_to_attr(r,g,b,a), row)
self.refresh()
@VisiData.api
def save_png(vd, p, vs):
if isinstance(vs, Canvas):
return save_png(p, vs.source)
palette = collections.OrderedDict()
palette[(0,0,0,0)] = 0 # invisible black is 0
pixels = list([0]*vs.width for y in range(vs.height))
for x,y,r,g,b,a in Progress(sorted(vs.rows), 'saving'):
color = tuple((r,g,b,a))
colornum = palette.get(color, None)
if colornum is None:
colornum = palette[color] = len(palette)
pixels[y][x] = colornum
vd.status('saving %sx%sx%s' % (vs.width, vs.height, len(palette)))
import png
with open(p, 'wb') as fp:
w = png.Writer(vs.width, vs.height, palette=list(palette.keys()))
w.write(fp, pixels)
vd.status('saved')
PNGSheet.addCommand('.', 'plot-sheet', 'vd.push(PNGDrawing(name+"_plot", source=sheet, sourceRows=rows))', 'plot this png')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1609740788.0
visidata-2.2.1/visidata/loaders/postgres.py 0000660 0001750 0001750 00000007323 00000000000 021471 0 ustar 00kefala kefala 0000000 0000000 from visidata import *
__all__ = ['openurl_postgres', 'openurl_rds', 'PgTable', 'PgTablesSheet']
option('postgres_schema', 'public', 'The desired schema for the Postgres database')
def codeToType(type_code, colname):
import psycopg2
try:
tname = psycopg2._psycopg.string_types[type_code].name
if 'INTEGER' in tname:
return int
if 'STRING' in tname:
return str
except KeyError:
vd.status('unknown postgres type_code %s for %s' % (type_code, colname))
return anytype
def openurl_rds(url, filetype=None):
import boto3
import psycopg2
rds = boto3.client('rds')
url = urlparse(url.given)
_, region, dbname = url.path.split('/')
token = rds.generate_db_auth_token(url.hostname, url.port, url.username, region)
conn = psycopg2.connect(
user=url.username,
dbname=dbname,
host=url.hostname,
port=url.port,
password=token)
return PgTablesSheet(dbname+"_tables", sql=SQL(conn))
def openurl_postgres(url, filetype=None):
import psycopg2
url = urlparse(url.given)
dbname = url.path[1:]
conn = psycopg2.connect(
user=url.username,
dbname=dbname,
host=url.hostname,
port=url.port,
password=url.password)
return PgTablesSheet(dbname+"_tables", sql=SQL(conn))
class SQL:
def __init__(self, conn):
self.conn = conn
def cur(self, qstr):
import string
randomname = ''.join(random.choice(string.ascii_uppercase) for _ in range(6))
cur = self.conn.cursor(randomname)
cur.execute(qstr)
return cur
@asyncthread
def query_async(self, qstr, callback=None):
with self.cur(qstr) as cur:
callback(cur)
cur.close()
def cursorToColumns(cur, sheet):
sheet.columns = []
for i, coldesc in enumerate(cur.description):
sheet.addColumn(ColumnItem(coldesc.name, i, type=codeToType(coldesc.type_code, coldesc.name)))
# rowdef: (table_name, ncols)
class PgTablesSheet(Sheet):
rowtype = 'tables'
def reload(self):
schema = options.postgres_schema
qstr = f'''
SELECT relname table_name, column_count.ncols, reltuples::bigint est_nrows
FROM pg_class, pg_namespace, (
SELECT table_name, COUNT(column_name) AS ncols FROM information_schema.COLUMNS WHERE table_schema = '{schema}' GROUP BY table_name
) AS column_count
WHERE pg_class.relnamespace = pg_namespace.oid AND pg_namespace.nspname = '{schema}' AND column_count.table_name = relname;
'''
with self.sql.cur(qstr) as cur:
self.nrowsPerTable = {}
self.rows = []
# try to get first row to make cur.description available
r = cur.fetchone()
if r:
self.addRow(r)
cursorToColumns(cur, self)
self.setKeys(self.columns[0:1]) # table_name is the key
for r in cur:
self.addRow(r)
def openRow(self, row):
return PgTable(self.name+"."+row[0], source=row[0], sql=self.sql)
# rowdef: tuple of values as returned by fetchone()
class PgTable(Sheet):
@asyncthread
def reload(self):
if self.options.postgres_schema:
source = f"{self.options.postgres_schema}.{self.source}"
else:
source = self.source
with self.sql.cur(f"SELECT * FROM {source}") as cur:
self.rows = []
r = cur.fetchone()
if r:
self.addRow(r)
cursorToColumns(cur, self)
for r in cur:
self.addRow(r)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608541899.0
visidata-2.2.1/visidata/loaders/rec.py 0000660 0001750 0001750 00000010675 00000000000 020400 0 ustar 00kefala kefala 0000000 0000000 from visidata import *
@VisiData.api
def open_rec(vd, p):
return RecIndexSheet(p.name, source=p)
def decode_multiline(line, fp):
'Parse *line* and lookahead into *fp* as iterator for continuing lines. Return (multiline, next_line) where *multiline* can contain newlines and *next_line is the line after the combined *multiline*. Handle "\\" at end and "+" at beginning of lines. *next_line* will be None iff iterator is exhausted.'
while True:
try:
next_line = next(fp)
except StopIteration:
return line, None
if line.endswith('\\'):
line = line[:-1] + next_line
elif next_line.startswith('+'):
# strip leading r'+ ?'
next_line = next_line[2:] if next_line.startswith('+ ') else next_line[1:]
line += '\n' + next_line
else:
return line, next_line
def encode_multiline(s):
return '\n+ '.join(s.splitlines())
def get_kv(line):
return re.split(r':[ \t]?', line, maxsplit=1)
class RecSheet(TableSheet):
def addColumn(self, c, index=None):
super().addColumn(c, index=index)
self.colnames[c.name] = c
RecSheet.init('colnames', dict)
class RecIndexSheet(IndexSheet):
def iterload(self):
sheet = None
row = None
newRecord = True
next_line = ''
comments = []
fp = iter(self.source)
while next_line is not None:
line, next_line = decode_multiline(next_line, fp)
line = line.lstrip()
if not line: # end of record separator
newRecord = True
continue
elif line[0] == '#':
comments.append(line)
continue
if not sheet or (newRecord and line[0] == '%'):
sheet = RecSheet('', columns=[], rows=[], source=self, comments=comments)
comments = []
yield sheet
newRecord = False
if line[0] == '%':
desc, rest = get_kv(line[1:])
if desc == 'rec':
sheet.name = rest
elif desc in 'mandatory allowed':
for colname in rest.split():
if colname not in sheet.colnames:
sheet.addColumn(ItemColumn(colname))
elif desc in ['key', 'unique']:
for i, colname in enumerate(rest.split()):
if colname not in sheet.colnames:
sheet.addColumn(ItemColumn(colname, keycol=i+1))
elif desc in ['sort']:
sheet.orderBy([sheet.column(colname) for colname in rest.split()])
elif desc in ['type', 'typedef']:
pass
elif desc in ['auto']: # autoincrement columns should be present already
pass
elif desc in ['size', 'constraint']: # ignore constraints
pass
elif desc in ['confidential']: # encrypted
pass
else:
vd.warning('Unhandled descriptor: ' +line)
else:
if newRecord:
row = None
newRecord = False
if not row:
row = {}
sheet.addRow(row)
name, rest = get_kv(line)
if name not in sheet.colnames:
sheet.addColumn(ColumnItem(name))
if name in row:
if not isinstance(row[name], list):
row[name] = [row[name]]
row[name].append(rest)
else:
row[name] = rest
for sheet in Progress(self.rows):
sheet.sort()
@VisiData.api
def save_rec(vd, p, *vsheets):
with p.open_text(mode='w') as fp:
for vs in vsheets:
comments = getattr(vs, 'comments', [])
if comments:
fp.write('# ' + '\n# '.join(comments) + '\n')
fp.write('%rec: ' + vs.name + '\n')
fp.write('\n')
for col in vs.visibleCols:
if col.keycol:
fp.write('%key: ' + col.name + '\n')
for row in Progress(vs.rows):
for col in vs.visibleCols:
fp.write(col.name+': '+encode_multiline(col.getDisplayValue(row))+'\n')
fp.write('\n')
fp.write('\n')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1606688371.0
visidata-2.2.1/visidata/loaders/sas.py 0000660 0001750 0001750 00000002011 00000000000 020376 0 ustar 00kefala kefala 0000000 0000000 from visidata import *
import logging
SASTypes = {
'string': str,
'number': float,
}
def open_xpt(p):
return XptSheet(p.name, source=p)
def open_sas7bdat(p):
return SasSheet(p.name, source=p)
class XptSheet(Sheet):
def iterload(self):
import xport
with open(self.source, 'rb') as fp:
self.rdr = xport.Reader(fp)
self.columns = []
for i, var in enumerate(self.rdr._variables):
self.addColumn(ColumnItem(var.name, i, type=float if var.numeric else str))
yield from self.rdr
class SasSheet(Sheet):
def iterload(self):
import sas7bdat
self.dat = sas7bdat.SAS7BDAT(str(self.source), skip_header=True, log_level=logging.CRITICAL)
self.columns = []
for col in self.dat.columns:
self.addColumn(ColumnItem(col.name.decode('utf-8'), col.col_id, type=SASTypes.get(col.type, anytype)))
with self.dat as fp:
yield from Progress(fp, total=self.dat.properties.row_count)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612075960.0
visidata-2.2.1/visidata/loaders/shp.py 0000660 0001750 0001750 00000006477 00000000000 020426 0 ustar 00kefala kefala 0000000 0000000 from visidata import *
# requires pyshp
def open_shp(p):
return ShapeSheet(p.name, source=p)
open_dbf = open_shp
shptypes = {
'C': str,
'N': float,
'L': float,
'F': float,
'D': date,
'M': str,
}
def shptype(ftype, declen):
t = shptypes[ftype[:1]]
if t is float and declen == 0:
return int
return t
# rowdef: shaperec
class ShapeSheet(Sheet):
rowtype = 'shapes'
columns = [
Column('shapeType', width=0, getter=lambda col,row: row.shape.shapeType)
]
def iterload(self):
import shapefile
self.sf = shapefile.Reader(str(self.source))
self.reloadCols()
for shaperec in Progress(self.sf.iterShapeRecords(), total=self.sf.numRecords):
yield shaperec
def reloadCols(self):
self.columns = []
for c in ShapeSheet.columns:
self.addColumn(copy(c))
for i, (fname, ftype, fieldlen, declen) in enumerate(self.sf.fields[1:]): # skip DeletionFlag
self.addColumn(Column(fname, getter=lambda col,row,i=i: row.record[i], type=shptype(ftype, declen)))
class ShapeMap(InvertedCanvas):
aspectRatio = 1.0
filetype = 'geojson'
@asyncthread
def reload(self):
self.reset()
for row in Progress(self.sourceRows):
# color according to key
k = self.source.rowkey(row)
if row.shape.shapeType in (5, 15, 25):
self.polygon(row.shape.points, self.plotColor(k), row)
elif row.shape.shapeType in (3, 13, 23):
self.polyline(row.shape.points, self.plotColor(k), row)
elif row.shape.shapeType in (1, 11, 21):
x, y = row.shape.points[0]
self.point(x, y, self.plotColor(k), row)
else:
vd.status('notimpl shapeType %s' % row.shape.shapeType)
x1, y1, x2, y2 = row.shape.bbox
textx, texty = (x1+x2)/2, (y1+y2)/2
disptext = self.textCol.getDisplayValue(row)
self.label(textx, texty, disptext, self.plotColor(k), row)
self.refresh()
@ShapeMap.api
def save_geojson(vd, p, vs):
features = []
for coords, attr, row in Progress(vs.polylines, 'saving'):
feat = {
'type': 'Feature',
'geometry': {
'type': 'LineString',
'coordinates': [[x, y] for x, y in coords],
},
'properties': {
col.name: col.getTypedValue(row) for col in vs.source.visibleCols
}
}
features.append(feat)
featcoll = {
'type': 'FeatureCollection',
'features': features,
}
with p.open_text(mode='w') as fp:
for chunk in json.JSONEncoder().iterencode(featcoll):
fp.write(chunk)
ShapeSheet.addCommand('.', 'plot-row', 'vd.push(ShapeMap(name+"_map", source=sheet, sourceRows=[cursorRow], textCol=cursorCol))', 'plot geospatial vector in current row')
ShapeSheet.addCommand('g.', 'plot-rows', 'vd.push(ShapeMap(name+"_map", source=sheet, sourceRows=rows, textCol=cursorCol))', 'plot all geospatial vectors in current sheet')
ShapeMap.addCommand('^S', 'save-sheet', 'vd.saveSheets(inputPath("save to: ", value=getDefaultSaveName(sheet)), sheet, confirm_overwrite=options.confirm_overwrite)', 'save current sheet to filename in format determined by extension (default .geojson)')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1606688371.0
visidata-2.2.1/visidata/loaders/spss.py 0000660 0001750 0001750 00000001217 00000000000 020607 0 ustar 00kefala kefala 0000000 0000000 from visidata import *
def open_spss(p):
return SpssSheet(p.name, source=p)
open_sav = open_spss
class SpssSheet(Sheet):
@asyncthread
def reload(self):
import savReaderWriter
self.rdr = savReaderWriter.SavReader(str(self.source))
with self.rdr as reader:
self.columns = []
for i, vname in enumerate(reader.varNames):
vtype = float if reader.varTypes[vname] == 0 else str
self.addColumn(ColumnItem(vname.decode('utf-8'), i, type=vtype))
self.rows = []
for r in Progress(reader, total=reader.shape.nrows):
self.addRow(r)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608541899.0
visidata-2.2.1/visidata/loaders/sqlite.py 0000660 0001750 0001750 00000014222 00000000000 021120 0 ustar 00kefala kefala 0000000 0000000 from visidata import *
def open_sqlite(p):
return SqliteIndexSheet(p.name, source=p)
open_sqlite3 = open_sqlite
open_db = open_sqlite
# rowdef: list of values
class SqliteSheet(Sheet):
'Provide functionality for importing SQLite databases.'
savesToSource = True
defer = True
def resolve(self):
'Resolve all the way back to the original source Path.'
return self.source.resolve()
def conn(self):
import sqlite3
return sqlite3.connect(str(self.resolve()))
def execute(self, conn, sql, parms=None):
parms = parms or []
vd.status(sql)
return conn.execute(sql, parms)
def iterload(self):
sqltypes = {
'INTEGER': int,
'TEXT': anytype,
'BLOB': str,
'REAL': float
}
with self.conn() as conn:
tblname = self.tableName
if not isinstance(self, SqliteIndexSheet):
self.columns = []
self.addColumn(ColumnItem('rowid', 0, type=int, width=0))
for i, r in enumerate(self.execute(conn, 'PRAGMA TABLE_INFO("%s")' % tblname)):
c = ColumnItem(r[1], i+1, type=sqltypes.get(r[2].upper(), anytype))
self.addColumn(c)
if r[-1]:
self.setKeys([c])
r = self.execute(conn, 'SELECT COUNT(*) FROM "%s"' % tblname).fetchall()
rowcount = r[0][0]
for row in Progress(self.execute(conn, 'SELECT rowid, * FROM "%s"' % tblname), total=rowcount-1):
yield list(row)
@asyncthread
def putChanges(self):
adds, mods, dels = self.getDeferredChanges()
options_safe_error = options.safe_error
def value(row, col):
v = col.getTypedValue(row)
if isinstance(v, TypedWrapper):
if isinstance(v, TypedExceptionWrapper):
return options_safe_error
else:
return None
elif not isinstance(v, (int, float, str)):
v = col.getDisplayValue(r)
return v
def values(row, cols):
vals = []
for c in cols:
vals.append(value(row, c))
return vals
with self.conn() as conn:
wherecols = [self.columns[0]] # self.column("rowid")
for r in adds.values():
cols = self.visibleCols
sql = 'INSERT INTO "%s" ' % self.tableName
sql += '(%s)' % ','.join(c.name for c in cols)
sql += ' VALUES (%s)' % ','.join('?' for c in cols)
res = self.execute(conn, sql, parms=values(r, cols))
if res.rowcount != res.arraysize:
vd.warning('not all rows inserted') # f'{res.rowcount}/{res.arraysize} rows inserted'
for row, rowmods in mods.values():
sql = 'UPDATE "%s" SET ' % self.tableName
sql += ', '.join('%s=?' % c.name for c, _ in rowmods.items())
sql += ' WHERE %s' % ' AND '.join('"%s"=?' % c.name for c in wherecols)
newvals=values(row, [c for c, _ in rowmods.items()])
# calcValue gets the 'previous' value (before update)
wherevals=list(Column.calcValue(c, row) or '' for c in wherecols)
res = self.execute(conn, sql, parms=newvals+wherevals)
if res.rowcount != res.arraysize:
vd.warning('not all rows updated') # f'{res.rowcount}/{res.arraysize} rows updated'
for row in dels.values():
sql = 'DELETE FROM "%s" ' % self.tableName
sql += ' WHERE %s' % ' AND '.join('"%s"=?' % c.name for c in wherecols)
wherevals=list(Column.calcValue(c, row) for c in wherecols)
res = self.execute(conn, sql, parms=wherevals)
if res.rowcount != res.arraysize:
vd.warning('not all rows deleted') # f'{res.rowcount}/{res.arraysize} rows deleted'
conn.commit()
self.preloadHook()
self.reload()
class SqliteIndexSheet(SqliteSheet, IndexSheet):
tableName = 'sqlite_master'
def iterload(self):
for row in SqliteSheet.iterload(self):
if row[1] != 'index':
tblname = row[2]
yield SqliteSheet(tblname, source=self, tableName=tblname, row=row)
class SqliteQuerySheet(SqliteSheet):
def iterload(self):
with self.conn() as conn:
self.columns = []
self.addColumn(ColumnItem('rowid', 0, type=int))
self.result = self.execute(conn, self.query, parms=getattr(self, 'parms', []))
for i, desc in enumerate(self.result.description):
self.addColumn(ColumnItem(desc[0], i+1))
for row in self.result:
yield row
@VisiData.api
def save_sqlite(vd, p, *vsheets):
import sqlite3
conn = sqlite3.connect(str(p))
c = conn.cursor()
sqltypes = {
int: 'INTEGER',
float: 'REAL',
currency: 'REAL'
}
for vs in vsheets:
vs.ensureLoaded()
vd.sync()
for vs in vsheets:
tblname = clean_to_id(vs.name)
sqlcols = []
for col in vs.visibleCols:
sqlcols.append('"%s" %s' % (col.name, sqltypes.get(col.type, 'TEXT')))
sql = 'CREATE TABLE IF NOT EXISTS "%s" (%s)' % (tblname, ', '.join(sqlcols))
c.execute(sql)
for r in Progress(vs.rows, 'saving'):
sqlvals = []
for col in vs.visibleCols:
v = col.getTypedValue(r)
if isinstance(v, TypedWrapper):
if isinstance(v, TypedExceptionWrapper):
v = options.safe_error
else:
v = None
elif not isinstance(v, (int, float, str)):
v = col.getDisplayValue(r)
sqlvals.append(v)
sql = 'INSERT INTO "%s" VALUES (%s)' % (tblname, ','.join('?' for v in sqlvals))
c.execute(sql, sqlvals)
conn.commit()
vd.status("%s save finished" % p)
SqliteSheet.class_options.header = 0
VisiData.save_db = VisiData.save_sqlite
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608541899.0
visidata-2.2.1/visidata/loaders/texttables.py 0000660 0001750 0001750 00000001177 00000000000 022003 0 ustar 00kefala kefala 0000000 0000000 import functools
from visidata import vd, Progress
try:
import tabulate
for fmt in tabulate.tabulate_formats:
def save_table(path, *sheets, fmt=fmt):
import tabulate
with path.open_text(mode='w') as fp:
for vs in sheets:
fp.write(tabulate.tabulate(
vs.itervals(*vs.visibleCols, format=True),
headers=[ col.name for col in vs.visibleCols ],
tablefmt=fmt))
if not getattr(vd, 'save_'+fmt, None):
setattr(vd, 'save_'+fmt, save_table)
except ModuleNotFoundError:
pass
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608178530.0
visidata-2.2.1/visidata/loaders/tsv.py 0000660 0001750 0001750 00000006135 00000000000 020437 0 ustar 00kefala kefala 0000000 0000000 import os
import contextlib
import itertools
import collections
from visidata import asyncthread, options, Progress, ColumnItem, SequenceSheet, Sheet, FileExistsError, getType, option, VisiData
from visidata import namedlist, filesize
option('delimiter', '\t', 'field delimiter to use for tsv/usv filetype', replay=True)
option('row_delimiter', '\n', 'row delimiter to use for tsv/usv filetype', replay=True)
option('tsv_safe_newline', '\u001e', 'replacement for newline character when saving to tsv', replay=True)
option('tsv_safe_tab', '\u001f', 'replacement for tab character when saving to tsv', replay=True)
def open_tsv(p):
return TsvSheet(p.name, source=p)
def splitter(fp, delim='\n'):
'Generates one line/row/record at a time from fp, separated by delim'
buf = ''
while True:
nextbuf = fp.read(512)
if not nextbuf:
break
buf += nextbuf
*rows, buf = buf.split(delim)
yield from rows
yield from buf.rstrip(delim).split(delim)
# rowdef: list
class TsvSheet(SequenceSheet):
def iterload(self):
delim = self.options.delimiter
rowdelim = self.options.row_delimiter
with self.source.open_text() as fp:
with Progress(total=filesize(self.source)) as prog:
for line in splitter(fp, rowdelim):
if not line:
continue
prog.addProgress(len(line))
row = list(line.split(delim))
if len(row) < self.nVisibleCols:
# extend rows that are missing entries
row.extend([None]*(self.nVisibleCols-len(row)))
yield row
def load_tsv(fn):
vs = open_tsv(Path(fn))
yield from vs.iterload()
@VisiData.api
def save_tsv(vd, p, vs):
'Write sheet to file `fn` as TSV.'
unitsep = vs.options.delimiter
rowsep = vs.options.row_delimiter
trdict = vs.safe_trdict()
with p.open_text(mode='w') as fp:
colhdr = unitsep.join(col.name.translate(trdict) for col in vs.visibleCols) + options.row_delimiter
fp.write(colhdr)
for dispvals in vs.iterdispvals(format=True):
fp.write(unitsep.join(dispvals.values()))
fp.write(rowsep)
vd.status('%s save finished' % p)
def append_tsv_row(vs, row):
'Append `row` to vs.source, creating file with correct headers if necessary. For internal use only.'
if not vs.source.exists():
with contextlib.suppress(FileExistsError):
parentdir = vs.source.parent
if parentdir:
os.makedirs(parentdir)
# Write tsv header for Sheet `vs` to Path `p`
trdict = vs.safe_trdict()
unitsep = options.delimiter
with vs.source.open_text(mode='w') as fp:
colhdr = unitsep.join(col.name.translate(trdict) for col in vs.visibleCols) + options.row_delimiter
if colhdr.strip(): # is anything but whitespace
fp.write(colhdr)
with vs.source.open_text(mode='a') as fp:
fp.write('\t'.join(col.getDisplayValue(row) for col in vs.visibleCols) + '\n')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608178530.0
visidata-2.2.1/visidata/loaders/ttf.py 0000660 0001750 0001750 00000005353 00000000000 020421 0 ustar 00kefala kefala 0000000 0000000 from visidata import *
def open_ttf(p):
return TTFTablesSheet(p.name, source=p)
open_otf = open_ttf
class TTFTablesSheet(Sheet):
rowtype = 'font tables'
columns = [
ColumnAttr('cmap'),
ColumnAttr('format', type=int),
ColumnAttr('language', type=int),
ColumnAttr('length', type=int),
ColumnAttr('platEncID', type=int),
ColumnAttr('platformID', type=int),
Column('isSymbol', getter=lambda col,row: row.isSymbol()),
Column('isUnicode', getter=lambda col,row: row.isUnicode()),
]
def openRow(self, row):
return TTFGlyphsSheet(self.name+'_glyphs', source=self, sourceRows=[row], ttf=self.ttf)
def iterload(self):
import fontTools.ttLib
self.ttf = fontTools.ttLib.TTFont(str(self.source), 0, allowVID=0, ignoreDecompileErrors=True, fontNumber=-1)
for cmap in self.ttf["cmap"].tables:
yield cmap
class TTFGlyphsSheet(Sheet):
rowtype = 'glyphs' # rowdef: (codepoint, glyphid, fontTools.ttLib.ttFont._TTGlyphGlyf)
columns = [
ColumnItem('codepoint', 0, type=int, fmtstr='%0X'),
ColumnItem('glyphid', 1),
SubColumnItem(2, ColumnAttr('height', type=int)),
SubColumnItem(2, ColumnAttr('width', type=int)),
SubColumnItem(2, ColumnAttr('lsb')),
SubColumnItem(2, ColumnAttr('tsb')),
]
def openRow(self, row):
return makePen(self.name+"_"+row[1], source=row[2], glyphSet=self.ttf.getGlyphSet())
def iterload(self):
glyphs = self.ttf.getGlyphSet()
for cmap in self.sourceRows:
for codepoint, glyphid in Progress(cmap.cmap.items(), total=len(cmap.cmap)):
yield (codepoint, glyphid, glyphs[glyphid])
def makePen(*args, **kwargs):
try:
from fontTools.pens.basePen import BasePen
except ImportError as e:
vd.error('fonttools not installed')
class GlyphPen(InvertedCanvas, BasePen):
aspectRatio = 1.0
def __init__(self, name, **kwargs):
super().__init__(name, **kwargs)
self.lastxy = None
self.attr = self.plotColor(('glyph',))
def _moveTo(self, xy):
self.lastxy = xy
def _lineTo(self, xy):
x1, y1 = self.lastxy
x2, y2 = xy
self.line(x1, y1, x2, y2, self.attr)
self._moveTo(xy)
def _curveToOne(self, xy1, xy2, xy3):
vd.error('NotImplemented')
def _qCurveToOne(self, xy1, xy2):
self.qcurve([self.lastxy, xy1, xy2], self.attr)
self._moveTo(xy2)
def reload(self):
self.reset()
self.source.draw(self)
self.refresh()
return GlyphPen(*args, **kwargs)
#TTFGlyphsSheet.bindkey('.', 'open-row')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1606688371.0
visidata-2.2.1/visidata/loaders/usv.py 0000660 0001750 0001750 00000000640 00000000000 020433 0 ustar 00kefala kefala 0000000 0000000 from copy import copy
from visidata import Sheet, TsvSheet, options, vd, VisiData
def open_usv(p):
vs = TsvSheet(p.name, source=p)
vs.options.delimiter = '\u241e'
vs.options.row_delimiter = '\u241f'
return vs
@VisiData.api
def save_usv(vd, p, vs):
usvs = copy(vs)
usvs.rows = vs.rows
usvs.options.delimiter = '\u241e'
usvs.options.row_delimiter = '\u241f'
vd.save_tsv(p, usvs)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608178530.0
visidata-2.2.1/visidata/loaders/vcf.py 0000660 0001750 0001750 00000002030 00000000000 020367 0 ustar 00kefala kefala 0000000 0000000 from visidata import *
# requires (deb): libbz2-dev libcurl4-openssl-dev liblzma-dev
def open_vcf(p):
return VcfSheet(p.name, source=p)
def unbox(col, row):
v = getitemdef(row, col.expr)
if not v:
return None
if len(v) == 1:
return v[0].value
return v
class VcfSheet(PythonSheet):
rowtype = 'cards'
@asyncthread
def reload(self):
import vobject
self.rows = []
self.columns = []
addedCols = set()
lines = []
for line in self.source.open_text():
lines.append(line)
if line.startswith('END:'):
row = vobject.readOne('\n'.join(lines))
for k, v in row.contents.items():
if v and str(v[0].value).startswith('(None)'):
continue
if not k in addedCols:
addedCols.add(k)
self.addColumn(Column(k, expr=k, getter=unbox))
self.addRow(row.contents)
lines = []
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612075960.0
visidata-2.2.1/visidata/loaders/vds.py 0000660 0001750 0001750 00000004637 00000000000 020424 0 ustar 00kefala kefala 0000000 0000000 'Custom VisiData save format'
import json
from visidata import *
NL='\n'
@VisiData.api
def open_vds(vd, p):
return VdsIndexSheet(p.name, source=p)
@VisiData.api
def save_vds(vd, p, *sheets):
'Save in custom VisiData format, preserving columns and their attributes.'
with p.open_text(mode='w') as fp:
for vs in sheets:
# class and attrs for vs
d = { 'name': vs.name, }
fp.write('#'+json.dumps(d)+NL)
# class and attrs for each column in vs
for col in vs.visibleCols:
d = col.__getstate__()
d['col'] = type(col).__name__
fp.write('#'+json.dumps(d)+NL)
with Progress(gerund='saving'):
for row in vs.iterdispvals(*vs.visibleCols, format=True):
d = {col.name:val for col, val in row.items()}
fp.write(json.dumps(d)+NL)
class VdsIndexSheet(IndexSheet):
def iterload(self):
vs = None
with self.source.open_text() as fp:
line = fp.readline()
while line:
if line.startswith('#{'):
d = json.loads(line[1:])
if 'col' not in d:
vs = VdsSheet(d.pop('name'), columns=[], source=self.source, source_fpos=fp.tell())
yield vs
line = fp.readline()
class VdsSheet(Sheet):
def newRow(self):
return {} # rowdef: dict
def iterload(self):
self.colnames = {}
self.columns = []
with self.source.open_text() as fp:
fp.seek(self.source_fpos)
# consume all metadata, create columns
line = fp.readline()
while line and line.startswith('#{'):
d = json.loads(line[1:])
if 'col' not in d:
raise Exception(d)
classname = d.pop('col')
if classname == 'Column':
classname = 'ItemColumn'
d['expr'] = d['name']
c = globals()[classname](d.pop('name'))
self.colnames[c.name] = c
self.addColumn(c)
for k, v in d.items():
setattr(c, k, v)
line = fp.readline()
while line and not line.startswith('#{'):
d = json.loads(line)
yield d
line = fp.readline()
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612075960.0
visidata-2.2.1/visidata/loaders/xlsb.py 0000660 0001750 0001750 00000000534 00000000000 020570 0 ustar 00kefala kefala 0000000 0000000 from visidata import vd, IndexSheet
'Requires visidata/deps/pyxlsb fork'
def open_xlsb(p):
return XlsbIndex(p.name, source=p)
class XlsbIndex(IndexSheet):
def iterload(self):
from pyxlsb import open_workbook
wb = open_workbook(str(self.source))
for name in wb.sheets:
yield wb.get_sheet(name, True)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612075960.0
visidata-2.2.1/visidata/loaders/xlsx.py 0000660 0001750 0001750 00000006615 00000000000 020624 0 ustar 00kefala kefala 0000000 0000000 from visidata import *
def open_xls(p):
return XlsIndexSheet(p.name, source=p)
def open_xlsx(p):
return XlsxIndexSheet(p.name, source=p)
class XlsxIndexSheet(IndexSheet):
'Load XLSX file (in Excel Open XML format).'
rowtype = 'sheets' # rowdef: xlsxSheet
columns = [
Column('sheet', getter=lambda col,row: row.source.title), # xlsx sheet title
ColumnAttr('name', width=0), # visidata Sheet name
ColumnAttr('nRows', type=int),
ColumnAttr('nCols', type=int),
Column('active', getter=lambda col,row: row.source is col.sheet.workbook.active),
]
nKeys = 1
def iterload(self):
import openpyxl
self.workbook = openpyxl.load_workbook(str(self.source), data_only=True, read_only=True)
for sheetname in self.workbook.sheetnames:
src = self.workbook[sheetname]
yield XlsxSheet(self.name, sheetname, source=src)
class XlsxSheet(SequenceSheet):
def iterload(self):
worksheet = self.source
for row in Progress(worksheet.iter_rows(), total=worksheet.max_row or 0):
yield list(wrapply(getattr, cell, 'value') for cell in row)
class XlsIndexSheet(IndexSheet):
'Load XLS file (in Excel format).'
rowtype = 'sheets' # rowdef: xlsSheet
columns = [
Column('sheet', getter=lambda col,row: row.source.name), # xls sheet name
ColumnAttr('name', width=0), # visidata sheet name
ColumnAttr('nRows', type=int),
ColumnAttr('nCols', type=int),
]
nKeys = 1
def iterload(self):
import xlrd
self.workbook = xlrd.open_workbook(str(self.source))
for sheetname in self.workbook.sheet_names():
yield XlsSheet(self.name, sheetname, source=self.workbook.sheet_by_name(sheetname))
class XlsSheet(SequenceSheet):
def iterload(self):
worksheet = self.source
for rownum in Progress(range(worksheet.nrows)):
yield list(worksheet.cell(rownum, colnum).value for colnum in range(worksheet.ncols))
def xls_name(name):
# sheet name can not be longer than 31 characters
xname = clean_name(name)[:31]
if xname != name:
vd.warning(f'{name} saved as {xname}')
return xname
@VisiData.api
def save_xlsx(vd, p, *sheets):
import openpyxl
wb = openpyxl.Workbook()
wb.remove_sheet(wb['Sheet'])
for vs in sheets:
ws = wb.create_sheet(title=xls_name(vs.name))
headers = [col.name for col in vs.visibleCols]
ws.append(headers)
for dispvals in vs.iterdispvals(format=False):
row = []
for col, v in dispvals.items():
if col.type == date:
v = datetime.datetime.fromtimestamp(int(v.timestamp()))
elif not vd.isNumeric(col):
v = str(v)
row.append(v)
ws.append(row)
wb.active = ws
wb.save(filename=p)
vd.status(f'{p} save finished')
@VisiData.api
def save_xls(vd, p, *sheets):
import xlwt
wb = xlwt.Workbook()
for vs in sheets:
ws1 = wb.add_sheet(xls_name(vs.name))
for col_i, col in enumerate(vs.visibleCols):
ws1.write(0, col_i, col.name)
for r_i, dispvals in enumerate(vs.iterdispvals(format=True)):
r_i += 1
for c_i, v in enumerate(dispvals.values()):
ws1.write(r_i, c_i, v)
wb.save(p)
vd.status(f'{p} save finished')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1608178530.0
visidata-2.2.1/visidata/loaders/xml.py 0000660 0001750 0001750 00000005435 00000000000 020425 0 ustar 00kefala kefala 0000000 0000000 from visidata import *
def open_xml(p):
return XmlSheet(p.name, source=p)
open_svg = open_xml
def unns(k):
'de-namespace key k'
if '}' in k:
return k[k.find('}')+1:]
return k
def AttribColumn(name, k, **kwargs):
return Column(name, getter=lambda c,r,k=k: r.attrib.get(k),
setter=lambda c,r,v,k=k: setitem(r.attrib, k, v), **kwargs)
# source is Path or xml.Element; root is xml.Element
class XmlSheet(Sheet):
rowtype = 'elements' # rowdef: lxml.xml.Element
columns = [
ColumnAttr('sourceline', type=int, width=0),
ColumnAttr('prefix', width=0),
ColumnAttr('nstag', 'tag', width=0),
Column('path', width=0, getter=lambda c,r: c.sheet.root.getpath(r)),
Column('tag', getter=lambda c,r: unns(r.tag)),
Column('children', type=vlen, getter=lambda c,r: r.getchildren()),
ColumnAttr('text'),
ColumnAttr('tail', width=0),
]
colorizers = [
RowColorizer(8, None, lambda s,c,r,v: 'green' if r is s.source else None)
]
def showColumnsBasedOnRow(self, row):
for c in self.columns:
nstag = getattr(c, 'nstag', '')
if nstag:
c.hide(nstag not in row.attrib)
def iterload(self):
if isinstance(self.source, Path):
from lxml import etree, objectify
self.root = etree.parse(self.source.open_text())
objectify.deannotate(self.root, cleanup_namespaces=True)
else: # elif isinstance(self.source, XmlElement):
self.root = self.source
self.attribcols = {}
self.columns = []
for c in XmlSheet.columns:
self.addColumn(copy(c))
if getattr(self.root, 'iterancestors', None):
for elem in Progress(list(self.root.iterancestors())[::-1]):
yield elem
for elem in self.root.iter():
yield elem
def openRow(self, row):
return XmlSheet("%s_%s" % (unns(row.tag), row.attrib.get("id")), source=row)
def addRow(self, elem):
super().addRow(elem)
for k in elem.attrib:
if k not in self.attribcols:
c = AttribColumn(unns(k), k)
self.addColumn(c)
self.attribcols[k] = c
c.nstag = k
@VisiData.api
def save_xml(vd, p, vs):
isinstance(XmlSheet) or vd.fail('must save xml from XmlSheet')
vs.root.write(str(p), encoding=options.encoding, standalone=False, pretty_print=True)
VisiData.save_svg = VisiData.save_xml
XmlSheet.addCommand('za', 'addcol-xmlattr', 'attr=input("add attribute: "); addColumnAtCursor(AttribColumn(attr, attr))', 'add column for xml attribute')
XmlSheet.addCommand('v', 'visibility', 'showColumnsBasedOnRow(cursorRow)', 'show only columns in current row attributes')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1606688371.0
visidata-2.2.1/visidata/loaders/xword.py 0000770 0001750 0001750 00000006431 00000000000 020767 0 ustar 00kefala kefala 0000000 0000000 from collections import defaultdict
import string
from visidata import *
theme('color_xword_active', 'green', 'color of active clue')
def open_puz(p):
return PuzSheet(p.name, source=p)
def open_xd(p):
if p.is_dir():
return CrosswordsSheet(p.name, source=p)
return CrosswordSheet(p.name, source=p)
class CrosswordsSheet(Sheet):
rowtype = 'puzzles'
columns = [
Column('Author', getter=lambda col, row: row.author),
Column('Copyright', getter=lambda col, row: row.copyright),
Column('Notes', getter=lambda col, row: row.notes),
Column('Postscript', getter=lambda col, row: ''.join(x for x in row.postscript if ord(x) >= ord(' '))),
Column('Preamble', getter=lambda col, row: row.preamble),
Column('Title', getter=lambda col, row: row.title)
]
@asyncthread
def reload(self):
self.rows = []
for p in self.source.iterdir():
self.addRow(Crossword(p.read(), str(p)))
class GridSheet(Sheet):
rowtype = 'gridrow' # rowdef: puzzle_row:str
colorizers = [
CellColorizer(7, 'color_xword_active', lambda s,c,r,v: r and s.pos in s.cells[(s.rows.index(r),c)])
]
@asyncthread
def reload(self):
grid = self.source.xd.grid
ncols = len(grid[0])
self.columns = [ColumnItem('', i, width=2) for i in range(ncols)]
for row in grid:
row = list(row)
self.addRow(row)
self.cells = defaultdict(list) # [rownum, col] -> [ Apos, Dpos ] or [] (if black)
# find starting r,c from self.pos
for cluedir, cluenum, answer, r, c in self.source.xd.iteranswers_full():
# across
if cluedir == 'A':
for i in range(0, len(answer)):
self.cells[(r, self.columns[c+i])].append(('A', cluenum))
if cluedir == 'D':
for i in range(0, len(answer)):
self.cells[(r+i, self.columns[c])].append(('D', cluenum))
if cluenum == self.pos[1]:
self.cursorRowIndex, self.cursorVisibleColIndex = r, c
class CrosswordSheet(Sheet):
rowtype = 'clues' # rowdef: (cluenum, clue, answer)
columns = [
Column('clue_number', getter=lambda col, row: row[0][0]+str(row[0][1])),
Column('clue', getter=lambda col, row: row[1]),
Column('answer', getter=lambda col, row: row[2])
]
def reload(self):
import xdfile
self.xd = xdfile.xdfile(xd_contents=self.source.read_text(), filename=self.source)
self.rows = self.xd.clues
class PuzSheet(CrosswordSheet):
@asyncthread
def reload(self):
import xdfile.puz2xd
self.xd = xdfile.puz2xd.parse_puz(self.source.read_bytes(), str(self.source))
self.rows = self.xd.clues
@VisiData.api
def save_xd(vd, p, vs):
with p.open_text(mode='w') as fp:
fp.write(vs.xd.to_unicode())
CrosswordsSheet.addCommand(ENTER, 'open-clues', 'vd.push(CrosswordSheet("clues_"+cursorRow.title, source=cursorRow))', 'open CrosswordSheet: clue answer pair for crossword')
CrosswordSheet.addCommand(ENTER, 'open-grid', 'vd.push(GridSheet("grid", source=sheet, pos=cursorRow[0]))', 'open GridSheet: grid for crossword')
GridSheet.class_options.disp_column_sep = ''
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1606688371.0
visidata-2.2.1/visidata/loaders/yaml.py 0000660 0001750 0001750 00000002453 00000000000 020564 0 ustar 00kefala kefala 0000000 0000000 from visidata import *
from itertools import chain
def open_yml(p):
return YamlSheet(p.name, source=p)
open_yaml = open_yml
class YamlSheet(JsonSheet):
def iterload(self):
import yaml
with self.source.open_text() as fp:
documents = yaml.safe_load_all(fp)
self.columns = []
self.colnames = {}
# Peek at the document stream to determine how to best DWIM.
#
# This code is a bit verbose because it avoids slurping the generator
# all at once into memory.
try:
first = next(documents)
except StopIteration:
# Empty file‽
yield None
return
try:
second = next(documents)
except StopIteration:
if isinstance(first, list):
# A file with a single YAML list: yield one row per list item.
yield from Progress(first)
else:
# A file with a single YAML non-list value, e.g a dict.
yield first
else:
# A file containing multiple YAML documents: yield one row per document.
yield from Progress(chain([first, second], documents), total=0)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1609740788.0
visidata-2.2.1/visidata/macros.py 0000660 0001750 0001750 00000004274 00000000000 017460 0 ustar 00kefala kefala 0000000 0000000 from visidata import *
from functools import wraps
vd.macroMode = None
vd.macrobindings = {}
@VisiData.lazy_property
def macrosheet(vd):
macrospath = Path(os.path.join(options.visidata_dir, 'macros.tsv'))
macrosheet = vd.loadInternalSheet(TsvSheet, macrospath, columns=(ColumnItem('command', 0), ColumnItem('filename', 1))) or vd.error('error loading macros')
real_macrosheet = IndexSheet('user_macros', rows=[], source=macrosheet)
for ks, fn in macrosheet.rows:
vs = vd.loadInternalSheet(CommandLog, Path(fn))
vd.status(f"setting {ks}")
setMacro(ks, vs)
real_macrosheet.addRow(vs)
return real_macrosheet
@VisiData.api
def runMacro(vd, macro):
vd.replay_sync(macro, live=True)
def setMacro(ks, vs):
vd.macrobindings[ks] = vs
if vd.isLongname(ks):
BaseSheet.addCommand('', ks, 'runMacro(vd.macrobindings[longname])')
else:
BaseSheet.addCommand(ks, vs.name, 'runMacro(vd.macrobindings[keystrokes])')
@CommandLog.api
def saveMacro(self, rows, ks):
vs = copy(self)
vs.rows = rows
macropath = Path(fnSuffix(options.visidata_dir+"macro"))
vd.save_vd(macropath, vs)
setMacro(ks, vs)
append_tsv_row(vd.macrosheet.source, (ks, macropath))
@CommandLog.api
@wraps(CommandLog.afterExecSheet)
def afterExecSheet(cmdlog, sheet, escaped, err):
if vd.macroMode and (vd.activeCommand is not None) and (vd.activeCommand is not UNLOADED):
cmd = copy(vd.activeCommand)
cmd.row = cmd.col = cmd.sheet = ''
vd.macroMode.addRow(cmd)
# the following needs to happen at the end, bc
# once cmdlog.afterExecSheet.__wrapped__ runs, vd.activeCommand resets to None
cmdlog.afterExecSheet.__wrapped__(cmdlog, sheet, escaped, err)
@CommandLog.api
def startMacro(cmdlog):
if vd.macroMode:
ks = vd.input('save macro for keystroke: ')
vd.cmdlog.saveMacro(vd.macroMode.rows, ks)
vd.macroMode = None
else:
vd.status("recording macro")
vd.macroMode = CommandLog('current_macro', rows=[])
vd.status(vd.macrosheet)
Sheet.addCommand('m', 'macro-record', 'vd.cmdlog.startMacro()')
Sheet.addCommand('gm', 'macro-sheet', 'vd.push(vd.macrosheet)')
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 011453 x ustar 00 0000000 0000000 22 mtime=1612763790.0
visidata-2.2.1/visidata/main.py 0000770 0001750 0001750 00000024014 00000000000 017114 0 ustar 00kefala kefala 0000000 0000000 #
# Usage: $0 [] [ ...]
# $0 [] --play [--batch] [-w ] [-o