pax_global_header00006660000000000000000000000064135323137530014517gustar00rootroot0000000000000052 comment=6319a904117b79b522aedca0118fb1fbafb2435e urlscan-0.9.4/000077500000000000000000000000001353231375300132005ustar00rootroot00000000000000urlscan-0.9.4/.gitignore000066400000000000000000000001021353231375300151610ustar00rootroot00000000000000*.pyc *.egg *.egg-info *.gz *.xz build dist test_emails/ MANIFEST urlscan-0.9.4/COPYING000066400000000000000000000431031353231375300142340ustar00rootroot00000000000000 GNU GENERAL PUBLIC LICENSE Version 2, June 1991 Copyright (C) 1989, 1991 Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. This General Public License applies to most of the Free Software Foundation's software and to any other program whose authors commit to using it. (Some other Free Software Foundation software is covered by the GNU Lesser General Public License instead.) You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things. To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies of the software, or if you modify it. For example, if you distribute copies of such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distribute and/or modify the software. Also, for each author's protection and ours, we want to make certain that everyone understands that there is no warranty for this free software. If the software is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introduced by others will not reflect on the original authors' reputations. Finally, any free program is threatened constantly by software patents. We wish to avoid the danger that redistributors of a free program will individually obtain patent licenses, in effect making the program proprietary. To prevent this, we have made it clear that any patent must be licensed for everyone's free use or not licensed at all. The precise terms and conditions for copying, distribution and modification follow. GNU GENERAL PUBLIC LICENSE TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 0. This License applies to any program or other work which contains a notice placed by the copyright holder saying it may be distributed under the terms of this General Public License. The "Program", below, refers to any such program or work, and a "work based on the Program" means either the Program or any derivative work under copyright law: that is to say, a work containing the Program or a portion of it, either verbatim or with modifications and/or translated into another language. (Hereinafter, translation is included without limitation in the term "modification".) Each licensee is addressed as "you". Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted, and the output from the Program is covered only if its contents constitute a work based on the Program (independent of having been made by running the Program). Whether that is true depends on what the Program does. 1. You may copy and distribute verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and give any other recipients of the Program a copy of this License along with the Program. You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. 2. You may modify your copy or copies of the Program or any portion of it, thus forming a work based on the Program, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions: a) You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. b) You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License. c) If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in the most ordinary way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, saying that you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this License. (Exception: if the Program itself is interactive but does not normally print such an announcement, your work based on the Program is not required to print an announcement.) These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it. Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Program. In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License. 3. You may copy and distribute the Program (or a work based on it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you also do one of the following: a) Accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, b) Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, c) Accompany it with the information you received as to the offer to distribute corresponding source code. (This alternative is allowed only for noncommercial distribution and only if you received the program in object code or executable form with such an offer, in accord with Subsection b above.) The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable. However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable. If distribution of executable or object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place counts as distribution of the source code, even though third parties are not compelled to copy the source along with the object code. 4. You may not copy, modify, sublicense, or distribute the Program except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. 5. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Program or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Program (or any work based on the Program), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Program or works based on it. 6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties to this License. 7. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Program at all. For example, if a patent license would not permit royalty-free redistribution of the Program by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Program. If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply and the section as a whole is intended to apply in other circumstances. It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system, which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice. This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License. 8. If the distribution and/or use of the Program is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Program under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License. 9. The Free Software Foundation may publish revised and/or new versions of the General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies a version number of this License which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of this License, you may choose any version ever published by the Free Software Foundation. 10. If you wish to incorporate parts of the Program into other free programs whose distribution conditions are different, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally. NO WARRANTY 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. Also add information on how to contact you by electronic and paper mail. If the program is interactive, make it output a short notice like this when it starts in an interactive mode: Gnomovision version 69, Copyright (C) year name of author Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, the commands you use may be called something other than `show w' and `show c'; they could even be mouse-clicks or menu items--whatever suits your program. You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the program, if necessary. Here is a sample; alter the names: Yoyodyne, Inc., hereby disclaims all copyright interest in the program `Gnomovision' (which makes passes at compilers) written by James Hacker. , 1 April 1989 Ty Coon, President of Vice This General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. urlscan-0.9.4/MANIFEST.in000066400000000000000000000001221353231375300147310ustar00rootroot00000000000000include COPYING include urlscan.1 include urlscan/assets/tlds-alpha-by-domain.txt urlscan-0.9.4/README.rst000066400000000000000000000147231353231375300146760ustar00rootroot00000000000000Urlscan ======= Contributors ------------ Daniel Burrows (Original Author) Scott Hansen (Maintainer) Maxime Chatelle (Debian Maintainer) Purpose and Requirements ------------------------ Urlscan is a small program that is designed to integrate with the "mutt" mailreader to allow you to easily launch a Web browser for URLs contained in email messages. It is a replacement for the "urlview" program. *NOTE* The last version that is Python 2 compatible is 0.9.3. Requires: Python 3.3+ and the python-urwid library Features -------- Urlscan parses an email message or file and scans it for URLs and email addresses. It then displays the URLs and their context within the message, and allows you to choose one or more URLs to send to your Web browser. Alternatively, it send a list of all URLs to stdout. Relative to urlview, urlscan has the following additional features: - Support for emails in quoted-printable and base64 encodings. No more stripping out =40D from URLs by hand! - The context of each URL is provided along with the URL. For HTML mails, a crude parser is used to render the HTML into text. Context view can be toggled on/off with `c`. - URLs are shortened by default to fit on one line. Viewing full URL (for one or all) is toggled with `s` or `S`. - Jump to a URL by typing the number. - Incremental case-insensitive search with `/`. - Execute an arbitrary function (for example, copy URL to clipboard) instead of opening URL in a browser. - Use `l` to cycle through whether URLs are opened using the Python webbrowser module (default), xdg-open (if installed) or opened by a function passed on the command line with `--run`. - Configure colors and keybindings via ~/.config/urlscan/config.json. Generate default config file for editing by running `urlscan -g`. Cycle through available palettes with `p`. - Copy URL to clipboard with `C` or to primary selection with `P`. Requires xsel or xclip. - Run a command with the selected URL as the argument or pipe the selected URL to a command. - Show complete help menu with `F1`. Hide header on startup with `--nohelp`. Installation and setup ---------------------- To install urlscan, install from your distribution repositories (Archlinux), from Pypi, or do a local development install with pip -e:: pip install --user urlscan OR cd && pip install --user -e . .. NOTE:: The minimum required version of urwid is 1.2.1. Once urlscan is installed, add the following lines to your .muttrc: macro index,pager \\cb " urlscan" "call urlscan to extract URLs out of a message" macro attach,compose \\cb " urlscan" "call urlscan to extract URLs out of a message" Once this is done, Control-b while reading mail in mutt will automatically invoke urlscan on the message. To choose a particular browser, set the environment variable BROWSER. If BROWSER is not set, xdg-open will control which browser is used, if it's available.: export BROWSER=/usr/bin/epiphany Command Line usage ------------------ :: urlscan [-g, --genconf] [-n, --no-browser] [-c, --compact] [-d, --dedupe] [-r, --run ] [-p, --pipe] [-H, --nohelp] Urlscan can extract URLs and email addresses from emails or any text file. Calling with no flags will start the curses browser. Calling with '-n' will just output a list of URLs/email addressess to stdout. The '-c' flag removes the context from around the URLs in the curses browser, and the '-d' flag removes duplicate URLs. Files can also be piped to urlscan using normal shell pipe mechanisms: `cat | urlscan` or `urlscan < ` Instead of opening a web browser, the selected URL can be passed as the argument to a command using `--run " {}"`. Note the use of `{}` in the command string to denote the selected URL. Alternatively, the URL can be piped to the command using `--run --pipe`. Using --run with --pipe is preferred if the command supports it, as it is marginally more secure and tolerant of special characters in the URL. Theming ------- Run `urlscan -g` to generate ~/.config/urlscan/config.json with the default color and black & white palettes. This can be edited or added to, as desired. The first palette in the list will be the default. Configure the palettes according to the `Urwid display attributes`_. Keybindings ----------- Run `urlscan -g` to generate ~/.config/urlscan/config.json. All of the keys will be listed. You can either leave in place or delete any that will not be altered. To unset a binding, set it equal to "". For example: `"P": ""` The follow actions are supported: - `all_escape` -- toggle unescape all URLs (default: `u`) - `all_shorten` -- toggle shorten all URLs (default: `S`) - `bottom` -- move cursor to last item (default: `G`) - `clear_screen` -- redraw screen (default: `Ctrl-l`) - `clipboard` -- copy highlighted URL to clipboard using xsel/xclip (default: `C`) - `clipboard_pri` -- copy highlighted URL to primary selection using xsel/xclip (default: `P`) - `context` -- show/hide context (default: `c`) - `down` -- cursor down (default: `j`) - `help_menu` -- show/hide help menu (default: `F1`) - `link_handler` -- cycle link handling (webbrowser, xdg-open or --run) (default: `l`) - `open_url` -- open selected URL (default: `space` or `enter`) - `palette` -- cycle through palettes (default: `p`) - `quit` -- quit (default: `q` or `Q`) - `shorten` -- toggle shorten highlighted URL (default: `s`) - `top` -- move to first list item (default: `g`) - `up` -- cursor up (default: `k`) Update TLD list (for developers, not users) ------------------------------------------- `wget https://data.iana.org/TLD/tlds-alpha-by-domain.txt` Known bugs and limitations -------------------------- - Running urlscan sometimes "messes up" the terminal background. This seems to be an urwid bug, but I haven't tracked down just what's going on. - Extraction of context from HTML messages leaves something to be desired. Probably the ideal solution would be to extract context on a word basis rather than on a paragraph basis. - The HTML message handling is a bit kludgy in general. - multipart/alternative sections are handled by descending into all the sub-parts, rather than just picking one, which may lead to URLs and context appearing twice. (Bypass this by selecting the '--dedupe' option) .. _Urwid display attributes: http://urwid.org/manual/displayattributes.html#display-attributes urlscan-0.9.4/bin/000077500000000000000000000000001353231375300137505ustar00rootroot00000000000000urlscan-0.9.4/bin/urlscan000077500000000000000000000160341353231375300153510ustar00rootroot00000000000000#!/usr/bin/env python """ A simple urlview replacement that handles things like quoted-printable properly. aka "urlview minus teh suck" """ # # Copyright (C) 2006-2007 Daniel Burrows # Copyright (C) 2019 Scott Hansen # # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 2 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA from __future__ import unicode_literals import argparse import io import locale import os import sys from urlscan import urlchoose, urlscan try: from email.Parser import Parser as parser except ImportError: from email.parser import Parser as parser def parse_arguments(): """Parse command line options. Returns: args """ arg_parse = argparse.ArgumentParser(description="Parse and display URLs") arg_parse.add_argument('--genconf', '-g', action='store_true', default=False, help="Generate config file and exit.") arg_parse.add_argument('--compact', '-c', action='store_true', default=False, help="Don't display the context of each URL.") arg_parse.add_argument('--no-browser', '-n', dest="nobrowser", action='store_true', default=False, help="Pipe URLs to stdout") arg_parse.add_argument('--dedupe', '-d', dest="dedupe", action='store_true', default=False, help="Remove duplicate URLs from list") arg_parse.add_argument('--run', '-r', help="Alternate command to run on selected URL " "instead of opening URL in browser. Use {} to " "represent the URL value in the expression. " "For example: --run 'echo {} | xclip -i'") arg_parse.add_argument('--pipe', '-p', dest='pipe', action='store_true', default=False, help='Pipe URL into the command specified by --run') arg_parse.add_argument('--nohelp', '-H', dest='nohelp', action='store_true', default=False, help='Hide help menu by default') arg_parse.add_argument('message', nargs='?', default=sys.stdin, help="Filename of the message to parse") return arg_parse.parse_args() def close_stdin(): """This section closes out sys.stdin if necessary so as not to block curses keyboard inputs """ if not os.isatty(0): fdesc = os.open('/dev/tty', os.O_RDONLY) if fdesc < 0: sys.stderr.write('Unable to open an input tty.\n') sys.exit(-1) else: os.dup2(fdesc, 0) os.close(fdesc) def process_input(fname): """Return the parsed text of stdin or the message. Accounts for possible file encoding differences. Args: fname - filename or sys.stdin Returns: mesg - parsed (email parser) text of the message with the correct encoding set """ enc_list = ['UTF-8', 'LATIN-1', 'iso8859-1', 'iso8859-2', 'UTF-16', 'CP720', 'CP437'] locale.setlocale(locale.LC_ALL, '') code = locale.getpreferredencoding() if code not in enc_list: enc_list.insert(0, code) if fname is sys.stdin: try: stdin_file = fname.buffer.read() except AttributeError: stdin_file = fname.read() else: stdin_file = None for enc in enc_list: try: if stdin_file is not None: fobj = io.StringIO(stdin_file.decode(enc)) else: fobj = io.open(fname, mode='r', encoding=(enc)) f_keep = fobj mesg = parser().parse(fobj) if 'From' not in mesg.keys() and 'Date' not in mesg.keys(): # If it's not an email message, don't let the email parser # delete the first line. If it is, let the parser do its job so # we don't get mailto: links for all the To and From addresses fobj = _fix_first_line(f_keep) mesg = parser().parse(fobj) except (UnicodeDecodeError, UnicodeError): continue else: break finally: try: fobj.close() except NameError: pass raise Exception("Encoding not detected. Please pass encoding value manually") close_stdin() # Handle multiple nested message parts _msg_set_charset(mesg, enc) return mesg def _fix_first_line(fline): """If the first line starts with http* or [ or other non-text characters, the URLs on that line will not be parsed by email.Parser. Add a blank line at the top of the file to ensure everything is read in a non-email file. 1. Take the file object 'f'. 2. Create a new StringIO object that starts with a blank line and read the file into that. Return as open StringIO object 'f' 3. Return 'f' """ fline.seek(0) new = io.StringIO() new.write("\n{}".format(fline.read())) fline.close() new.seek(0) return new def _msg_set_charset(mesg, encoding): """Recursive function to set the charset of nested message parts. """ encoding = mesg.get_content_charset() or encoding try: mesg.set_charset(encoding) except (AttributeError, TypeError): for part in mesg.get_payload(): try: # Try once to set correct encoding on the message part, then # continue without crashing if it fails _msg_set_charset(part, encoding) except UnicodeEncodeError: continue def main(): """Entrypoint function for urlscan """ args = parse_arguments() if args.genconf is True: urlchoose.URLChooser([], genconf=True) return msg = process_input(args.message) if args.nobrowser is False: tui = urlchoose.URLChooser(urlscan.msgurls(msg), compact=args.compact, nohelp=args.nohelp, dedupe=args.dedupe, run=args.run, pipe=args.pipe) tui.main() else: out = urlchoose.URLChooser(urlscan.msgurls(msg), dedupe=args.dedupe, shorten=False) print("\n".join(out.urls)) if __name__ == "__main__": main() urlscan-0.9.4/conf.py000066400000000000000000000003041353231375300144740ustar00rootroot00000000000000URLINTERNALPATTERN = r'[{}()@\w/\\\-%?!&.=:;+,#~]' URLTRAILINGPATTERN = r'[{}(@\w/\-%&=+#]' HTTPURLPATTERN = (r'(?:(https?|file|ftps?)://' + URLINTERNALPATTERN + r'*' + URLTRAILINGPATTERN + r')') urlscan-0.9.4/requirements.txt000066400000000000000000000000151353231375300164600ustar00rootroot00000000000000urwid>=1.2.1 urlscan-0.9.4/setup.py000077500000000000000000000012321353231375300147130ustar00rootroot00000000000000#!/usr/bin/env python from setuptools import setup setup(name="urlscan", version="0.9.4", description="View/select the URLs in an email message or file", author="Scott Hansen", author_email="firecat4153@gmail.com", url="https://github.com/firecat53/urlscan", download_url="https://github.com/firecat53/urlscan/archive/0.9.4.zip", packages=['urlscan'], scripts=['bin/urlscan'], package_data={'urlscan': ['assets/*']}, data_files=[('share/doc/urlscan', ['README.rst', 'COPYING']), ('share/man/man1', ['urlscan.1'])], license="GPLv2", install_requires=["urwid>=1.2.1"] ) urlscan-0.9.4/urlscan.1000066400000000000000000000127451353231375300147420ustar00rootroot00000000000000.\" Hey, EMACS: -*- nroff -*- .TH URLSCAN 1 "30 August 2019" .SH NAME urlscan \- browse the URLs in an email message from a terminal .SH SYNOPSIS \fBurlscan\fR [options] < .I message \fBurlscan\fR [options] .I message .SH DESCRIPTION \fBurlscan\fR accepts a single email message on standard input, then displays a terminal-based list of the URLs in the given message. Selecting a URL uses the Python webbrowser module to determine which browser to open. The \fBBROWSER\fR environment variable will be used if it is set. \fBurlscan\fR is primarily intended to be used with the .B mutt (1) mailreader, but it should work well with any terminal-based mail program. \fBurlscan\fR is similar to \fBurlview\fR(1), but has the following additional features: \fB1.\fR Support for more message encodings, such as quoted-printable and base64. \fB2.\fR Extraction and display of the context surrounding each URL. Toggle context view on/off with \fBc\fR. \fB3.\fR Copy current URL to primary selection with \fBP\fR or to clipboard with \fBC\fR. \fB4.\fR URLs are shortened by default to fit on one line. Toggle one or all shortened URLs with \fBs\fR or \fBS\fR. \fB5.\fR Incremental case-insensitive search using \fB/\fR. Footer shows current search term. \fB/\fR again resets search. \fB6.\fR Cycle through all available palettes (color and black & white available by default) using \fBp\fR. Running \fBurlscan \-g\fR will generate a ~/.config/urlscan/config.json file for editing or adding additional pallettes and keybindings. See http://urwid.org/manual/displayattributes.html#display-attributes for color options and allowed values. \fB7.\fR \fBu\fR will unescape the highlighted URL if necessary. \fB8.\fR Run a command with the selected URL as the argument or pipe the selected URL to a command using the \fB--run\fR and \fB--pipe\fR arguments. \fB9.\fR Use \fBl\fR to cycle through whether URLs are opened using the Python webbrowser module (default), xdg-open (if installed) or a function passed on the command line with \fB--run\fR. The \fB--run\fR function will respect the value of \fB--pipe\fR. \fB10.\fR \fBF1\fR shows the help menu. .SH OPTIONS .TP .B \-g, \-\-genconf Generate ~/.config/urlscan/config.json with default options. .TP .B \-c, \-\-compact Display a simple list of the extracted URLs, instead of showing the context of each URL. Also toggle with `c` from within the viewer. .TP .B \-H, \-\-nohelp Start with header menu hidden. .TP .B \-d, \-\-dedupe Remove duplicated URLs from the list of URLs. .TP .B \-n, \-\-no-browser Disables the selection interface and print the links to standard output. Useful for scripting (implies \fB\-\-compact\fR). .TP .B \-r, \-\-run \ Execute \ in place of opening URL with a browser. Use {} in \ to substitute in the URL. Examples: $ urlscan --run 'echo {} | xclip -i' file.txt $ urlscan --run 'tmux set buffer {}' .TP .B \-p, \-\-pipe Pipe the selected URL to the command specified by `--run`. This is preferred when the command supports it, as it is more secure and tolerant of special characters in the URL. Example: $ urlscan --run 'xclip -i' --pipe file.txt .SH MUTT INTEGRATION To integrate urlscan with mutt, include the following two commands in \fB~/.muttrc\fR: .ad l macro index,pager \\cb " urlscan" "call urlscan to extract URLs out of a message" macro attach,compose \\cb " urlscan" "call urlscan to extract URLs out of a message" .ad b Once these lines are in your mutt configuration file, pressing Control-b will allow you to browse and open the URLs in the currently selected message. Alternately, you can pipe a message into urlscan using the '|' operator. This can be useful for applying a different flag (such as the '-d' or '-c' options). .SH KEYBINDINGS Run \fBurlscan \-g\fR to generate ~/.config/urlscan/config.json. All of the keys will be listed. You can either leave in place or delete any that will not be altered. To unset a binding, set it equal to "". For example: \fB"P": ""\fR The follow actions are supported: .TP \fBall_escape\fR \-\- toggle unescape all URLs (Default: \fBu\fR) .TP \fBall_shorten\fR \-\- toggle shorten all URLs (Default: \fBS\fR) .TP \fBbottom\fR \-\- move cursor to last item (Default: \fBG\fR) .TP \fBclear_screen\fR \-\- redraw screen (Default: \fBCtrl-l\fR) .TP \fBclipboard\fR \-\- copy highlighted URL to clipboard using xsel/xclip (Default: \fBC\fR) .TP \fBclipboard_pri\fR \-\- copy highlighted URL to primary selection using xsel/xclip (Default: \fBP\fR) .TP \fBcontext\fR \-\- show/hide context (Default: \fBc\fR) .TP \fBdown\fR \-\- cursor down (Default: \fBj\fR) .TP \fBhelp_menu\fR \-\- show/hide help menu (Default: \fBF1\fR) .TP \fBlink_handler\fR \-\- cycle link handling (webbrowser, xdg-open or custom) (Default: \fBl\fR) .TP \fBopen_url\fR \-\- open selected URL (Default: \fBspace\fR or \fBenter\fR) .TP \fBpalette\fR \-\- cycle through palettes (Default: \fBp\fR) .TP \fBquit\fR \-\- quit (Default: \fBq\fR or \fBQ\fR) .TP \fBshorten\fR \-\- toggle shorten highlighted URL (Default: \fBs\fR) .TP \fBtop\fR \-\- move to first list item (Default: \fBg\fR) .TP \fBup\fR \-\- cursor up (Default: \fBk\fR) .SH FILES $HOME/.config/urlscan/config.json Only required if additional or modified palettes are desired. .SH SEE ALSO \fI/usr/share/doc/urlscan/README\fR, \fBurlview\fR(1), \fBmutt\fR(1) .SH AUTHOR This manual page was written by Daniel Burrows and Scott Hansen urlscan-0.9.4/urlscan/000077500000000000000000000000001353231375300146475ustar00rootroot00000000000000urlscan-0.9.4/urlscan/__init__.py000066400000000000000000000000561353231375300167610ustar00rootroot00000000000000__all__ = ['browser', 'urlchoose', 'urlscan'] urlscan-0.9.4/urlscan/assets/000077500000000000000000000000001353231375300161515ustar00rootroot00000000000000urlscan-0.9.4/urlscan/assets/tlds-alpha-by-domain.txt000066400000000000000000000241471353231375300226300ustar00rootroot00000000000000# Version 2019021501, Last Updated Fri Feb 15 07:07:01 2019 UTC AAA AARP ABARTH ABB ABBOTT ABBVIE ABC ABLE ABOGADO ABUDHABI AC ACADEMY ACCENTURE ACCOUNTANT ACCOUNTANTS ACO ACTIVE ACTOR AD ADAC ADS ADULT AE AEG AERO AETNA AF AFAMILYCOMPANY AFL AFRICA AG AGAKHAN AGENCY AI AIG AIGO AIRBUS AIRFORCE AIRTEL AKDN AL ALFAROMEO ALIBABA ALIPAY ALLFINANZ ALLSTATE ALLY ALSACE ALSTOM AM AMERICANEXPRESS AMERICANFAMILY AMEX AMFAM AMICA AMSTERDAM ANALYTICS ANDROID ANQUAN ANZ AO AOL APARTMENTS APP APPLE AQ AQUARELLE AR ARAB ARAMCO ARCHI ARMY ARPA ART ARTE AS ASDA ASIA ASSOCIATES AT ATHLETA ATTORNEY AU AUCTION AUDI AUDIBLE AUDIO AUSPOST AUTHOR AUTO AUTOS AVIANCA AW AWS AX AXA AZ AZURE BA BABY BAIDU BANAMEX BANANAREPUBLIC BAND BANK BAR BARCELONA BARCLAYCARD BARCLAYS BAREFOOT BARGAINS BASEBALL BASKETBALL BAUHAUS BAYERN BB BBC BBT BBVA BCG BCN BD BE BEATS BEAUTY BEER BENTLEY BERLIN BEST BESTBUY BET BF BG BH BHARTI BI BIBLE BID BIKE BING BINGO BIO BIZ BJ BLACK BLACKFRIDAY BLOCKBUSTER BLOG BLOOMBERG BLUE BM BMS BMW BN BNL BNPPARIBAS BO BOATS BOEHRINGER BOFA BOM BOND BOO BOOK BOOKING BOSCH BOSTIK BOSTON BOT BOUTIQUE BOX BR BRADESCO BRIDGESTONE BROADWAY BROKER BROTHER BRUSSELS BS BT BUDAPEST BUGATTI BUILD BUILDERS BUSINESS BUY BUZZ BV BW BY BZ BZH CA CAB CAFE CAL CALL CALVINKLEIN CAM CAMERA CAMP CANCERRESEARCH CANON CAPETOWN CAPITAL CAPITALONE CAR CARAVAN CARDS CARE CAREER CAREERS CARS CARTIER CASA CASE CASEIH CASH CASINO CAT CATERING CATHOLIC CBA CBN CBRE CBS CC CD CEB CENTER CEO CERN CF CFA CFD CG CH CHANEL CHANNEL CHARITY CHASE CHAT CHEAP CHINTAI CHRISTMAS CHROME CHRYSLER CHURCH CI CIPRIANI CIRCLE CISCO CITADEL CITI CITIC CITY CITYEATS CK CL CLAIMS CLEANING CLICK CLINIC CLINIQUE CLOTHING CLOUD CLUB CLUBMED CM CN CO COACH CODES COFFEE COLLEGE COLOGNE COM COMCAST COMMBANK COMMUNITY COMPANY COMPARE COMPUTER COMSEC CONDOS CONSTRUCTION CONSULTING CONTACT CONTRACTORS COOKING COOKINGCHANNEL COOL COOP CORSICA COUNTRY COUPON COUPONS COURSES CR CREDIT CREDITCARD CREDITUNION CRICKET CROWN CRS CRUISE CRUISES CSC CU CUISINELLA CV CW CX CY CYMRU CYOU CZ DABUR DAD DANCE DATA DATE DATING DATSUN DAY DCLK DDS DE DEAL DEALER DEALS DEGREE DELIVERY DELL DELOITTE DELTA DEMOCRAT DENTAL DENTIST DESI DESIGN DEV DHL DIAMONDS DIET DIGITAL DIRECT DIRECTORY DISCOUNT DISCOVER DISH DIY DJ DK DM DNP DO DOCS DOCTOR DODGE DOG DOHA DOMAINS DOT DOWNLOAD DRIVE DTV DUBAI DUCK DUNLOP DUNS DUPONT DURBAN DVAG DVR DZ EARTH EAT EC ECO EDEKA EDU EDUCATION EE EG EMAIL EMERCK ENERGY ENGINEER ENGINEERING ENTERPRISES EPSON EQUIPMENT ER ERICSSON ERNI ES ESQ ESTATE ESURANCE ET ETISALAT EU EUROVISION EUS EVENTS EVERBANK EXCHANGE EXPERT EXPOSED EXPRESS EXTRASPACE FAGE FAIL FAIRWINDS FAITH FAMILY FAN FANS FARM FARMERS FASHION FAST FEDEX FEEDBACK FERRARI FERRERO FI FIAT FIDELITY FIDO FILM FINAL FINANCE FINANCIAL FIRE FIRESTONE FIRMDALE FISH FISHING FIT FITNESS FJ FK FLICKR FLIGHTS FLIR FLORIST FLOWERS FLY FM FO FOO FOOD FOODNETWORK FOOTBALL FORD FOREX FORSALE FORUM FOUNDATION FOX FR FREE FRESENIUS FRL FROGANS FRONTDOOR FRONTIER FTR FUJITSU FUJIXEROX FUN FUND FURNITURE FUTBOL FYI GA GAL GALLERY GALLO GALLUP GAME GAMES GAP GARDEN GB GBIZ GD GDN GE GEA GENT GENTING GEORGE GF GG GGEE GH GI GIFT GIFTS GIVES GIVING GL GLADE GLASS GLE GLOBAL GLOBO GM GMAIL GMBH GMO GMX GN GODADDY GOLD GOLDPOINT GOLF GOO GOODYEAR GOOG GOOGLE GOP GOT GOV GP GQ GR GRAINGER GRAPHICS GRATIS GREEN GRIPE GROCERY GROUP GS GT GU GUARDIAN GUCCI GUGE GUIDE GUITARS GURU GW GY HAIR HAMBURG HANGOUT HAUS HBO HDFC HDFCBANK HEALTH HEALTHCARE HELP HELSINKI HERE HERMES HGTV HIPHOP HISAMITSU HITACHI HIV HK HKT HM HN HOCKEY HOLDINGS HOLIDAY HOMEDEPOT HOMEGOODS HOMES HOMESENSE HONDA HONEYWELL HORSE HOSPITAL HOST HOSTING HOT HOTELES HOTELS HOTMAIL HOUSE HOW HR HSBC HT HU HUGHES HYATT HYUNDAI IBM ICBC ICE ICU ID IE IEEE IFM IKANO IL IM IMAMAT IMDB IMMO IMMOBILIEN IN INC INDUSTRIES INFINITI INFO ING INK INSTITUTE INSURANCE INSURE INT INTEL INTERNATIONAL INTUIT INVESTMENTS IO IPIRANGA IQ IR IRISH IS ISELECT ISMAILI IST ISTANBUL IT ITAU ITV IVECO JAGUAR JAVA JCB JCP JE JEEP JETZT JEWELRY JIO JLL JM JMP JNJ JO JOBS JOBURG JOT JOY JP JPMORGAN JPRS JUEGOS JUNIPER KAUFEN KDDI KE KERRYHOTELS KERRYLOGISTICS KERRYPROPERTIES KFH KG KH KI KIA KIM KINDER KINDLE KITCHEN KIWI KM KN KOELN KOMATSU KOSHER KP KPMG KPN KR KRD KRED KUOKGROUP KW KY KYOTO KZ LA LACAIXA LADBROKES LAMBORGHINI LAMER LANCASTER LANCIA LANCOME LAND LANDROVER LANXESS LASALLE LAT LATINO LATROBE LAW LAWYER LB LC LDS LEASE LECLERC LEFRAK LEGAL LEGO LEXUS LGBT LI LIAISON LIDL LIFE LIFEINSURANCE LIFESTYLE LIGHTING LIKE LILLY LIMITED LIMO LINCOLN LINDE LINK LIPSY LIVE LIVING LIXIL LK LLC LOAN LOANS LOCKER LOCUS LOFT LOL LONDON LOTTE LOTTO LOVE LPL LPLFINANCIAL LR LS LT LTD LTDA LU LUNDBECK LUPIN LUXE LUXURY LV LY MA MACYS MADRID MAIF MAISON MAKEUP MAN MANAGEMENT MANGO MAP MARKET MARKETING MARKETS MARRIOTT MARSHALLS MASERATI MATTEL MBA MC MCKINSEY MD ME MED MEDIA MEET MELBOURNE MEME MEMORIAL MEN MENU MERCKMSD METLIFE MG MH MIAMI MICROSOFT MIL MINI MINT MIT MITSUBISHI MK ML MLB MLS MM MMA MN MO MOBI MOBILE MOBILY MODA MOE MOI MOM MONASH MONEY MONSTER MOPAR MORMON MORTGAGE MOSCOW MOTO MOTORCYCLES MOV MOVIE MOVISTAR MP MQ MR MS MSD MT MTN MTR MU MUSEUM MUTUAL MV MW MX MY MZ NA NAB NADEX NAGOYA NAME NATIONWIDE NATURA NAVY NBA NC NE NEC NET NETBANK NETFLIX NETWORK NEUSTAR NEW NEWHOLLAND NEWS NEXT NEXTDIRECT NEXUS NF NFL NG NGO NHK NI NICO NIKE NIKON NINJA NISSAN NISSAY NL NO NOKIA NORTHWESTERNMUTUAL NORTON NOW NOWRUZ NOWTV NP NR NRA NRW NTT NU NYC NZ OBI OBSERVER OFF OFFICE OKINAWA OLAYAN OLAYANGROUP OLDNAVY OLLO OM OMEGA ONE ONG ONL ONLINE ONYOURSIDE OOO OPEN ORACLE ORANGE ORG ORGANIC ORIGINS OSAKA OTSUKA OTT OVH PA PAGE PANASONIC PARIS PARS PARTNERS PARTS PARTY PASSAGENS PAY PCCW PE PET PF PFIZER PG PH PHARMACY PHD PHILIPS PHONE PHOTO PHOTOGRAPHY PHOTOS PHYSIO PIAGET PICS PICTET PICTURES PID PIN PING PINK PIONEER PIZZA PK PL PLACE PLAY PLAYSTATION PLUMBING PLUS PM PN PNC POHL POKER POLITIE PORN POST PR PRAMERICA PRAXI PRESS PRIME PRO PROD PRODUCTIONS PROF PROGRESSIVE PROMO PROPERTIES PROPERTY PROTECTION PRU PRUDENTIAL PS PT PUB PW PWC PY QA QPON QUEBEC QUEST QVC RACING RADIO RAID RE READ REALESTATE REALTOR REALTY RECIPES RED REDSTONE REDUMBRELLA REHAB REISE REISEN REIT RELIANCE REN RENT RENTALS REPAIR REPORT REPUBLICAN REST RESTAURANT REVIEW REVIEWS REXROTH RICH RICHARDLI RICOH RIGHTATHOME RIL RIO RIP RMIT RO ROCHER ROCKS RODEO ROGERS ROOM RS RSVP RU RUGBY RUHR RUN RW RWE RYUKYU SA SAARLAND SAFE SAFETY SAKURA SALE SALON SAMSCLUB SAMSUNG SANDVIK SANDVIKCOROMANT SANOFI SAP SARL SAS SAVE SAXO SB SBI SBS SC SCA SCB SCHAEFFLER SCHMIDT SCHOLARSHIPS SCHOOL SCHULE SCHWARZ SCIENCE SCJOHNSON SCOR SCOT SD SE SEARCH SEAT SECURE SECURITY SEEK SELECT SENER SERVICES SES SEVEN SEW SEX SEXY SFR SG SH SHANGRILA SHARP SHAW SHELL SHIA SHIKSHA SHOES SHOP SHOPPING SHOUJI SHOW SHOWTIME SHRIRAM SI SILK SINA SINGLES SITE SJ SK SKI SKIN SKY SKYPE SL SLING SM SMART SMILE SN SNCF SO SOCCER SOCIAL SOFTBANK SOFTWARE SOHU SOLAR SOLUTIONS SONG SONY SOY SPACE SPORT SPOT SPREADBETTING SR SRL SRT SS ST STADA STAPLES STAR STARHUB STATEBANK STATEFARM STC STCGROUP STOCKHOLM STORAGE STORE STREAM STUDIO STUDY STYLE SU SUCKS SUPPLIES SUPPLY SUPPORT SURF SURGERY SUZUKI SV SWATCH SWIFTCOVER SWISS SX SY SYDNEY SYMANTEC SYSTEMS SZ TAB TAIPEI TALK TAOBAO TARGET TATAMOTORS TATAR TATTOO TAX TAXI TC TCI TD TDK TEAM TECH TECHNOLOGY TEL TELEFONICA TEMASEK TENNIS TEVA TF TG TH THD THEATER THEATRE TIAA TICKETS TIENDA TIFFANY TIPS TIRES TIROL TJ TJMAXX TJX TK TKMAXX TL TM TMALL TN TO TODAY TOKYO TOOLS TOP TORAY TOSHIBA TOTAL TOURS TOWN TOYOTA TOYS TR TRADE TRADING TRAINING TRAVEL TRAVELCHANNEL TRAVELERS TRAVELERSINSURANCE TRUST TRV TT TUBE TUI TUNES TUSHU TV TVS TW TZ UA UBANK UBS UCONNECT UG UK UNICOM UNIVERSITY UNO UOL UPS US UY UZ VA VACATIONS VANA VANGUARD VC VE VEGAS VENTURES VERISIGN VERSICHERUNG VET VG VI VIAJES VIDEO VIG VIKING VILLAS VIN VIP VIRGIN VISA VISION VISTAPRINT VIVA VIVO VLAANDEREN VN VODKA VOLKSWAGEN VOLVO VOTE VOTING VOTO VOYAGE VU VUELOS WALES WALMART WALTER WANG WANGGOU WARMAN WATCH WATCHES WEATHER WEATHERCHANNEL WEBCAM WEBER WEBSITE WED WEDDING WEIBO WEIR WF WHOSWHO WIEN WIKI WILLIAMHILL WIN WINDOWS WINE WINNERS WME WOLTERSKLUWER WOODSIDE WORK WORKS WORLD WOW WS WTC WTF XBOX XEROX XFINITY XIHUAN XIN XN--11B4C3D XN--1CK2E1B XN--1QQW23A XN--2SCRJ9C XN--30RR7Y XN--3BST00M XN--3DS443G XN--3E0B707E XN--3HCRJ9C XN--3OQ18VL8PN36A XN--3PXU8K XN--42C2D9A XN--45BR5CYL XN--45BRJ9C XN--45Q11C XN--4GBRIM XN--54B7FTA0CC XN--55QW42G XN--55QX5D XN--5SU34J936BGSG XN--5TZM5G XN--6FRZ82G XN--6QQ986B3XL XN--80ADXHKS XN--80AO21A XN--80AQECDR1A XN--80ASEHDB XN--80ASWG XN--8Y0A063A XN--90A3AC XN--90AE XN--90AIS XN--9DBQ2A XN--9ET52U XN--9KRT00A XN--B4W605FERD XN--BCK1B9A5DRE4C XN--C1AVG XN--C2BR7G XN--CCK2B3B XN--CG4BKI XN--CLCHC0EA0B2G2A9GCD XN--CZR694B XN--CZRS0T XN--CZRU2D XN--D1ACJ3B XN--D1ALF XN--E1A4C XN--ECKVDTC9D XN--EFVY88H XN--ESTV75G XN--FCT429K XN--FHBEI XN--FIQ228C5HS XN--FIQ64B XN--FIQS8S XN--FIQZ9S XN--FJQ720A XN--FLW351E XN--FPCRJ9C3D XN--FZC2C9E2C XN--FZYS8D69UVGM XN--G2XX48C XN--GCKR3F0F XN--GECRJ9C XN--GK3AT1E XN--H2BREG3EVE XN--H2BRJ9C XN--H2BRJ9C8C XN--HXT814E XN--I1B6B1A6A2E XN--IMR513N XN--IO0A7I XN--J1AEF XN--J1AMH XN--J6W193G XN--JLQ61U9W7B XN--JVR189M XN--KCRX77D1X4A XN--KPRW13D XN--KPRY57D XN--KPU716F XN--KPUT3I XN--L1ACC XN--LGBBAT1AD8J XN--MGB9AWBF XN--MGBA3A3EJT XN--MGBA3A4F16A XN--MGBA7C0BBN0A XN--MGBAAKC7DVF XN--MGBAAM7A8H XN--MGBAB2BD XN--MGBAH1A3HJKRD XN--MGBAI9AZGQP6J XN--MGBAYH7GPA XN--MGBB9FBPOB XN--MGBBH1A XN--MGBBH1A71E XN--MGBC0A9AZCG XN--MGBCA7DZDO XN--MGBERP4A5D4AR XN--MGBGU82A XN--MGBI4ECEXP XN--MGBPL2FH XN--MGBT3DHD XN--MGBTX2B XN--MGBX4CD0AB XN--MIX891F XN--MK1BU44C XN--MXTQ1M XN--NGBC5AZD XN--NGBE9E0A XN--NGBRX XN--NODE XN--NQV7F XN--NQV7FS00EMA XN--NYQY26A XN--O3CW4H XN--OGBPF8FL XN--OTU796D XN--P1ACF XN--P1AI XN--PBT977C XN--PGBS0DH XN--PSSY2U XN--Q9JYB4C XN--QCKA1PMC XN--QXAM XN--RHQV96G XN--ROVU88B XN--RVC1E0AM3E XN--S9BRJ9C XN--SES554G XN--T60B56A XN--TCKWE XN--TIQ49XQYJ XN--UNUP4Y XN--VERMGENSBERATER-CTB XN--VERMGENSBERATUNG-PWB XN--VHQUV XN--VUQ861B XN--W4R85EL8FHU5DNRA XN--W4RS40L XN--WGBH1C XN--WGBL6A XN--XHQ521B XN--XKC2AL3HYE2A XN--XKC2DL3A5EE0H XN--Y9A3AQ XN--YFRO4I67O XN--YGBI2AMMX XN--ZFR164B XXX XYZ YACHTS YAHOO YAMAXUN YANDEX YE YODOBASHI YOGA YOKOHAMA YOU YOUTUBE YT YUN ZA ZAPPOS ZARA ZERO ZIP ZM ZONE ZUERICH ZW urlscan-0.9.4/urlscan/urlchoose.py000066400000000000000000000722601353231375300172330ustar00rootroot00000000000000# Copyright (C) 2006-2007 Daniel Burrows # Copyright (C) 2019 Scott Hansen # # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 2 # of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA """An urwid listview-based widget that lets you choose a URL from a list of URLs.""" import errno import json import os from os.path import dirname, exists, expanduser import re import shlex from subprocess import call, Popen, PIPE, DEVNULL import sys from threading import Thread from time import sleep import webbrowser import urwid import urwid.curses_display import urwid.raw_display def shorten_url(url, cols, shorten): """Shorten long URLs to fit on one line. """ cols = ((cols - 6) * .85) # 6 cols for urlref and don't use while line if shorten is False or len(url) < cols: return url split = int(cols * .5) return url[:split] + "..." + url[-split:] def grp_list(items): """Organize list of items [a,2,3,4,a,4,2,a,1, etc...] like: [[a,2,3,4], [a,4,2], [a,1]], where 'a' is a urwid.Divider """ grp = [] res = [] for item in items: if isinstance(item, urwid.Divider): res.append(grp) grp = [items[0]] else: grp.append(item) res.append(grp) return res[1:] def splittext(text, search, attr): """Split a text string by search string and add Urwid display attribute to the search term. Args: text - string search - search string attr - attribute string to add Returns: urwid markup list ["string", ("default", " mo"), "re string"] for search="mo", text="string more string" and attr="default" """ if search: pat = re.compile("({})".format(re.escape(search)), re.IGNORECASE) else: return text final = pat.split(text) final = [(attr, i) if i.lower() == search.lower() else i for i in final] return final class URLChooser: def __init__(self, extractedurls, compact=False, nohelp=False, dedupe=False, shorten=True, run="", pipe=False, genconf=False): self.conf = expanduser("~/.config/urlscan/config.json") self.keys = {'/': self._search_key, '0': self._digits, '1': self._digits, '2': self._digits, '3': self._digits, '4': self._digits, '5': self._digits, '6': self._digits, '7': self._digits, '8': self._digits, '9': self._digits, 'C': self._clipboard, 'c': self._context, 'ctrl l': self._clear_screen, 'f1': self._help_menu, 'G': self._bottom, 'g': self._top, 'j': self._down, 'k': self._up, 'P': self._clipboard_pri, 'l': self._link_handler, 'p': self._palette, 'Q': self._quit, 'q': self._quit, 'S': self._all_shorten, 's': self._shorten, 'u': self._all_escape } self.palettes = {} # Default color palette default = [('header', 'white', 'dark blue', 'standout'), ('footer', 'white', 'dark red', 'standout'), ('search', 'white', 'dark green', 'standout'), ('msgtext', '', ''), ('msgtext:ellipses', 'light gray', 'black'), ('urlref:number:braces', 'light gray', 'black'), ('urlref:number', 'yellow', 'black', 'standout'), ('urlref:url', 'white', 'black', 'standout'), ('url:sel', 'white', 'dark blue', 'bold')] # Default black & white palette blw = [('header', 'black', 'light gray', 'standout'), ('footer', 'black', 'light gray', 'standout'), ('search', 'black', 'light gray', 'standout'), ('msgtext', '', ''), ('msgtext:ellipses', 'white', 'black'), ('urlref:number:braces', 'white', 'black'), ('urlref:number', 'white', 'black', 'standout'), ('urlref:url', 'white', 'black', 'standout'), ('url:sel', 'black', 'light gray', 'bold')] self.palettes.update([("default", default), ("bw", blw)]) if genconf is True: self._config_create() try: with open(self.conf, 'r') as conf_file: data = json.load(conf_file) try: for pal_name, pal in data['palettes'].items(): self.palettes.update([(pal_name, [tuple(i) for i in pal])]) except KeyError: pass try: items = data['keys'].items() for key, value in items: if value: if value == "open_url": urwid.Button._command_map._command[key] = 'activate' value = getattr(self, "_{}".format(value)) else: del self.keys[key] continue self.keys.update([(key, value)]) except KeyError: pass except FileNotFoundError: pass try: call(['xdg-open'], stdout=DEVNULL) self.xdg = True except OSError: self.xdg = False self.shorten = shorten self.compact = compact self.run = run self.pipe = pipe self.search = False self.search_string = "" self.no_matches = False self.enter = False self.activate_keys = [i for i, j in urwid.Button._command_map._command.items() if j == 'activate'] self.items, self.urls = self.process_urls(extractedurls, dedupe=dedupe, shorten=self.shorten) # Store 'compact' mode items self.items_com = [i for i in self.items if isinstance(i, urwid.Columns) is True] if self.compact is True: self.items, self.items_com = self.items_com, self.items self.urls_unesc = [i.replace('\\', '') for i in self.urls] self.unesc = False # Original version of all items self.items_orig = self.items # Store items grouped into sections self.items_org = grp_list(self.items) listbox = urwid.ListBox(self.items) self.header = (":: F1 - help/keybindings :: " "q - quit :: " "/ - search :: " "URL opening mode - {}") self.link_open_modes = ["Web Browser", "Xdg-Open"] if self.xdg is True else ["Web Browser"] if self.run: self.link_open_modes.insert(0, self.run) self.nohelp = nohelp if nohelp is False: self.headerwid = urwid.AttrMap(urwid.Text( self.header.format(self.link_open_modes[0])), 'header') else: self.headerwid = None self.top = urwid.Frame(listbox, self.headerwid) if self.urls: self.top.body.focus_position = \ (2 if self.compact is False else 0) self.tui = urwid.curses_display.Screen() self.palette_names = list(self.palettes.keys()) self.palette_idx = 0 self.number = "" self.help_menu = False def main(self): """Urwid main event loop """ self.loop = urwid.MainLoop(self.top, self.palettes[self.palette_names[0]], screen=self.tui, handle_mouse=False, input_filter=self.handle_keys, unhandled_input=self.unhandled) self.loop.run() def handle_keys(self, keys, raw): """Handle widget default keys - 'Enter' or 'space' to load URL - 'Enter' to end search mode - add 'space' to search string in search mode - Workaround some small positioning bugs """ for j, k in enumerate(keys): if self.search is True: text = "Search: {}".format(self.search_string) if k == 'enter': # Catch 'enter' key to prevent opening URL in mkbrowseto self.enter = True if not self.items: self.search = False self.enter = False if self.search_string: footer = 'search' else: footer = 'default' text = "" footerwid = urwid.AttrMap(urwid.Text(text), footer) self.top.footer = footerwid elif k in self.activate_keys: self.search_string += k self._search() elif k == 'backspace': self.search_string = self.search_string[:-1] self._search() elif k in self.activate_keys and \ self.urls and \ self.search is False and \ self.help_menu is False: self._open_url() elif self.help_menu is True: self._help_menu() return [] if k == 'up': # Works around bug where the up arrow goes higher than the top list # item and unintentionally triggers context and palette switches. # Remaps 'up' to 'k' keys[j] = 'k' if k == 'home': # Remap 'home' to 'g'. Works around small bug where 'home' takes the cursor # above the top list item. keys[j] = 'g' # filter backspace out before the widget, it has a weird interaction return [i for i in keys if i != 'backspace'] def unhandled(self, key): """Handle other keyboard actions not handled by the ListBox widget. """ self.key = key self.size = self.tui.get_cols_rows() if self.search is True: if self.enter is False and self.no_matches is False: if len(key) == 1 and key.isprintable(): self.search_string += key self._search() elif self.enter is True and not self.search_string: self.search = False self.enter = False return if not self.urls and key not in "Qq": return # No other actions are useful with no URLs if self.help_menu is False: try: self.keys[key]() except KeyError: pass def _quit(self): """q/Q""" raise urwid.ExitMainLoop() def _open_url(self): """ or """ load_text = "Loading URL..." if self.link_open_modes[0] != self.run \ else "Executing: {}".format(self.run) if os.environ.get('BROWSER') not in ['elinks', 'links', 'w3m', 'lynx']: self._footer_start_thread(load_text, 5) def _help_menu(self): """F1""" if self.help_menu is False: self.focus_pos_saved = self.top.body.focus_position help_men = "\n".join(["{} - {}".format(i, j.__name__.strip('_')) for i, j in self.keys.items() if j.__name__ != '_digits']) help_men = "KEYBINDINGS\n" + help_men + "\n<0-9> - Jump to item" docs = ("OPTIONS\n" "all_escape -- toggle unescape all URLs\n" "all_shorten -- toggle shorten all URLs\n" "bottom -- move cursor to last item\n" "clear_screen -- redraw screen\n" "clipboard -- copy highlighted URL to clipboard using xsel/xclip\n" "clipboard_pri -- copy highlighted URL to primary selection using xsel/xclip\n" "config_create -- create ~/.config/urlscan/config.json\n" "context -- show/hide context\n" "down -- cursor down\n" "help_menu -- show/hide help menu\n" "link_handler -- cycle through xdg-open, webbrowser and user-defined function\n" "open_url -- open selected URL\n" "palette -- cycle through palettes\n" "quit -- quit\n" "shorten -- toggle shorten highlighted URL\n" "top -- move to first list item\n" "up -- cursor up\n") self.top.body = \ urwid.ListBox(urwid.SimpleListWalker([urwid.Columns([(30, urwid.Text(help_men)), urwid.Text(docs)])])) else: self.top.body = urwid.ListBox(self.items) self.top.body.focus_position = self.focus_pos_saved self.help_menu = not self.help_menu def _search_key(self): """ / """ if self.urls: self.search = True if self.compact is True: self.compact = False self.items, self.items_com = self.items_com, self.items else: return self.no_matches = False self.search_string = "" # Reset the search highlighting self._search() footerwid = urwid.AttrMap(urwid.Text("Search: "), 'footer') self.top.footer = footerwid self.items = self.items_orig self.top.body = urwid.ListBox(self.items) def _digits(self): """ 0-9 """ self.number += self.key try: if self.compact is False: self.top.body.focus_position = \ self.items.index(self.items_com[max(int(self.number) - 1, 0)]) else: self.top.body.focus_position = \ self.items.index(self.items[max(int(self.number) - 1, 0)]) except IndexError: self.number = self.number[:-1] self.top.keypress(self.size, "") # Trick urwid into redisplaying the cursor if self.number: self._footer_start_thread("Selection: {}".format(self.number), 1) def _clear_screen(self): """ Ctrl-l """ self.draw_screen(self.size) def _down(self): """ j """ self.top.keypress(self.size, "down") def _up(self): """ k """ self.top.keypress(self.size, "up") def _top(self): """ g """ # Goto top of the list self.top.body.focus_position = 2 if self.compact is False else 0 self.top.keypress(self.size, "") # Trick urwid into redisplaying the cursor def _bottom(self): """ G """ # Goto bottom of the list self.top.body.focus_position = len(self.items) - 1 self.top.keypress(self.size, "") # Trick urwid into redisplaying the cursor def _shorten(self): """ s """ # Toggle shortened URL for selected item fpo = self.top.body.focus_position url_idx = len([i for i in self.items[:fpo + 1] if isinstance(i, urwid.Columns)]) - 1 if self.compact is False and fpo <= 1: return url = self.urls[url_idx] short = not "..." in self.items[fpo][1].label self.items[fpo][1].set_label(shorten_url(url, self.size[0], short)) def _all_shorten(self): """ S """ # Toggle all shortened URLs self.shorten = not self.shorten urls = iter(self.urls) for item in self.items: # Each Column has (Text, Button). Update the Button label if isinstance(item, urwid.Columns): item[1].set_label(shorten_url(next(urls), self.size[0], self.shorten)) def _all_escape(self): """ u """ # Toggle all escaped URLs self.unesc = not self.unesc self.urls, self.urls_unesc = self.urls_unesc, self.urls urls = iter(self.urls) for item in self.items: # Each Column has (Text, Button). Update the Button label if isinstance(item, urwid.Columns): item[1].set_label(shorten_url(next(urls), self.size[0], self.shorten)) def _context(self): """ c """ # Show/hide context if self.search_string: # Reset search when toggling compact mode footerwid = urwid.AttrMap(urwid.Text(""), 'default') self.top.footer = footerwid self.search_string = "" self.items = self.items_orig fpo = self.top.body.focus_position self.items, self.items_com = self.items_com, self.items self.top.body = urwid.ListBox(self.items) self.top.body.focus_position = self._cur_focus(fpo) self.compact = not self.compact def _clipboard(self, pri=False): """ C """ # Copy highlighted url to clipboard fpo = self.top.body.focus_position url_idx = len([i for i in self.items[:fpo + 1] if isinstance(i, urwid.Columns)]) - 1 if self.compact is False and fpo <= 1: return url = self.urls[url_idx] if pri is True: cmds = ("xsel -i", "xclip -i") else: cmds = ("xsel -ib", "xclip -i -selection clipboard") for cmd in cmds: try: proc = Popen(shlex.split(cmd), stdin=PIPE) proc.communicate(input=url.encode(sys.getdefaultencoding())) self._footer_start_thread("Copied url to {} selection".format( "primary" if pri is True else "clipboard"), 5) except OSError: continue break def _clipboard_pri(self): """ P """ # Copy highlighted url to primary selection self._clipboard(pri=True) def _palette(self): """ p """ # Loop through available palettes self.palette_idx += 1 try: self.loop.screen.register_palette(self.palettes[self.palette_names[self.palette_idx]]) except IndexError: self.loop.screen.register_palette(self.palettes[self.palette_names[0]]) self.palette_idx = 0 self.loop.screen.clear() def _config_create(self): """ --genconf """ # Create ~/.config/urlscan/config.json if if doesn't exist if not exists(self.conf): os.makedirs(dirname(expanduser(self.conf)), exist_ok=True) keys = dict(zip(self.keys.keys(), [i.__name__.strip('_') for i in self.keys.values()])) with open(expanduser(self.conf), 'w') as pals: pals.writelines(json.dumps({"palettes": self.palettes, "keys": keys}, indent=4)) print("Created ~/.config/urlscan/config.json") else: print("~/.config/urlscan/config.json already exists") def _footer_start_thread(self, text, time): """Display given text in the footer. Clears after