cvdupdate-1.1.1/ 0000755 0001751 0000164 00000000000 14270561637 014305 5 ustar runner docker 0000000 0000000 cvdupdate-1.1.1/LICENSE 0000644 0001751 0000164 00000026135 14270561627 015320 0 ustar runner docker 0000000 0000000
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. cvdupdate-1.1.1/PKG-INFO 0000644 0001751 0000164 00000031642 14270561637 015410 0 ustar runner docker 0000000 0000000 Metadata-Version: 2.1
Name: cvdupdate
Version: 1.1.1
Summary: ClamAV Private Database Mirror Updater Tool
Home-page: https://github.com/Cisco-Talos/cvdupdate
Author: The ClamAV Team
Author-email: clamav-bugs@external.cisco.com
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Description-Content-Type: text/markdown
License-File: LICENSE
A tool to download and update clamav databases and database patch files
for the purposes of hosting your own database mirror.
Copyright (C) 2021-2022 Cisco Systems, Inc. and/or its affiliates. All rights reserved.
## About
This tool downloads the latest ClamAV databases along with the latest database patch files.
This project replaces the `clamdownloader.pl` Perl script by Frederic Vanden Poel, formerly provided here: https://www.clamav.net/documents/private-local-mirrors
Run this tool as often as you like, but it will only download new content if there is new content to download. If you somehow manage to download too frequently (eg: by using `cvd clean all` and `cvd update` repeatedly), then the official database server may refuse your download request, and one or more databases may go on cool-down until it's safe to try again.
## Requirements
- Python 3.6 or newer.
- An internet connection with DNS enabled.
- The following Python packages. These will be installed automatically if you use `pip`, but may need to be installed manually otherwise:
- `click` v7.0 or newer
- `coloredlogs` v10.0 or newer
- `colorama`
- `requests`
- `dnspython` v2.1.0 or newer
- `rangehttpserver`
## Installation
You may install `cvdupdate` from PyPI using `pip`, or you may clone the project Git repository and use `pip` to install it locally.
Install `cvdupdate` from PyPI:
```bash
python3 -m pip install --user cvdupdate
```
## Basic Usage
Use the `--help` option with any `cvd` command to get help.
```bash
cvd --help
```
> _Tip_: You may not be able to run the `cvd` or `cvdupdate` shortcut directly if your Python Scripts directory is not in your `PATH` environment variable. If you run into this issue, and do not wish to add the Python Scripts directory to your path, you can run CVD-Update like this:
>
> ```bash
> python -m cvdupdate --help
> ```
(optional) You may wish to customize where the databases are stored:
```bash
cvd config set --dbdir
```
Run this to download the latest database and associated CDIFF patch files:
```bash
cvd update
```
Downloaded databases will be placed in `~/.cvdupdate/database` unless you customized it to use a different directory.
Newly downloaded databases will replace the previous database version, but the CDIFF patch files will accumulate up to a configured maximum before it starts deleting old CDIFFs (default: 30 CDIFFs). You can configure it to keep more CDIFFs by manually editing the config (default: `~/.cvdupdate/config.json`). The same behavior applies for CVD-Update log rotation.
Run this to serve up the database directory on `http://localhost:8000` so you can test it with FreshClam.
```bash
cvd serve
```
> _Disclaimer_: The `cvd serve` feature is not intended for production use, just for testing. You probably want to use a more robust HTTP server for production work.
Install ClamAV if you don't already have it and, in another terminal window, modify your `freshclam.conf` file. Replace:
```
DatabaseMirror database.clamav.net
```
... with:
```
DatabaseMirror http://localhost:8000
```
> _Tip_: A default install on Linux/Unix places `freshclam.conf` in `/usr/local/etc/freshclam.conf`. If one does not exist, you may need to create it using `freshclam.conf.sample` as a template.
Now, run `freshclam -v` or `freshclam.exe -v` to see what happens. You should see FreshClam successfully update it's own database directory from your private database server.
Run `cvd update` as often as you need. Maybe put it in a `cron` job.
> _Tip_: Each command supports a `--verbose` (`-V`) mode, which often provides more details about what's going on under the hood.
### Cron Example
Cron is a popular choice to automate frequent tasks on Linux / Unix systems.
1. Open a terminal running as the user which you want CVD-Update to run under, do the following:
```bash
crontab -e
```
2. Press `i` to insert new text, and add this line:
```bash
30 */4 * * * /bin/sh -c "~/.local/bin/cvd update &> /dev/null"
```
Or instead of `~/`, you can do this, replacing `username` with your user name:
```bash
30 */4 * * * /bin/sh -c "/home/username/.local/bin/cvd update &> /dev/null"
```
3. Press , then type `:wq` and press to write the file to disk and quit.
**About these settings**:
I selected `30 */4 * * *` to run at minute 30 past every 4th hour. CVD-Update uses a DNS check to do version checks before it attempts to download any files, just like FreshClam. Running CVD-Update more than once a day should not be an issue.
CVD-Update will write logs to the `~/.cvdupdate/logs` directory, which is why I directed `stdout` and `stderr` to `/dev/null` instead of a log file. You can use the `cvd config set` command to customize the log directory if you like, or redirect `stdout` and `stderr` to a log file if you prefer everything in one log instead of separate daily logs.
## Optional Functionality
### Using a custom DNS server
DNS is required for CVD-Update to function properly (to gather the TXT record containing the current definition database version). You can select a specific nameserver to ensure said nameserver is used when querying the TXT record containing the current database definition version available
1. Set the nameserver in the config. Eg:
```bash
cvd config set --nameserver 208.67.222.222
```
2. Set the environment variable `CVDUPDATE_NAMESERVER`. Eg:
```bash
CVDUPDATE_NAMESERVER="208.67.222.222" cvd update
```
The environment variable will take precedence over the nameserver config setting.
Note: Both options can be used to provide a comma-delimited list of nameservers to utilize for resolution.
### Using a proxy
Depending on your type of proxy, you may be able to use CVD-Update with your proxy by running CVD-Update like this:
First, set a custom domain name server to use the proxy:
```bash
cvd config set --nameserver
```
Then run CVD-Update like this:
```bash
http_proxy=http://: https_proxy=http://: cvd update -V
```
Or create a script to wrap the CVD-Update call. Something like:
```bash
#!/bin/bash
http_proxy=http://:
export http_proxy
https_proxy=http://:
export https_proxy
cvd update -V
```
> _Disclaimer_: CVD-Update doesn't support proxies that require authentication at this time. If your network admin allows it, you may be able to work around it by updating your proxy to allow HTTP requests through unauthenticated if the User-Agent matches your specific CVD-Update user agent. The CVD-Update User-Agent follows the form `CVDUPDATE/ ()` where the `uuid` is unique to your installation and can be found in the `~/.cvdupdate/state.json` file (or `~/.cvdupdate/config.json` for cvdupdate <=1.0.2). See https://github.com/Cisco-Talos/cvdupdate/issues/9 for more details.
>
> Adding support for proxy authentication is a ripe opportunity for a community contribution to the project.
## Files and directories created by CVD-Update
This tool is to creates the following directories:
- `~/.cvdupdate`
- `~/.cvdupdate/logs`
- `~/.cvdupdate/databases`
This tool creates the following files:
- `~/.cvdupdate/config.json`
- `~/.cvdupdate/state.json`
- `~/.cvdupdate/databases/.cvd`
- `~/.cvdupdate/databases/-.cdiff`
- `~/.cvdupdate/logs/.log`
> _Tip_: You can set custom `database` and `logs` directories with the `cvd config set` command. It is likely you will want to customize the `database` directory to point to your HTTP server's `www` directory (or equivalent). Bare in mind that if you already downloaded the databases to the old directory, you may want to move them to the new directory.
> _Important_: If you want to use a custom config path, you'll have to use it in every command. If you're fine with having it go in `~/.cvdupdate/config.json`, don't worry about it.
## Additional Usage
### Get familiar with the tool
Familiarize yourself with the various commands using the `--help` option.
```bash
cvd --help
cvd config --help
cvd update --help
cvd clean --help
```
Print out the current list of databases.
```bash
cvd list -V
```
Print out the config to see what it looks like.
```bash
cvd config show
```
### Do an update
Do an update, use "verbose mode" to so you can get a feel for how it works.
```bash
cvd update -V
```
List out the databases again:
```bash
cvd list -V
```
The print out the config again so you can see what's changed.
```bash
cvd config show
```
And maybe take a peek in the database directory as well to see it for yourself.
```bash
ls ~/.cvdupdate/database
```
Have a look at the logs if you wish.
```bash
ls ~/.cvdupdate/logs
cat ~/.cvdupdate/logs/*
```
### Serve it up, Test out FreshClam
Test out your mirror with FreshClam on the same computer.
This tool includes a `--serve` feature that will host the current database directory on http://localhost (default port: 8000).
You can test it by running `freshclam` or `freshclam.exe` locally, where you've configured `freshclam.conf` with:
```
DatabaseMirror http://localhost:8000
```
## Contribute
We'd love your help. There are many ways to contribute!
### Community
Join the ClamAV community on the [ClamAV Discord chat server](https://discord.gg/sGaxA5Q).
### Report issues
If you find an issue with CVD-Update or the CVD-Update documentation, please submit an issue to our [GitHub issue tracker](https://github.com/Cisco-Talos/cvdupdate/issues). Before you submit, please check to if someone else has already reported the issue.
### Development
If you find a bug and you're able to craft a fix yourself, consider submitting the fix in a [pull request](https://github.com/Cisco-Talos/cvdupdate/pulls). Your help will be greatly appreciated.
If you want to contribute to the project and don't have anything specific in mind, please check out our [issue tracker](https://github.com/Cisco-Talos/cvdupdate/issues). Perhaps you'll be able to fix a bug or add a cool new feature.
_By submitting a contribution to the project, you acknowledge and agree to assign Cisco Systems, Inc the copyright for the contribution. If you submit a significant contribution such as a new feature or capability or a large amount of code, you may be asked to sign a contributors license agreement comfirming that Cisco will have copyright license and patent license and that you are authorized to contribute the code._
#### Development Set-up
The following steps are intended to help users that wish to contribute to development of the CVD-Update project get started.
1. Create a fork of the [CVD-Update git repository](https://github.com/Cisco-Talos/cvdupdate), and then clone your fork to a local directory.
For example:
```bash
git clone https://github.com//cvdupdate.git
```
2. Make sure CVD-Update is not already installed. If it is, remove it.
```bash
python3 -m pip uninstall cvdupdate
```
3. Use pip to install CVD-Update in "edit" mode.
```bash
python3 -m pip install -e --user ./cvdupdate
```
Once installed in "edit" mode, any changes you make to your clone of the CVD-Update code will be immediately usable simply by running the `cvdupdate` / `cvd` commands.
### Conduct
This project has not selected a specific Code-of-Conduct document at this time. However, contributors are expected to behave in professional and respectful manner. Disrespectful or inappropriate behavior will not be tolerated.
## License
CVD-Update is licensed under the Apache License, Version 2.0 (the "License"). You may not use the CVD-Update project except in compliance with the License.
A copy of the license is located [here](LICENSE), and is also available online at [apache.org](http://www.apache.org/licenses/LICENSE-2.0).
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
cvdupdate-1.1.1/README.md 0000644 0001751 0000164 00000030742 14270561627 015571 0 ustar runner docker 0000000 0000000 A tool to download and update clamav databases and database patch files
for the purposes of hosting your own database mirror.
Copyright (C) 2021-2022 Cisco Systems, Inc. and/or its affiliates. All rights reserved.
## About
This tool downloads the latest ClamAV databases along with the latest database patch files.
This project replaces the `clamdownloader.pl` Perl script by Frederic Vanden Poel, formerly provided here: https://www.clamav.net/documents/private-local-mirrors
Run this tool as often as you like, but it will only download new content if there is new content to download. If you somehow manage to download too frequently (eg: by using `cvd clean all` and `cvd update` repeatedly), then the official database server may refuse your download request, and one or more databases may go on cool-down until it's safe to try again.
## Requirements
- Python 3.6 or newer.
- An internet connection with DNS enabled.
- The following Python packages. These will be installed automatically if you use `pip`, but may need to be installed manually otherwise:
- `click` v7.0 or newer
- `coloredlogs` v10.0 or newer
- `colorama`
- `requests`
- `dnspython` v2.1.0 or newer
- `rangehttpserver`
## Installation
You may install `cvdupdate` from PyPI using `pip`, or you may clone the project Git repository and use `pip` to install it locally.
Install `cvdupdate` from PyPI:
```bash
python3 -m pip install --user cvdupdate
```
## Basic Usage
Use the `--help` option with any `cvd` command to get help.
```bash
cvd --help
```
> _Tip_: You may not be able to run the `cvd` or `cvdupdate` shortcut directly if your Python Scripts directory is not in your `PATH` environment variable. If you run into this issue, and do not wish to add the Python Scripts directory to your path, you can run CVD-Update like this:
>
> ```bash
> python -m cvdupdate --help
> ```
(optional) You may wish to customize where the databases are stored:
```bash
cvd config set --dbdir
```
Run this to download the latest database and associated CDIFF patch files:
```bash
cvd update
```
Downloaded databases will be placed in `~/.cvdupdate/database` unless you customized it to use a different directory.
Newly downloaded databases will replace the previous database version, but the CDIFF patch files will accumulate up to a configured maximum before it starts deleting old CDIFFs (default: 30 CDIFFs). You can configure it to keep more CDIFFs by manually editing the config (default: `~/.cvdupdate/config.json`). The same behavior applies for CVD-Update log rotation.
Run this to serve up the database directory on `http://localhost:8000` so you can test it with FreshClam.
```bash
cvd serve
```
> _Disclaimer_: The `cvd serve` feature is not intended for production use, just for testing. You probably want to use a more robust HTTP server for production work.
Install ClamAV if you don't already have it and, in another terminal window, modify your `freshclam.conf` file. Replace:
```
DatabaseMirror database.clamav.net
```
... with:
```
DatabaseMirror http://localhost:8000
```
> _Tip_: A default install on Linux/Unix places `freshclam.conf` in `/usr/local/etc/freshclam.conf`. If one does not exist, you may need to create it using `freshclam.conf.sample` as a template.
Now, run `freshclam -v` or `freshclam.exe -v` to see what happens. You should see FreshClam successfully update it's own database directory from your private database server.
Run `cvd update` as often as you need. Maybe put it in a `cron` job.
> _Tip_: Each command supports a `--verbose` (`-V`) mode, which often provides more details about what's going on under the hood.
### Cron Example
Cron is a popular choice to automate frequent tasks on Linux / Unix systems.
1. Open a terminal running as the user which you want CVD-Update to run under, do the following:
```bash
crontab -e
```
2. Press `i` to insert new text, and add this line:
```bash
30 */4 * * * /bin/sh -c "~/.local/bin/cvd update &> /dev/null"
```
Or instead of `~/`, you can do this, replacing `username` with your user name:
```bash
30 */4 * * * /bin/sh -c "/home/username/.local/bin/cvd update &> /dev/null"
```
3. Press , then type `:wq` and press to write the file to disk and quit.
**About these settings**:
I selected `30 */4 * * *` to run at minute 30 past every 4th hour. CVD-Update uses a DNS check to do version checks before it attempts to download any files, just like FreshClam. Running CVD-Update more than once a day should not be an issue.
CVD-Update will write logs to the `~/.cvdupdate/logs` directory, which is why I directed `stdout` and `stderr` to `/dev/null` instead of a log file. You can use the `cvd config set` command to customize the log directory if you like, or redirect `stdout` and `stderr` to a log file if you prefer everything in one log instead of separate daily logs.
## Optional Functionality
### Using a custom DNS server
DNS is required for CVD-Update to function properly (to gather the TXT record containing the current definition database version). You can select a specific nameserver to ensure said nameserver is used when querying the TXT record containing the current database definition version available
1. Set the nameserver in the config. Eg:
```bash
cvd config set --nameserver 208.67.222.222
```
2. Set the environment variable `CVDUPDATE_NAMESERVER`. Eg:
```bash
CVDUPDATE_NAMESERVER="208.67.222.222" cvd update
```
The environment variable will take precedence over the nameserver config setting.
Note: Both options can be used to provide a comma-delimited list of nameservers to utilize for resolution.
### Using a proxy
Depending on your type of proxy, you may be able to use CVD-Update with your proxy by running CVD-Update like this:
First, set a custom domain name server to use the proxy:
```bash
cvd config set --nameserver
```
Then run CVD-Update like this:
```bash
http_proxy=http://: https_proxy=http://: cvd update -V
```
Or create a script to wrap the CVD-Update call. Something like:
```bash
#!/bin/bash
http_proxy=http://:
export http_proxy
https_proxy=http://:
export https_proxy
cvd update -V
```
> _Disclaimer_: CVD-Update doesn't support proxies that require authentication at this time. If your network admin allows it, you may be able to work around it by updating your proxy to allow HTTP requests through unauthenticated if the User-Agent matches your specific CVD-Update user agent. The CVD-Update User-Agent follows the form `CVDUPDATE/ ()` where the `uuid` is unique to your installation and can be found in the `~/.cvdupdate/state.json` file (or `~/.cvdupdate/config.json` for cvdupdate <=1.0.2). See https://github.com/Cisco-Talos/cvdupdate/issues/9 for more details.
>
> Adding support for proxy authentication is a ripe opportunity for a community contribution to the project.
## Files and directories created by CVD-Update
This tool is to creates the following directories:
- `~/.cvdupdate`
- `~/.cvdupdate/logs`
- `~/.cvdupdate/databases`
This tool creates the following files:
- `~/.cvdupdate/config.json`
- `~/.cvdupdate/state.json`
- `~/.cvdupdate/databases/.cvd`
- `~/.cvdupdate/databases/-.cdiff`
- `~/.cvdupdate/logs/.log`
> _Tip_: You can set custom `database` and `logs` directories with the `cvd config set` command. It is likely you will want to customize the `database` directory to point to your HTTP server's `www` directory (or equivalent). Bare in mind that if you already downloaded the databases to the old directory, you may want to move them to the new directory.
> _Important_: If you want to use a custom config path, you'll have to use it in every command. If you're fine with having it go in `~/.cvdupdate/config.json`, don't worry about it.
## Additional Usage
### Get familiar with the tool
Familiarize yourself with the various commands using the `--help` option.
```bash
cvd --help
cvd config --help
cvd update --help
cvd clean --help
```
Print out the current list of databases.
```bash
cvd list -V
```
Print out the config to see what it looks like.
```bash
cvd config show
```
### Do an update
Do an update, use "verbose mode" to so you can get a feel for how it works.
```bash
cvd update -V
```
List out the databases again:
```bash
cvd list -V
```
The print out the config again so you can see what's changed.
```bash
cvd config show
```
And maybe take a peek in the database directory as well to see it for yourself.
```bash
ls ~/.cvdupdate/database
```
Have a look at the logs if you wish.
```bash
ls ~/.cvdupdate/logs
cat ~/.cvdupdate/logs/*
```
### Serve it up, Test out FreshClam
Test out your mirror with FreshClam on the same computer.
This tool includes a `--serve` feature that will host the current database directory on http://localhost (default port: 8000).
You can test it by running `freshclam` or `freshclam.exe` locally, where you've configured `freshclam.conf` with:
```
DatabaseMirror http://localhost:8000
```
## Contribute
We'd love your help. There are many ways to contribute!
### Community
Join the ClamAV community on the [ClamAV Discord chat server](https://discord.gg/sGaxA5Q).
### Report issues
If you find an issue with CVD-Update or the CVD-Update documentation, please submit an issue to our [GitHub issue tracker](https://github.com/Cisco-Talos/cvdupdate/issues). Before you submit, please check to if someone else has already reported the issue.
### Development
If you find a bug and you're able to craft a fix yourself, consider submitting the fix in a [pull request](https://github.com/Cisco-Talos/cvdupdate/pulls). Your help will be greatly appreciated.
If you want to contribute to the project and don't have anything specific in mind, please check out our [issue tracker](https://github.com/Cisco-Talos/cvdupdate/issues). Perhaps you'll be able to fix a bug or add a cool new feature.
_By submitting a contribution to the project, you acknowledge and agree to assign Cisco Systems, Inc the copyright for the contribution. If you submit a significant contribution such as a new feature or capability or a large amount of code, you may be asked to sign a contributors license agreement comfirming that Cisco will have copyright license and patent license and that you are authorized to contribute the code._
#### Development Set-up
The following steps are intended to help users that wish to contribute to development of the CVD-Update project get started.
1. Create a fork of the [CVD-Update git repository](https://github.com/Cisco-Talos/cvdupdate), and then clone your fork to a local directory.
For example:
```bash
git clone https://github.com//cvdupdate.git
```
2. Make sure CVD-Update is not already installed. If it is, remove it.
```bash
python3 -m pip uninstall cvdupdate
```
3. Use pip to install CVD-Update in "edit" mode.
```bash
python3 -m pip install -e --user ./cvdupdate
```
Once installed in "edit" mode, any changes you make to your clone of the CVD-Update code will be immediately usable simply by running the `cvdupdate` / `cvd` commands.
### Conduct
This project has not selected a specific Code-of-Conduct document at this time. However, contributors are expected to behave in professional and respectful manner. Disrespectful or inappropriate behavior will not be tolerated.
## License
CVD-Update is licensed under the Apache License, Version 2.0 (the "License"). You may not use the CVD-Update project except in compliance with the License.
A copy of the license is located [here](LICENSE), and is also available online at [apache.org](http://www.apache.org/licenses/LICENSE-2.0).
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
cvdupdate-1.1.1/cvdupdate/ 0000755 0001751 0000164 00000000000 14270561637 016264 5 ustar runner docker 0000000 0000000 cvdupdate-1.1.1/cvdupdate/__init__.py 0000644 0001751 0000164 00000000000 14270561627 020362 0 ustar runner docker 0000000 0000000 cvdupdate-1.1.1/cvdupdate/__main__.py 0000644 0001751 0000164 00000022617 14270561627 020365 0 ustar runner docker 0000000 0000000 #!/usr/bin/env python3
"""
CVD-Update: ClamAV Database Updater
"""
_description = """
A tool to download and update clamav databases and database patch files
for the purposes of hosting your own database mirror.
"""
_copyright = """
Copyright (C) 2021-2022 Cisco Systems, Inc. and/or its affiliates. All rights reserved.
"""
"""
Author: Micah Snyder
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
"""
import logging
import os
from pathlib import Path
import sys
import click
import coloredlogs
from http.server import HTTPServer
from RangeHTTPServer import RangeRequestHandler
from cvdupdate import auto_updater
import pkg_resources
from cvdupdate.cvdupdate import CVDUpdate
logging.basicConfig()
module_logger = logging.getLogger("cvdupdate")
coloredlogs.install(level="DEBUG", fmt="%(asctime)s %(name)s %(levelname)s %(message)s")
module_logger.setLevel(logging.DEBUG)
from colorama import Fore, Back, Style
#
# CLI Interface
#
@click.group(
epilog=Fore.BLUE
+ __doc__ + "\n"
+ Fore.GREEN
+ _description + "\n"
+ f"\nVersion {pkg_resources.get_distribution('cvdupdate').version}\n"
+ Style.RESET_ALL
+ _copyright,
)
def cli():
pass
@cli.command("list")
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
def db_list(config: str, verbose: bool):
"""
List the DBs found in the database directory.
"""
m = CVDUpdate(config=config, verbose=verbose)
m.db_list()
@cli.command("show")
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
@click.argument("db", required=True)
def db_show(config: str, verbose: bool, db: str):
"""
Show details about a specific database.
"""
m = CVDUpdate(config=config, verbose=verbose)
if not m.db_show(db):
sys.exit(1)
@cli.command("update")
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
@click.option("--debug-mode", "-D", is_flag=True, default=False, help="Print out HTTP headers for debugging purposes. [optional]")
@click.argument("db", required=False, default="")
def db_update(config: str, verbose: bool, db: str, debug_mode: bool):
"""
Update the DBs from the internet. Will update all DBs if DB not specified.
"""
m = CVDUpdate(config=config, verbose=verbose)
errors = m.db_update(db, debug_mode)
if errors > 0:
sys.exit(errors)
@cli.command("add")
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
@click.argument("db", required=True)
@click.argument("url", required=True)
def db_add(config: str, verbose: bool, db: str, url: str):
"""
Add a db to the list of known DBs.
"""
m = CVDUpdate(config=config, verbose=verbose)
if not m.config_add_db(db, url=url):
sys.exit(1)
@cli.command("remove")
@click.option("--config", "-c", type=str, required=False, default="")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
@click.argument("db", required=True)
def db_remove(config: str, verbose: bool, db: str):
"""
Remove a db from the list of known DBs and delete local copies of the DB.
"""
m = CVDUpdate(config=config, verbose=verbose)
if not m.config_remove_db(db):
sys.exit(1)
@cli.group(help="Commands to configure.")
def config():
pass
@config.command("set")
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
@click.option("--logdir", "-l", type=click.Path(), required=False, default="", help="Set a custom log directory. [optional]")
@click.option("--dbdir", "-d", type=click.Path(), required=False, default="", help="Set a custom database directory. [optional]")
@click.option("--nameserver", "-n", type=click.STRING, required=False, default="", help="Set a custom DNS nameserver. [optional]")
def config_set(config: str, verbose: bool, logdir: str, dbdir: str, nameserver: str):
"""
Set up first time configuration.
The default directories will be in ~/.cvdupdate
"""
CVDUpdate(
config=config,
verbose=verbose,
log_dir=logdir,
db_dir=dbdir,
nameserver=nameserver)
@config.command("show")
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
def config_show(config: str, verbose: bool):
"""
Print out the current configuration.
"""
m = CVDUpdate(config=config, verbose=verbose)
m.config_show()
@cli.group(help="Commands to clean up.")
def clean():
pass
@clean.command("dbs")
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
def clean_dbs(config: str, verbose: bool):
"""
Delete all files in the database directory.
"""
m = CVDUpdate(config=config, verbose=verbose)
m.clean_dbs()
@clean.command("logs")
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
def clean_logs(config: str, verbose: bool):
"""
Delete all files in the logs directory
"""
m = CVDUpdate(config=config, verbose=verbose)
m.clean_logs()
@clean.command("all")
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
def clean_all(config: str, verbose: bool):
"""
Delete the logs, databases, and config file.
"""
m = CVDUpdate(config=config, verbose=verbose)
m.clean_all()
@cli.command("serve")
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
@click.option("--update-interval-seconds", "-U", type=click.INT, required=False, default=0, help="Time in seconds before the next database update")
@click.argument("port", type=int, required=False, default=8000)
def serve(port: int, config: str, verbose: bool, update_interval_seconds: int):
"""
Serve up the database directory. Not a production quality server.
Intended for testing purposes.
"""
m = CVDUpdate(config=config, verbose=verbose)
os.chdir(str(m.db_dir))
m.logger.info(f"Serving up {m.db_dir} on localhost:{port}...")
auto_updater.start(update_interval_seconds)
RangeRequestHandler.protocol_version = 'HTTP/1.0'
# TODO(danvk): pick a random, available port
httpd = HTTPServer(('', port), RangeRequestHandler)
httpd.serve_forever()
#
# Command Aliases
#
@cli.command("list")
@click.pass_context
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
def list_alias(ctx, config: str, verbose: bool):
"""
List the DBs found in the database directory.
This is just an alias for `db list`.
"""
ctx.forward(db_list)
@cli.command("show")
@click.pass_context
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
@click.argument("db", required=True)
def show_alias(ctx, config: str, verbose: bool, db: str):
"""
Show details about a specific database.
This is just an alias for `db show`.
"""
ctx.forward(db_show)
@cli.command("update")
@click.pass_context
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
@click.option("--debug-mode", "-D", is_flag=True, default=False, help="Print out HTTP headers for debugging purposes. [optional]")
@click.argument("db", required=False, default="")
def update_alias(ctx, config: str, verbose: bool, db: str, debug_mode: bool):
"""
Update local copy of DBs.
This is just an alias for `db show`.
"""
ctx.forward(db_update)
if __name__ == "__main__":
sys.argv[0] = "cvdupdate"
cli(sys.argv[1:])
cvdupdate-1.1.1/cvdupdate/auto_updater.py 0000644 0001751 0000164 00000001545 14270561627 021336 0 ustar runner docker 0000000 0000000 from threading import Event, Thread
from cvdupdate.cvdupdate import CVDUpdate
def start(interval: int) -> None:
"""Spawn a thread to update the AV db after "interval" seconds
:param interval: the interval in seconds between 2 updates of the db
"""
if interval > 0:
Thread(target=_update, daemon=True, args=[interval]).start()
def _update(interval: int) -> None:
"""Don't call this directly
Updates the AV db after every "interval" seconds when it was started
:param interval: the interval in seconds between 2 updates of the db
"""
ticker = Event()
m = CVDUpdate()
m.logger.info(f"Updating the database every {interval} seconds")
while not ticker.wait(interval):
errors = m.db_update(debug_mode=True)
if errors > 0:
m.logger.error("Failed to fetch updates from ClamAV databases")
cvdupdate-1.1.1/cvdupdate/cvdupdate.py 0000644 0001751 0000164 00000131777 14270561627 020634 0 ustar runner docker 0000000 0000000 """
Copyright (C) 2021-2022 Cisco Systems, Inc. and/or its affiliates. All rights reserved.
This module provides a tool to download and update clamav databases and database
patch files (CDIFFs) for the purposes of hosting your own database mirror.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
"""
from pathlib import Path
import copy
import datetime
from enum import Enum
import json
import logging
import os
import platform
import pkg_resources
import re
import subprocess
import sys
import time
from typing import *
import uuid
import http.client as http_client
from dns import resolver
import requests
from requests import status_codes
class CvdStatus(Enum):
NO_UPDATE = 0
UPDATED = 1
ERROR = 2
class CVDUpdate:
default_config_path: Path = Path.home() / ".cvdupdate" / "config.json"
default_config: dict = {
"nameserver" : "",
"max retry" : 3, # No `cvd config set` option to set this, because we don't
# _really_ want people hammering the CDN with partial downloads.
"log directory" : str(Path.home() / ".cvdupdate" / "logs"),
"rotate logs" : True,
"# logs to keep" : 30,
"db directory" : str(Path.home() / ".cvdupdate" / "database"),
"rotate cdiffs" : True,
"# cdiffs to keep" : 30,
"state file": "",
}
default_state: dict = {
"dbs" : {
"main.cvd" : {
"url" : "https://database.clamav.net/main.cvd",
"retry after" : 0,
"last modified" : 0,
"last checked" : 0,
"DNS field" : 1, # only for CVDs
"local version" : 0, # only for CVDs
"CDIFFs" : [] # only for CVDs
},
"daily.cvd" : {
"url" : "https://database.clamav.net/daily.cvd",
"retry after" : 0,
"last modified" : 0,
"last checked" : 0,
"DNS field" : 2,
"local version" : 0,
"CDIFFs" : []
},
"bytecode.cvd" : {
"url" : "https://database.clamav.net/bytecode.cvd",
"retry after" : 0,
"last modified" : 0,
"last checked" : 0,
"DNS field" : 7,
"local version" : 0,
"CDIFFs" : []
},
},
}
config_path: Path
config: dict
state: dict
db_dir: Path
log_dir: Path
version: str
def __init__(
self,
config: str = "",
log_dir: str = "",
db_dir: str = "",
nameserver: str = "",
verbose: bool = False,
) -> None:
"""
CVDUpdate class.
Args:
log_dir: path output log.
db_dir: path where databases will be downloaded.
verbose: Enable DEBUG-level logs and other verbose messages.
"""
self.version = pkg_resources.get_distribution('cvdupdate').version
self.verbose = verbose
self._read_config(
config,
db_dir,
log_dir,
nameserver)
self._init_logging()
def _init_logging(self) -> None:
"""
Initializes the logging parameters.
"""
self.logger = logging.getLogger(f"cvdupdate-{self.version}")
if self.verbose:
self.logger.setLevel(logging.DEBUG)
else:
self.logger.setLevel(logging.INFO)
formatter = logging.Formatter(
fmt="%(asctime)s - %(levelname)s: %(message)s",
datefmt="%Y-%m-%d %I:%M:%S %p",
)
today = datetime.datetime.now()
self.log_file = self.log_dir / f"{today.strftime('%Y-%m-%d')}.log"
if not self.log_dir.exists():
# Make a new log directory
os.makedirs(os.path.split(self.log_file)[0])
else:
# Log dir already exists, lets check if we need to prune old logs
logs = self.log_dir.glob('*.log')
for log in logs:
log_date_str = str(log.stem)
log_date = datetime.datetime.strptime(log_date_str, "%Y-%m-%d")
if log_date + datetime.timedelta(days=self.config["# logs to keep"]) < today:
# Log is too old, delete!
os.remove(str(log))
self.filehandler = logging.FileHandler(filename=self.log_file)
self.filehandler.setLevel(self.logger.level)
self.filehandler.setFormatter(formatter)
self.logger.addHandler(self.filehandler)
# Also set the log level for urllib3, because it's "DEBUG" by default,
# and we may not like that.
urllib3_logger = logging.getLogger("urllib3.connectionpool")
urllib3_logger.setLevel(self.logger.level)
def _read_config(self,
config: str,
db_dir: str,
log_dir: str,
nameserver: str) -> None:
"""
Read in the config file.
Create a new one if one does not already exist.
"""
need_save = False
if config == "":
self.config_path = copy.deepcopy(self.default_config_path)
else:
self.config_path = Path(config)
if self.config_path.exists():
# Config already exists, load it.
with self.config_path.open('r') as config_file:
self.config = json.load(config_file)
else:
# Config does not exist, use default
self.config = copy.deepcopy(self.default_config)
need_save = True
if db_dir != "":
self.config["db directory"] = db_dir
need_save = True
self.db_dir = Path(self.config["db directory"])
if log_dir != "":
self.config["log directory"] = log_dir
need_save = True
self.log_dir = Path(self.config["log directory"])
if nameserver != "":
self.config['nameserver'] = nameserver
need_save = True
# For backwards compatibility with older configs.
if 'nameserver' not in self.config:
self.config['nameserver'] = ""
need_save = True
if 'max retry' not in self.config:
self.config['max retry'] = 3
need_save = True
if not hasattr(self, 'state'):
self.state = {}
# keep database state in a separate file, defaulting to same dir as config file
if 'state file' not in self.config or self.config['state file'] == '':
self.config['state file'] = str(self.config_path.parent / "state.json")
need_save = True
# handle migration from config.json to state.json
if 'dbs' in self.config:
self.state['dbs'] = self.config['dbs']
del self.config['dbs']
if 'uuid' in self.config:
self.state['uuid'] = self.config['uuid']
del self.config['uuid']
state_file = Path(self.config['state file'])
if state_file.exists():
# state file exists, load it.
with state_file.open('r') as st_fi:
self.state = json.load(st_fi)
elif self.state == {} or 'dbs' not in self.state:
# state file does not exist
# so we either have a fresh install or we have a messed up json
# create a skeleton structure
self.state = copy.deepcopy(self.default_state)
need_save = True
if 'uuid' not in self.state:
# Create a UUID to put in our User-Agent for better (anonymous) metrics
self.state['uuid'] = str(uuid.uuid4())
need_save = True
if need_save:
self._save_config()
def _save_config(self) -> None:
"""
Save the current configuration.
"""
for fi in (self.config_path, Path(self.config['state file'])):
if not fi.parent.exists():
# parent directory doesn't exist yet
try:
os.makedirs(str(fi.parent))
except Exception as exc:
print("Failed to create config directory!")
raise exc
try:
with self.config_path.open('w') as config_file:
json.dump(self.config, config_file, indent=4)
except Exception as exc:
print("Failed to create config file!")
raise exc
try:
with open(self.config['state file'], 'w') as state_file:
json.dump(self.state, state_file, indent=4)
except Exception as exc:
print("Failed to create state file!")
raise exc
if self.verbose:
print(f"Saved: {self.config_path}\n")
print(f"Saved: {self.config['state file']}\n")
def config_show(self):
"""
Print out the config
"""
print(f"Config file: {self.config_path}\n")
print(f"Config:\n{json.dumps(self.config, indent=4)}\n")
print(f"State file: {self.config['state file']}\n")
print(f"State:\n{json.dumps(self.state, indent=4)}\n")
def update(self, db: str = "") -> bool:
"""
Update a specific database or all the databases.
"""
def clean_dbs(self):
"""
Delete cvd controlled files in the database directory.
"""
dbs = self.state['dbs'].keys()
for db in dbs:
cvddb = self.db_dir / db
if cvddb.exists():
try:
self.logger.info(f"Deleting: {db}")
os.remove(str(cvddb))
except Exception as exc:
self.logger.debug(f"Tried to remove {db}")
raise exc
# Remove / clear all CDIFFs
cdiff_files = self.db_dir.glob('*.cdiff')
for cdiff in cdiff_files:
try:
self.logger.info(f"Deleting CDIFF: {cdiff.name}")
os.remove(str(cdiff))
except Exception as exc:
self.logger.debug(f"Tried to remove cdiffs.")
# Config cleanup
for db in dbs:
self.state['dbs'][db]['CDIFFs'] = []
self.state['dbs'][db]['last modified'] = 0
self.state['dbs'][db]['last checked'] = 0
self.state['dbs'][db]['local version'] = 0
# Save config
self._save_config()
def clean_logs(self):
"""
Delete all files in the log directory.
"""
self.logger.info(f"Deleting log files...")
logs = self.log_dir.glob('*')
for log in logs:
os.remove(str(log))
print(f"Deleted: {log}")
def clean_all(self):
"""
Delete all logs and databases and the config.
"""
self.clean_dbs()
self.clean_logs()
os.remove(str(self.config_path))
print(f"Deleted: {self.config_path}")
os.remove(str(self.config['state file']))
print(f"Deleted: {self.config['state file']}")
def _index_local_databases(self) -> dict:
need_save = False
dbs = copy.deepcopy(self.state['dbs'])
db_paths = self.db_dir.glob('*')
for db in db_paths:
if db.name.endswith('.cdiff'):
# Ignore CDIFFs, they'll get printed later.
continue
if db.name not in dbs:
version = 0
# Found a file in here that ISN'T a part of the config
if db.name.endswith('.cvd'):
# Found a CVD in here that ISN'T a part of the config!
# Very odd BTW.
self.logger.warning(f"Found a CVD in the DB directory that isn't in the config: {db.name}")
try:
version = self._get_cvd_version_from_file(db)
except Exception as exc:
self.logger.debug(f"EXCEPTION OCCURRED: {exc}")
self.logger.error(f"Failed to determine version for {db.name}")
dbs[db.name] = {
"url" : "n/a",
"retry after" : 0,
"last modified" : os.path.getmtime(str(db)),
"last checked" : 0,
"DNS field" : 0,
"local version" : version,
"CDIFFs" : []
}
else:
# DB on disk is from our config
if db.name.endswith(".cvd") and self.state['dbs'][db.name]['local version'] == 0:
# Seems like we somehow got a (config'd) CVD file in our database directory without
# saving the CVD info to the config. Let's just update the version field.
self.logger.info(f"Found {db.name} in the DB directory, though it wasn't downloaded using this tool.")
try:
dbs[db.name]['local version'] = self._get_cvd_version_from_file(self.db_dir / db.name)
self.logger.info(f"Identified mysterious {db.name} version: {dbs[db.name]['local version']}")
# Add the version info for this mysteriously deposited CVD to our config.
self.state['dbs'][db.name]['local version'] = dbs[db.name]['local version']
need_save = True
except Exception as exc:
self.logger.debug(f"EXCEPTION OCCURRED: {exc}")
self.logger.error(f"Failed to determine version # of mysterious {db.name} file. Perhaps it is corrupted?")
if need_save:
self._save_config()
return dbs
def db_list(self) -> None:
"""
Print list of databases
"""
dbs = self._index_local_databases()
for db in dbs:
updated = datetime.datetime.fromtimestamp(dbs[db]['last modified']).strftime('%Y-%m-%d %H:%M:%S')
checked = datetime.datetime.fromtimestamp(dbs[db]['last checked']).strftime('%Y-%m-%d %H:%M:%S')
self.logger.info(f"Database: {db}")
if dbs[db]['last modified'] == 0:
self.logger.info(" last modified: not downloaded")
else:
self.logger.info(f" last modified: {updated}")
if dbs[db]['last checked'] == 0:
self.logger.debug(" last checked: n/a")
else:
self.logger.debug(f" last checked: {checked}")
self.logger.debug(f" url: {dbs[db]['url']}")
if db.endswith(".cvd"):
# Only CVD's have versions.
self.logger.debug(f" local version: {dbs[db]['local version']}")
if len(dbs[db]['CDIFFs']) > 0:
self.logger.debug(f" CDIFFs:")
for cdiff in dbs[db]['CDIFFs']:
self.logger.debug(f" {cdiff}")
def db_show(self, name) -> bool:
"""
Show details for a specific database
"""
found = False
dbs = self._index_local_databases()
for db in dbs:
if db == name:
found = True;
updated = datetime.datetime.fromtimestamp(dbs[db]['last modified']).strftime('%Y-%m-%d %H:%M:%S')
checked = datetime.datetime.fromtimestamp(dbs[db]['last checked']).strftime('%Y-%m-%d %H:%M:%S')
self.logger.info(f"Database: {db}")
if dbs[db]['last modified'] == 0:
self.logger.info(" last modified: not downloaded")
else:
self.logger.info(f" last modified: {updated}")
if dbs[db]['last checked'] == 0:
self.logger.info(" last checked: n/a")
else:
self.logger.info(f" last checked: {checked}")
self.logger.info(f" url: {dbs[db]['url']}")
if db.endswith(".cvd"):
self.logger.info(f" local version: {dbs[db]['local version']}")
if len(dbs[db]['CDIFFs']) > 0:
self.logger.info(f" CDIFFs: \n{json.dumps(dbs[db]['CDIFFs'], indent=4)}")
return True
if not found:
self.logger.error(f"No such database: {name}")
return found
def _query_dns_txt_entry(self) -> bool:
'''
Attempt to get version from current.cvd.clamav.net DNS TXT entry
'''
got_it = False
self.logger.debug(f"Checking available versions via DNS TXT entry query of current.cvd.clamav.net")
try:
our_resolver = resolver.Resolver()
our_resolver.timeout = 5 # Explicitly setting query timeout to mitigate https://github.com/Cisco-Talos/cvdupdate/issues/17
nameservers = self._get_nameserver_configuration()
if nameservers:
our_resolver.nameservers = nameservers
self.logger.info(f"Using nameservers: {nameservers}")
else:
self.logger.info("Using system configured nameservers")
answer = str(our_resolver.resolve("current.cvd.clamav.net","TXT").response.answer[0])
versions = re.search('".*"', answer).group().strip('"')
self.dns_version_tokens = versions.split(':')
got_it = True
except Exception as exc:
self.logger.debug(f"EXCEPTION OCCURRED: {exc}")
self.logger.warning(f"Failed to determine available version via DNS TXT query!")
return got_it
def _get_nameserver_configuration(self) -> List[str]:
'''
Parse comma delimited nameserver string into a list for Resolver
'''
nameserver_string = self._get_nameserver_string()
nameservers = []
os_platform = platform.platform()
operating_system = os_platform.split("-")[0].lower()
if nameserver_string != "":
try:
nameservers = [x.strip() for x in nameserver_string.split(',')]
except Exception as exc:
self.logger.warning(f"Failed to parse nameserver configuration: {nameserver_string}, ignoring...")
nameservers = []
if nameservers == [] and operating_system == "windows":
# Try OpenDNS nameservers on Windows if not overridden by the user, because leaving it empty seems to fail.
nameservers = ['208.67.222.222', '208.67.220.220']
return nameservers
def _get_nameserver_string(self) -> str:
'''
Determine user provided nameserver configuration
'''
config_nameserver = self.config['nameserver']
env_nameserver = os.environ.get("CVDUPDATE_NAMESERVER")
# Environment variable overrides the configuration setting
if env_nameserver != None and env_nameserver != "":
self.logger.info(f"Found CVDUPDATE_NAMESERVER environment variable to provide nameservers: {env_nameserver}")
return env_nameserver
elif config_nameserver != None and config_nameserver != "":
self.logger.info(f"Found configuration provided nameservers: {config_nameserver}")
return config_nameserver
return ""
def _query_cvd_version_dns(self, db: str) -> int:
'''
This is a faux query.
Try to look up the version # from the DNS TXT entry we already have.
'''
version = 0
if self.dns_version_tokens == []:
# Query DNS if we haven't already
self._query_dns_txt_entry()
if self.dns_version_tokens == []:
# Query failed. Bail out.
return version
self.logger.debug(f"Checking {db} version via DNS TXT advertisement.")
if self.state['dbs'][db]['DNS field'] == 0:
# Invalid DNS field value for database version.
self.logger.warning(f"Failed to get DB version from DNS TXT entry: Invalid DNS field value for database version.")
return version
try:
version = int(self.dns_version_tokens[self.state['dbs'][db]['DNS field']])
self.logger.debug(f"{db} version advertised by DNS: {version}")
# Update the "last checked" time.
self.state['dbs'][db]['last checked'] = time.time()
except Exception as exc:
self.logger.debug(f"EXCEPTION OCCURRED: {exc}")
self.logger.warning(f"Failed to get DB version from DNS TXT entry!")
return version
def _query_cvd_version_http(self, db: str) -> int:
'''
Download the CVD header and read the CVD version
Return 1+ if queried version.
Return 0 if failed.
'''
version = 0
retry = 0
url = self.state['dbs'][db]['url']
self.logger.debug(f"Checking {db} version via HTTP download of CVD header.")
ims = datetime.datetime.utcfromtimestamp(self.state['dbs'][db]['last modified']).strftime('%a, %d %b %Y %H:%M:%S GMT')
while retry < self.config['max retry']:
response = requests.get(url, headers = {
'User-Agent': f'CVDUPDATE/{self.version} ({self.state["uuid"]})',
'Range': 'bytes=0-95',
'If-Modified-Since': ims,
})
if ((response.status_code == 200 or response.status_code == 206) and
('content-length' in response.headers) and
(int(response.headers['content-length']) > len(response.content))):
self.logger.warning(f"Response was truncated somehow...")
self.logger.warning(f" Expected {response.headers['content-length']}")
self.logger.warning(f" Received {response.content}, let's retry.")
retry += 1
else:
break
if response.status_code == 200 or response.status_code == 206:
# Looks like we downloaded something...
if (('content-length' in response.headers) and int(response.headers['content-length']) > len(response.content)):
self.logger.error(f"Failed to download {db} header to check the version #.")
return 0
# Successfully downloaded the header.
# We used the IMS header so this means it's probably newer, but we'll check just in case.
cvd_header = response.content
version = self._get_version_from_cvd_header(cvd_header)
self.logger.debug(f"{db} version available by HTTP download: {version}")
elif response.status_code == 304:
# HTTP Not-Modified, it's not newer.than what we already have.
# Just return the current local version.
version = self.state['dbs'][db]['local version']
self.logger.debug(f"{db} not-modified since: {ims} (local version {version})")
elif response.status_code == 429:
# Rejected because downloading the same file too frequently.
self.logger.warning(f"Failed to download {db} header to check the version #.")
self.logger.warning(f"Download request rejected because we've downloaded the same file too frequently.")
try_again_seconds = 60 * 60 * 12 # 12 hours
if 'Retry-After' in response.headers.keys():
try_again_seconds = int(response.headers['Retry-After'])
self.state['dbs'][db]['retry after'] = time.time() + float(try_again_seconds)
try_again_string = str(datetime.timedelta(seconds=try_again_seconds))
self.logger.warning(f"We won't try {db} again for {try_again_string} hours.")
else:
# Check failed!
self.logger.error(f"Failed to download {db} header to check the version #. Url: {url}")
if version > 0:
# Update the "last checked" time.
self.state['dbs'][db]['last checked'] = time.time()
return version
def _download_db_from_url(self, db: str, url: str, last_modified: int, version=0) -> CvdStatus:
'''
Download contents from a url and save to a filename in the database directory.
Will use If-Modified-Since
If Not-Modified, it will not replace the current database.
'''
retry = 0
ims: str = datetime.datetime.utcfromtimestamp(last_modified).strftime('%a, %d %b %Y %H:%M:%S GMT')
while retry < self.config['max retry']:
response = requests.get(url, headers = {
'User-Agent': f'CVDUPDATE/{self.version} ({self.state["uuid"]})',
'If-Modified-Since': ims,
})
if ((response.status_code == 200 or response.status_code == 206) and
('content-length' in response.headers) and
(int(response.headers['content-length']) > len(response.content))):
self.logger.warning(f"Response was truncated somehow...")
self.logger.warning(f" Expected {response.headers['content-length']}")
self.logger.warning(f" Received {response.content}, let's retry.")
retry += 1
else:
break
if response.status_code == 200:
# Looks like we downloaded something...
if (('content-length' in response.headers) and int(response.headers['content-length']) > len(response.content)):
self.logger.error(f"Failed to download {db}")
return CvdStatus.ERROR
# Download Success
if version > 0:
self.logger.info(f"Downloaded {db}. Version: {version}")
else:
self.logger.info(f"Downloaded {db}")
try:
with (self.db_dir / db).open('wb') as new_db:
new_db.write(response.content)
# Update config w/ new db info
self.state['dbs'][db]['last modified'] = time.time()
if db.endswith('.cvd'):
self.state['dbs'][db]['local version'] = self._get_version_from_cvd_header(response.content[:96])
except Exception as exc:
self.logger.debug(f"EXCEPTION OCCURRED: {exc}")
self.logger.error(f"Failed to save {db} to {self.db_dir}")
return CvdStatus.ERROR
elif response.status_code == 304:
# Not modified since IMS. We have the latest version.
version = self.state['dbs'][db]['local version']
self.logger.info(f"{db} not-modified since: {ims} (local version {version})")
return CvdStatus.NO_UPDATE
elif response.status_code == 429:
# Rejected because downloading the same file too frequently.
self.logger.warning(f"Failed to download {db}.")
self.logger.warning(f"Download request rejected because we've downloaded the same file too frequently.")
try_again_seconds = 60 * 60 * 12 # 12 hours
if 'Retry-After' in response.headers.keys():
try_again_seconds = int(response.headers['Retry-After'])
self.state['dbs'][db]['retry after'] = time.time() + float(try_again_seconds)
try_again_string = str(datetime.timedelta(seconds=try_again_seconds))
self.logger.warning(f"We won't try {db} again for {try_again_string} hours.")
# We'll have to retry after the cooldown.
return CvdStatus.ERROR
else:
# HTTP Get failed.
self.logger.error(f"Failed to download {db} from {url}")
return CvdStatus.ERROR
return CvdStatus.UPDATED
def _download_cvd(self, db: str, available_version: int) -> CvdStatus:
'''
Download the latest available version
If we already have some version of the database, attempt to download all CDIFFs in between.
If we don't, just get the last two CDIFFs.
'''
local_version = self.state['dbs'][db]['local version']
desired_version = local_version + 1
if local_version >= available_version:
# Oh! We're already up to date, don't worry about it.
self.logger.info(f"{db} is up-to-date. Version: {local_version}")
return CvdStatus.NO_UPDATE
elif local_version == 0:
# We don't have any version of the DB, let's just get the newest version + the last CDIFF
desired_version = available_version
# First try to get CDIFFs
self.logger.debug(f"Downloading CDIFFs first...")
while desired_version <= available_version:
# Attempt to download each CDIFF betwen our local version and the available version
# The url for CVDs should be https://database.clamav.net/
# Eg:
# https://database.clamav.net/daily.cvd
# For the daily cdiffs, we would want:
# https://database.clamav.net/daily-.cdiff
retry = 0
cdiff_filename = f"{db[:-len('.cvd')]}-{desired_version}.cdiff"
original_url = self.state['dbs'][db]['url']
url = f"{original_url[:-len(db)]}{cdiff_filename}"
if (self.db_dir / cdiff_filename).exists():
self.logger.debug(f"We already have {cdiff_filename}. Skipping...")
desired_version += 1
continue
self.logger.debug(f"Checking for {cdiff_filename}")
while retry < self.config['max retry']:
response = requests.get(url, headers = {
'User-Agent': f'CVDUPDATE/{self.version} ({self.state["uuid"]})',
})
if ((response.status_code == 200 or response.status_code == 206) and
('content-length' in response.headers) and
(int(response.headers['content-length']) > len(response.content))):
self.logger.warning(f"Response was truncated somehow...")
self.logger.warning(f" Expected {response.headers['content-length']}")
self.logger.warning(f" Received {response.content}, let's retry.")
retry += 1
else:
break
if response.status_code == 200:
# Looks like we downloaded something...
if (('content-length' in response.headers) and int(response.headers['content-length']) > len(response.content)):
self.logger.error(f"Failed to download {db} header to check the version #.")
return CvdStatus.ERROR
# Download Success
self.logger.info(f"Downloaded {cdiff_filename}")
try:
with (self.db_dir / f"{cdiff_filename}").open('wb') as new_db:
new_db.write(response.content)
except Exception as exc:
self.logger.debug(f"EXCEPTION OCCURRED: {exc}")
self.logger.error(f"Failed to save {cdiff_filename} to {self.db_dir}.")
# Update config with CDIFF, for posterity
self.state['dbs'][db]['CDIFFs'].append(cdiff_filename)
# Prune old CDIFFs if needed
if len(self.state['dbs'][db]['CDIFFs']) > self.config['# cdiffs to keep']:
try:
os.remove(self.db_dir / self.state['dbs'][db]['CDIFFs'][0])
except Exception as exc:
self.logger.debug(f"EXCEPTION OCCURRED: {exc}")
self.logger.debug(f"Tried to prune old cdiffs, but they weren't found, maybe someone else removed them already.")
self.state['dbs'][db]['CDIFFs'] = self.state['dbs'][db]['CDIFFs'][1:]
elif response.status_code == 429:
# Rejected because downloading the same file too frequently.
self.logger.warning(f"Failed to download {cdiff_filename}")
self.logger.warning(f"Download request rejected because we've downloaded the same file too frequently.")
try_again_seconds = 60 * 60 * 12 # 12 hours
if 'Retry-After' in response.headers.keys():
try_again_seconds = int(response.headers['Retry-After'])
self.state['dbs'][db]['retry after'] = time.time() + float(try_again_seconds)
try_again_string = str(datetime.timedelta(seconds=try_again_seconds))
self.logger.warning(f"We won't try {db} again for {try_again_string} hours.")
# Sure only a CDIFF failed, but if we want any chance of trying the CDIFF again
# in the future, let's bail out now and retry the CVD + CDIFFs after the cooldown.
return CvdStatus.ERROR
else:
# HTTP Get failed.
self.logger.info(f"No CDIFF found for {db} version # {desired_version}")
if desired_version < available_version:
desired_version = available_version - 1
self.logger.info(f"Will just skip to the last CDIFF instead.")
else:
self.logger.info(f"Giving up on CDIFFs for {db}")
break
desired_version += 1
# Now download the available version.
desired_version = available_version
url = f"{self.state['dbs'][db]['url']}?version={desired_version}"
return self._download_db_from_url(db, url, last_modified=0, version=desired_version)
def _get_version_from_cvd_header(self, cvd_header: bytes) -> int:
'''
Parse a CVD header to read the database version.
'''
header_fields = cvd_header.decode('utf-8', 'ignore').strip().split(':')
version_found = 0
try:
version_found = int(header_fields[2])
except Exception as exc:
self.logger.debug(f"EXCEPTION OCCURRED: {exc}")
self.logger.error(f"Failed to determine version from CVD header!")
return version_found
def _get_cvd_version_from_file(self, path: Path) -> int:
cvd_header: bytes = b''
version_found = 0
try:
with path.open('rb') as cvd_fd:
cvd_header = cvd_fd.read(96)
if len(cvd_header) < 96:
# Most likely a corrupted CVD. Delete.
self.logger.debug(f"Failed to read CVD header, perhaps {path.name} is corrupted.")
self.logger.debug(f"Will delete {path.name} so it will not cause further problems.")
os.remove(str(path))
else:
# Got the header, lets parse out the version.
version_found = self._get_version_from_cvd_header(cvd_header)
except Exception as exc:
self.logger.debug(f"EXCEPTION OCCURRED: {exc}")
self.logger.error(f"Failed to read version from CVD header from {path}.")
if version_found == 0:
self.logger.error(f"Failed to determine version from CVD header.")
return version_found
def pypi_update_check(self):
def check(name):
'''
Check if there's a newer version of the cvdupdate package.
From https://stackoverflow.com/questions/58648739/how-to-check-if-python-package-is-latest-version-programmatically
'''
self.logger.debug(f'Checking for a newer version of cvdupdate.')
result = subprocess.run([sys.executable, '-m', 'pip', 'install', '{}==random'.format('cvdupdate')], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
latest_version = result.stderr.decode("utf-8")
latest_version = latest_version[latest_version.find('(from versions:')+15:]
latest_version = latest_version[:latest_version.find(')')]
latest_version = latest_version.replace(' ','').split(',')[-1].strip()
result = subprocess.run([sys.executable, '-m', 'pip', 'show', '{}'.format('cvdupdate')], stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
current_version = result.stdout.decode("utf-8")
current_version = current_version[current_version.find('Version:')+8:]
current_version = current_version[:current_version.find('\n')].replace(' ','').strip()
if 'ERROR' in latest_version:
self.logger.debug(f"Version check didn't work, didn't get back a list of package versions.")
return True
elif latest_version == current_version:
self.logger.debug(f'cvdupdate is up-to-date: {current_version}.')
return True
else:
self.logger.warning(f'You are running cvdupdate version: {current_version}.')
self.logger.warning(f'There is a newer version on PyPI: {latest_version}. Please update!')
return False
return check('cvdupdate')
def db_update(self, db="", debug_mode=False) -> int:
"""
Update one or all of the databases.
Returns: Number of errors.
"""
self.update_errors = 0
self.dbs_updated = 0
self.dns_version_tokens = []
# Make sure we have a database directory to save files to
if not self.db_dir.exists():
os.makedirs(self.db_dir)
# Check if there is a newer version of CVD-Update
self.pypi_update_check()
# Query DNS so we can efficiently query CVD version #'s
self._query_dns_txt_entry()
if self.dns_version_tokens == []:
# Query failed. Bail out.
self.logger.error(f"Failed to update: DNS query failed.")
return 1
if debug_mode:
http_client.HTTPConnection.debuglevel = 1
def update(db) -> CvdStatus:
'''
Update a database
'''
if self.state['dbs'][db]['retry after'] > 0:
cooldown_date = datetime.datetime.fromtimestamp(self.state['dbs'][db]['retry after']).strftime('%Y-%m-%d %H:%M:%S')
if self.state['dbs'][db]['retry after'] > time.time():
self.logger.warning(f"Skipping {db} which is on cooldown until {cooldown_date}")
return CvdStatus.ERROR
else:
# Cooldown expired. Ok to try again.
self.state['dbs'][db]['retry after'] = 0
self.logger.info(f"{db} cooldown expired {cooldown_date}. OK to try again...")
if not self.state['dbs'][db]['url'].startswith('http'):
self.logger.error(f"Failed to update {db}. Missing or invalid URL: {self.state['dbs'][db]['url']}")
return CvdStatus.ERROR
self.logger.debug(f"Checking {db} for update from {self.state['dbs'][db]['url']}")
if db.endswith('.cvd'):
# It's a CVD (official signed clamav database)
advertised_version = 0
if (self.db_dir / db).exists():
if self.state['dbs'][db]['local version'] == 0:
# Seems like we somehow got a CVD in our database directory without
# saving the CVD info to the config. Let's just update the version field.
self.state['dbs'][db]['local version'] = self._get_cvd_version_from_file(self.db_dir / db)
else:
if self.state['dbs'][db]['local version'] != 0:
# We have a local version but no CVD in the database directory.
# Maybe it was moved or deleted? Let's just reset the local version.
self.state['dbs'][db]['local version'] = 0
if self.state['dbs'][db]['DNS field'] > 0:
# We can use the DNS TXT fields to check if our version is old.
advertised_version = self._query_cvd_version_dns(db)
else:
# We can't use DNS to see if our version is old.
# Use HTTP to pull just the CVD header to check.
# First, make sure no one tampered with the DNS field for
# main/daily/bytecode when using database.clamav.net
if (('database.clamav.net' in self.state['dbs'][db]['url']) and
(db == 'main.cvd' or db == 'daily.cvd' or db == 'bytecode.cvd')):
self.logger.error(f'It appears that the "DNS field" in {self.config_path} for "{db}" was modified from the default.')
self.logger.error(f'Updating {db} from database.clamav.net requires DNS for the version check in order to conserve bandwidth.')
self.logger.error(f'Please restore the default settings for the "DNS field" and try again.')
return CvdStatus.ERROR
advertised_version = self._query_cvd_version_http(db)
if advertised_version == 0:
self.logger.error(f"Failed to update {db}. Failed to query available CVD version")
return CvdStatus.ERROR
return self._download_cvd(db, advertised_version)
else:
# Try the download.
# Will use If-Modified-Since
# If Not-Modified, it will not replace the current database.
return self._download_db_from_url(
db,
self.state['dbs'][db]['url'],
self.state['dbs'][db]['last modified'])
if db == "":
# Update every DB.
for db in self.state['dbs']:
status = update(db)
if status == CvdStatus.ERROR:
self.update_errors += 1
elif status == CvdStatus.UPDATED:
self.dbs_updated += 1
else:
# Update a specific DB.
if db not in self.state['dbs']:
self.logger.error(f"Update failed. Unknown database: {db}")
else:
status = update(db)
if status == CvdStatus.ERROR:
self.update_errors += 1
elif status == CvdStatus.UPDATED:
self.dbs_updated += 1
self._save_config()
if self.update_errors == 0 and self.dbs_updated > 0:
with (self.db_dir / 'dns.txt').open('w') as dns_file:
dns_file.write(':'.join(self.dns_version_tokens))
self.logger.debug(f"Updated {self.db_dir / 'dns.txt'}")
return self.update_errors
def config_add_db(self, db: str, url: str) -> bool:
"""
Add another database + url to check when we update.
"""
extension = db.split('.')[-1]
if extension not in [
'cvd', 'cld', 'cud',
'cfg', 'cat', 'crb',
'ftm',
'ndb', 'ndu',
'ldb', 'ldu', 'idb',
'ydb', 'yar', 'yara',
'cdb',
'cbc',
'pdb', 'gdb', 'wdb',
'hdb', 'hsb', 'hdu', 'hsu',
'mdb', 'msb', 'mdu', 'msu',
'ign', 'ign2',
'info',
]:
self.logger.warning(f"{db} does not have valid clamav database file extension.")
if db in self.state['dbs']:
self.logger.info(f"Cannot add {db}, it is already in our list.")
self.logger.info(f"Hint: Try `db list -V` or `db show {db}` for more information.")
return False
self.state['dbs'][db] = {
"url" : url,
"retry after" : 0,
"last modified" : 0,
"last checked" : 0,
"DNS field" : 0,
"local version" : 0,
"CDIFFs" : []
}
self.logger.info(f"Added {db} ({url}) to DB list.")
self.logger.info(f"{db} will be downloaded next time you run `cvd update` or `cvd update {db}`")
self._save_config()
return True
def config_remove_db(self, db: str) -> bool:
"""
Remove a database from our list, and delete copies of the DB from the database directory.
"""
if db not in self.state['dbs']:
self.logger.info(f"Cannot remove {db}, it is not in our list.")
self.logger.info(f"Hint: Try `db list -V` for more information.")
return False
try:
if (self.db_dir / db).exists():
os.remove(str(self.db_dir / db))
self.logger.info(f"Deleted {db} from database directory.")
except Exception as exc:
self.logger.debug(f"An exception occured: {exc}")
self.logger.error(f"Failed to delete {db} from databse directory!")
for cdiff in self.state['dbs'][db]['CDIFFs']:
try:
if (self.db_dir / cdiff).exists():
os.remove(str(self.db_dir / cdiff))
self.logger.info(f"Deleted {cdiff} from database directory.")
except Exception as exc:
self.logger.debug(f"An exception occured: {exc}")
self.logger.error(f"Failed to delete {cdiff} from databse directory!")
self.state['dbs'].pop(db)
self.logger.info(f"Removed {db} from DB list.")
self._save_config()
return True
cvdupdate-1.1.1/cvdupdate.egg-info/ 0000755 0001751 0000164 00000000000 14270561637 017756 5 ustar runner docker 0000000 0000000 cvdupdate-1.1.1/cvdupdate.egg-info/PKG-INFO 0000644 0001751 0000164 00000031642 14270561637 021061 0 ustar runner docker 0000000 0000000 Metadata-Version: 2.1
Name: cvdupdate
Version: 1.1.1
Summary: ClamAV Private Database Mirror Updater Tool
Home-page: https://github.com/Cisco-Talos/cvdupdate
Author: The ClamAV Team
Author-email: clamav-bugs@external.cisco.com
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Description-Content-Type: text/markdown
License-File: LICENSE
A tool to download and update clamav databases and database patch files
for the purposes of hosting your own database mirror.
Copyright (C) 2021-2022 Cisco Systems, Inc. and/or its affiliates. All rights reserved.
## About
This tool downloads the latest ClamAV databases along with the latest database patch files.
This project replaces the `clamdownloader.pl` Perl script by Frederic Vanden Poel, formerly provided here: https://www.clamav.net/documents/private-local-mirrors
Run this tool as often as you like, but it will only download new content if there is new content to download. If you somehow manage to download too frequently (eg: by using `cvd clean all` and `cvd update` repeatedly), then the official database server may refuse your download request, and one or more databases may go on cool-down until it's safe to try again.
## Requirements
- Python 3.6 or newer.
- An internet connection with DNS enabled.
- The following Python packages. These will be installed automatically if you use `pip`, but may need to be installed manually otherwise:
- `click` v7.0 or newer
- `coloredlogs` v10.0 or newer
- `colorama`
- `requests`
- `dnspython` v2.1.0 or newer
- `rangehttpserver`
## Installation
You may install `cvdupdate` from PyPI using `pip`, or you may clone the project Git repository and use `pip` to install it locally.
Install `cvdupdate` from PyPI:
```bash
python3 -m pip install --user cvdupdate
```
## Basic Usage
Use the `--help` option with any `cvd` command to get help.
```bash
cvd --help
```
> _Tip_: You may not be able to run the `cvd` or `cvdupdate` shortcut directly if your Python Scripts directory is not in your `PATH` environment variable. If you run into this issue, and do not wish to add the Python Scripts directory to your path, you can run CVD-Update like this:
>
> ```bash
> python -m cvdupdate --help
> ```
(optional) You may wish to customize where the databases are stored:
```bash
cvd config set --dbdir
```
Run this to download the latest database and associated CDIFF patch files:
```bash
cvd update
```
Downloaded databases will be placed in `~/.cvdupdate/database` unless you customized it to use a different directory.
Newly downloaded databases will replace the previous database version, but the CDIFF patch files will accumulate up to a configured maximum before it starts deleting old CDIFFs (default: 30 CDIFFs). You can configure it to keep more CDIFFs by manually editing the config (default: `~/.cvdupdate/config.json`). The same behavior applies for CVD-Update log rotation.
Run this to serve up the database directory on `http://localhost:8000` so you can test it with FreshClam.
```bash
cvd serve
```
> _Disclaimer_: The `cvd serve` feature is not intended for production use, just for testing. You probably want to use a more robust HTTP server for production work.
Install ClamAV if you don't already have it and, in another terminal window, modify your `freshclam.conf` file. Replace:
```
DatabaseMirror database.clamav.net
```
... with:
```
DatabaseMirror http://localhost:8000
```
> _Tip_: A default install on Linux/Unix places `freshclam.conf` in `/usr/local/etc/freshclam.conf`. If one does not exist, you may need to create it using `freshclam.conf.sample` as a template.
Now, run `freshclam -v` or `freshclam.exe -v` to see what happens. You should see FreshClam successfully update it's own database directory from your private database server.
Run `cvd update` as often as you need. Maybe put it in a `cron` job.
> _Tip_: Each command supports a `--verbose` (`-V`) mode, which often provides more details about what's going on under the hood.
### Cron Example
Cron is a popular choice to automate frequent tasks on Linux / Unix systems.
1. Open a terminal running as the user which you want CVD-Update to run under, do the following:
```bash
crontab -e
```
2. Press `i` to insert new text, and add this line:
```bash
30 */4 * * * /bin/sh -c "~/.local/bin/cvd update &> /dev/null"
```
Or instead of `~/`, you can do this, replacing `username` with your user name:
```bash
30 */4 * * * /bin/sh -c "/home/username/.local/bin/cvd update &> /dev/null"
```
3. Press , then type `:wq` and press to write the file to disk and quit.
**About these settings**:
I selected `30 */4 * * *` to run at minute 30 past every 4th hour. CVD-Update uses a DNS check to do version checks before it attempts to download any files, just like FreshClam. Running CVD-Update more than once a day should not be an issue.
CVD-Update will write logs to the `~/.cvdupdate/logs` directory, which is why I directed `stdout` and `stderr` to `/dev/null` instead of a log file. You can use the `cvd config set` command to customize the log directory if you like, or redirect `stdout` and `stderr` to a log file if you prefer everything in one log instead of separate daily logs.
## Optional Functionality
### Using a custom DNS server
DNS is required for CVD-Update to function properly (to gather the TXT record containing the current definition database version). You can select a specific nameserver to ensure said nameserver is used when querying the TXT record containing the current database definition version available
1. Set the nameserver in the config. Eg:
```bash
cvd config set --nameserver 208.67.222.222
```
2. Set the environment variable `CVDUPDATE_NAMESERVER`. Eg:
```bash
CVDUPDATE_NAMESERVER="208.67.222.222" cvd update
```
The environment variable will take precedence over the nameserver config setting.
Note: Both options can be used to provide a comma-delimited list of nameservers to utilize for resolution.
### Using a proxy
Depending on your type of proxy, you may be able to use CVD-Update with your proxy by running CVD-Update like this:
First, set a custom domain name server to use the proxy:
```bash
cvd config set --nameserver
```
Then run CVD-Update like this:
```bash
http_proxy=http://: https_proxy=http://: cvd update -V
```
Or create a script to wrap the CVD-Update call. Something like:
```bash
#!/bin/bash
http_proxy=http://:
export http_proxy
https_proxy=http://:
export https_proxy
cvd update -V
```
> _Disclaimer_: CVD-Update doesn't support proxies that require authentication at this time. If your network admin allows it, you may be able to work around it by updating your proxy to allow HTTP requests through unauthenticated if the User-Agent matches your specific CVD-Update user agent. The CVD-Update User-Agent follows the form `CVDUPDATE/ ()` where the `uuid` is unique to your installation and can be found in the `~/.cvdupdate/state.json` file (or `~/.cvdupdate/config.json` for cvdupdate <=1.0.2). See https://github.com/Cisco-Talos/cvdupdate/issues/9 for more details.
>
> Adding support for proxy authentication is a ripe opportunity for a community contribution to the project.
## Files and directories created by CVD-Update
This tool is to creates the following directories:
- `~/.cvdupdate`
- `~/.cvdupdate/logs`
- `~/.cvdupdate/databases`
This tool creates the following files:
- `~/.cvdupdate/config.json`
- `~/.cvdupdate/state.json`
- `~/.cvdupdate/databases/.cvd`
- `~/.cvdupdate/databases/-.cdiff`
- `~/.cvdupdate/logs/.log`
> _Tip_: You can set custom `database` and `logs` directories with the `cvd config set` command. It is likely you will want to customize the `database` directory to point to your HTTP server's `www` directory (or equivalent). Bare in mind that if you already downloaded the databases to the old directory, you may want to move them to the new directory.
> _Important_: If you want to use a custom config path, you'll have to use it in every command. If you're fine with having it go in `~/.cvdupdate/config.json`, don't worry about it.
## Additional Usage
### Get familiar with the tool
Familiarize yourself with the various commands using the `--help` option.
```bash
cvd --help
cvd config --help
cvd update --help
cvd clean --help
```
Print out the current list of databases.
```bash
cvd list -V
```
Print out the config to see what it looks like.
```bash
cvd config show
```
### Do an update
Do an update, use "verbose mode" to so you can get a feel for how it works.
```bash
cvd update -V
```
List out the databases again:
```bash
cvd list -V
```
The print out the config again so you can see what's changed.
```bash
cvd config show
```
And maybe take a peek in the database directory as well to see it for yourself.
```bash
ls ~/.cvdupdate/database
```
Have a look at the logs if you wish.
```bash
ls ~/.cvdupdate/logs
cat ~/.cvdupdate/logs/*
```
### Serve it up, Test out FreshClam
Test out your mirror with FreshClam on the same computer.
This tool includes a `--serve` feature that will host the current database directory on http://localhost (default port: 8000).
You can test it by running `freshclam` or `freshclam.exe` locally, where you've configured `freshclam.conf` with:
```
DatabaseMirror http://localhost:8000
```
## Contribute
We'd love your help. There are many ways to contribute!
### Community
Join the ClamAV community on the [ClamAV Discord chat server](https://discord.gg/sGaxA5Q).
### Report issues
If you find an issue with CVD-Update or the CVD-Update documentation, please submit an issue to our [GitHub issue tracker](https://github.com/Cisco-Talos/cvdupdate/issues). Before you submit, please check to if someone else has already reported the issue.
### Development
If you find a bug and you're able to craft a fix yourself, consider submitting the fix in a [pull request](https://github.com/Cisco-Talos/cvdupdate/pulls). Your help will be greatly appreciated.
If you want to contribute to the project and don't have anything specific in mind, please check out our [issue tracker](https://github.com/Cisco-Talos/cvdupdate/issues). Perhaps you'll be able to fix a bug or add a cool new feature.
_By submitting a contribution to the project, you acknowledge and agree to assign Cisco Systems, Inc the copyright for the contribution. If you submit a significant contribution such as a new feature or capability or a large amount of code, you may be asked to sign a contributors license agreement comfirming that Cisco will have copyright license and patent license and that you are authorized to contribute the code._
#### Development Set-up
The following steps are intended to help users that wish to contribute to development of the CVD-Update project get started.
1. Create a fork of the [CVD-Update git repository](https://github.com/Cisco-Talos/cvdupdate), and then clone your fork to a local directory.
For example:
```bash
git clone https://github.com//cvdupdate.git
```
2. Make sure CVD-Update is not already installed. If it is, remove it.
```bash
python3 -m pip uninstall cvdupdate
```
3. Use pip to install CVD-Update in "edit" mode.
```bash
python3 -m pip install -e --user ./cvdupdate
```
Once installed in "edit" mode, any changes you make to your clone of the CVD-Update code will be immediately usable simply by running the `cvdupdate` / `cvd` commands.
### Conduct
This project has not selected a specific Code-of-Conduct document at this time. However, contributors are expected to behave in professional and respectful manner. Disrespectful or inappropriate behavior will not be tolerated.
## License
CVD-Update is licensed under the Apache License, Version 2.0 (the "License"). You may not use the CVD-Update project except in compliance with the License.
A copy of the license is located [here](LICENSE), and is also available online at [apache.org](http://www.apache.org/licenses/LICENSE-2.0).
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
cvdupdate-1.1.1/cvdupdate.egg-info/SOURCES.txt 0000644 0001751 0000164 00000000573 14270561637 021647 0 ustar runner docker 0000000 0000000 LICENSE
README.md
setup.py
cvdupdate/__init__.py
cvdupdate/__main__.py
cvdupdate/auto_updater.py
cvdupdate/cvdupdate.py
cvdupdate.egg-info/PKG-INFO
cvdupdate.egg-info/SOURCES.txt
cvdupdate.egg-info/dependency_links.txt
cvdupdate.egg-info/entry_points.txt
cvdupdate.egg-info/requires.txt
cvdupdate.egg-info/top_level.txt
tests/__init__.py
tests/conftest.py
tests/test_cvdupdate.py cvdupdate-1.1.1/cvdupdate.egg-info/dependency_links.txt 0000644 0001751 0000164 00000000001 14270561637 024024 0 ustar runner docker 0000000 0000000
cvdupdate-1.1.1/cvdupdate.egg-info/entry_points.txt 0000644 0001751 0000164 00000000122 14270561637 023247 0 ustar runner docker 0000000 0000000 [console_scripts]
cvd = cvdupdate.__main__:cli
cvdupdate = cvdupdate.__main__:cli
cvdupdate-1.1.1/cvdupdate.egg-info/requires.txt 0000644 0001751 0000164 00000000120 14270561637 022347 0 ustar runner docker 0000000 0000000 click>=7.0
coloredlogs>=10.0
colorama
requests
dnspython>=2.1.0
rangehttpserver
cvdupdate-1.1.1/cvdupdate.egg-info/top_level.txt 0000644 0001751 0000164 00000000020 14270561637 022500 0 ustar runner docker 0000000 0000000 cvdupdate
tests
cvdupdate-1.1.1/setup.cfg 0000644 0001751 0000164 00000000046 14270561637 016126 0 ustar runner docker 0000000 0000000 [egg_info]
tag_build =
tag_date = 0
cvdupdate-1.1.1/setup.py 0000644 0001751 0000164 00000002105 14270561627 016014 0 ustar runner docker 0000000 0000000 import setuptools
with open("README.md", "r") as fh:
long_description = fh.read()
setuptools.setup(
name="cvdupdate",
version="1.1.1",
author="The ClamAV Team",
author_email="clamav-bugs@external.cisco.com",
copyright="Copyright (C) 2022 Cisco Systems, Inc. and/or its affiliates. All rights reserved.",
description="ClamAV Private Database Mirror Updater Tool",
long_description=long_description,
long_description_content_type="text/markdown",
url="https://github.com/Cisco-Talos/cvdupdate",
packages=setuptools.find_packages(),
entry_points={
"console_scripts": [
"cvdupdate = cvdupdate.__main__:cli",
"cvd = cvdupdate.__main__:cli",
]
},
install_requires=[
"click>=7.0",
"coloredlogs>=10.0",
"colorama",
"requests",
"dnspython>=2.1.0",
"rangehttpserver",
],
classifiers=[
"Programming Language :: Python :: 3",
"License :: OSI Approved :: Apache Software License",
"Operating System :: OS Independent",
],
)
cvdupdate-1.1.1/tests/ 0000755 0001751 0000164 00000000000 14270561637 015447 5 ustar runner docker 0000000 0000000 cvdupdate-1.1.1/tests/__init__.py 0000644 0001751 0000164 00000000000 14270561627 017545 0 ustar runner docker 0000000 0000000 cvdupdate-1.1.1/tests/conftest.py 0000644 0001751 0000164 00000001351 14270561627 017645 0 ustar runner docker 0000000 0000000 from textwrap import dedent
from glob import glob
from pathlib import Path
import pytest
# grab all files from fixtures/
pytest_plugins = [
f'fixtures.{fi.stem}' for fi in Path(__file__).glob("fixtures/*.py") if fi.stem != '__init__'
]
# prevent any test from running if the default .cvdupdate directory already exists
@pytest.fixture(scope='session', autouse=True)
def fail_if_cvdupdate_dir_exists():
defaultdir = Path.home() / '.cvdupdate'
if defaultdir.exists():
pytest.exit(dedent(f'''
Error: {defaultdir} exists.
Aborting tests to prevent losing actual cvdupdate data.
Ensure tests are not running against an actual cvdupdate install.
'''),
returncode=1
)
cvdupdate-1.1.1/tests/test_cvdupdate.py 0000644 0001751 0000164 00000010170 14270561627 021035 0 ustar runner docker 0000000 0000000 import json
from pathlib import Path
import shutil
from tests.fixtures.revert import revert_homedir
from cvdupdate.cvdupdate import CVDUpdate
def test_instantiation(revert_homedir):
c = CVDUpdate()
def test_alternate_config_locations(revert_homedir, tmp_path):
''' Test that we can save config and state files to alternative locations '''
# ensure we're starting with a clean slate
default_cvdupdate_dir = Path.home() / '.cvdupdate'
assert not default_cvdupdate_dir.exists()
# set the config file to be in pytests's /tmp/pytest-*
config_file_path = tmp_path / 'config.json'
c = CVDUpdate(config=config_file_path)
# verify the file is created and has default config data inside
assert config_file_path.exists()
txt = config_file_path.read_text()
assert txt
config_file_json = json.loads(txt)
assert config_file_json == c.config
# state file value will differ, so blank it out for comparing
c.config['state file'] = ''
assert c.config == c.default_config
# verify the state file is created and has default data inside
state_file_path = tmp_path / 'state.json'
assert state_file_path.exists()
txt = state_file_path.read_text()
assert txt
state_file_json = json.loads(txt)
assert state_file_json == c.state
# again, uuid will differ, so toss it out
del c.state['uuid']
assert c.state == c.default_state
# ~/.cvdupdate exists, because we haven't changed the logdir location
# but that's all it should have in it
default_cvdupdate_dir = Path.home() / '.cvdupdate'
assert default_cvdupdate_dir.exists()
children = list(default_cvdupdate_dir.iterdir())
assert len(children) == 1
assert children[0] == default_cvdupdate_dir / 'logs'
def test_default_config_not_mutated(revert_homedir, tmp_path):
''' default_config and default_state are both class-level attributes
Ensure that when we copy these, we are actually copying them and not simply reassigning
note that this typically won't be a problem in normal usage,
but it was a problem during testing and was really annoying to track down
'''
a = CVDUpdate()
config_file_path = tmp_path / 'config.json'
# set the config file to be in pytests /tmp/pytest-*
b = CVDUpdate(config=config_file_path)
assert all(val == b.config[key] for key,val in a.config.items() if key != 'state file')
assert id(a.config) != id(b.config)
assert id(a.default_config) == id(b.default_config) == id(CVDUpdate.default_config)
assert a.state != b.state
assert id(a.default_state) == id(b.default_state)
def test_existing_state_migrates_successfully(revert_homedir):
''' specifically test migrating an existing config.json to config + state.json'''
default_cvdupdate_dir = Path.home() / '.cvdupdate'
default_cvdupdate_dir.mkdir(parents=True)
# create a .cvdupdate/config.json which also contains dbs definitions
old_config_file = 'tests/files/v1.0.2.config.json'
old_config_json = json.loads(Path(old_config_file).read_text())
old_config_json["log directory"] = str(default_cvdupdate_dir / 'logs')
old_config_json["db directory"] = str(default_cvdupdate_dir / 'database')
with (default_cvdupdate_dir / 'config.json').open('w') as test_config:
json.dump(old_config_json, test_config)
# create cvdupdate object, which will read config.json and split state into state.json
a = CVDUpdate()
new_config_json = old_config_json
# create expected state.json contents by copying the bits that move
new_state_json = {}
new_state_json['dbs'] = old_config_json['dbs']
new_state_json['uuid'] = old_config_json['uuid']
# new update the config.json contents
del new_config_json['dbs']
del new_config_json['uuid']
new_config_json['state file'] = str(default_cvdupdate_dir / 'state.json')
# compare actual result with expected transform
with open(default_cvdupdate_dir / 'config.json') as config:
assert new_config_json == json.loads(config.read())
with open(default_cvdupdate_dir / 'state.json') as state:
from pprint import pprint
assert new_state_json == json.loads(state.read())