cvdupdate-1.0.2/ 0000755 0001751 0000164 00000000000 14037577401 014302 5 ustar runner docker 0000000 0000000 cvdupdate-1.0.2/LICENSE 0000644 0001751 0000164 00000026135 14037577371 015324 0 ustar runner docker 0000000 0000000
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. cvdupdate-1.0.2/PKG-INFO 0000644 0001751 0000164 00000035756 14037577401 015417 0 ustar runner docker 0000000 0000000 Metadata-Version: 2.1
Name: cvdupdate
Version: 1.0.2
Summary: ClamAV Private Database Mirror Updater Tool
Home-page: https://github.com/Cisco-Talos/cvdupdate
Author: The ClamAV Team
Author-email: clamav-bugs@external.cisco.com
License: UNKNOWN
Description: A tool to download and update clamav databases and database patch files
for the purposes of hosting your own database mirror.
Copyright (C) 2021 Cisco Systems, Inc. and/or its affiliates. All rights reserved.
## About
This tool downloads the latest ClamAV databases along with the latest database patch files.
This project replaces the `clamdownloader.pl` Perl script by Frederic Vanden Poel, formerly provided here: https://www.clamav.net/documents/private-local-mirrors
Run this tool as often as you like, but it will only download new content if there is new content to download. If you somehow manage to download too frequently (eg: by using `cvd clean all` and `cvd update` repeatedly), then the official database server may refuse your download request, and one or more databases may go on cool-down until it's safe to try again.
## Requirements
- Python 3.6 or newer.
- An internet connection with DNS enabled.
## Installation
You may install `cvdupdate` from PyPI using `pip`, or you may clone the project Git repository and use `pip` to install it locally.
Install `cvdupdate` from PyPI:
```bash
python3 -m pip install --user cvdupdate
```
## Basic Usage
Use the `--help` option with any `cvd` command to get help.
```bash
cvd --help
```
> _Tip_: You may not be able to run the `cvd` or `cvdupdate` shortcut directly if your Python Scripts directory is not in your `PATH` environment variable. If you run into this issue, and do not wish to add the Python Scripts directory to your path, you can run CVD-Update like this:
>
> ```bash
> python -m cvdupdate --help
> ```
(optional) You may wish to customize where the databases are stored:
```bash
cvd config set --dbdir
```
Run this to download the latest database and associated CDIFF patch files:
```bash
cvd update
```
Downloaded databases will be placed in `~/.cvdupdate/database` unless you customized it to use a different directory.
Newly downloaded databases will replace the previous database version, but the CDIFF patch files will accumulate up to a configured maximum before it starts deleting old CDIFFs (default: 30 CDIFFs). You can configure it to keep more CDIFFs by manually editing the config (default: `~/.cvdupdate/config.json`). The same behavior applies for CVD-Update log rotation.
Run this to serve up the database directory on `http://localhost:8000` so you can test it with FreshClam.
```bash
cvd serve
```
> _Disclaimer_: The `cvd serve` feature is not intended for production use, just for testing. You probably want to use a more robust HTTP server for production work.
Install ClamAV if you don't already have it and, in another terminal window, modify your `freshclam.conf` file. Replace:
```
DatabaseMirror database.clamav.net
```
... with:
```
DatabaseMirror http://localhost:8000
```
> _Tip_: A default install on Linux/Unix places `freshclam.conf` in `/usr/local/etc/freshclam.conf`. If one does not exist, you may need to create it using `freshclam.conf.sample` as a template.
Now, run `freshclam -v` or `freshclam.exe -v` to see what happens. You should see FreshClam successfully update it's own database directory from your private database server.
Run `cvd update` as often as you need. Maybe put it in a `cron` job.
> _Tip_: Each command supports a `--verbose` (`-V`) mode, which often provides more details about what's going on under the hood.
### Cron Example
Cron is a popular choice to automate frequent tasks on Linux / Unix systems.
1. Open a terminal running as the user which you want CVD-Update to run under, do the following:
```bash
crontab -e
```
2. Press `i` to insert new text, and add this line:
```bash
30 */4 * * * /bin/sh -c "~/.local/bin/cvd update &> /dev/null"
```
Or instead of `~/`, you can do this, replacing `username` with your user name:
```bash
30 */4 * * * /bin/sh -c "/home/username/.local/bin/cvd update &> /dev/null"
```
3. Press , then type `:wq` and press to write the file to disk and quit.
**About these settings**:
I selected `30 */4 * * *` to run at minute 30 past every 4th hour. CVD-Update uses a DNS check to do version checks before it attempts to download any files, just like FreshClam. Running CVD-Update more than once a day should not be an issue.
CVD-Update will write logs to the `~/.cvdupdate/logs` directory, which is why I directed `stdout` and `stderr` to `/dev/null` instead of a log file. You can use the `cvd config set` command to customize the log directory if you like, or redirect `stdout` and `stderr` to a log file if you prefer everything in one log instead of separate daily logs.
## Optional Functionality
### Using a custom DNS server
DNS is required for CVD-Update to function properly (to gather the TXT record containing the current definition database version). You can select a specific nameserver to ensure said nameserver is used when querying the TXT record containing the current database definition version available
1. Set the nameserver in the config. Eg:
```bash
cvd config set --nameserver 208.67.222.222
```
2. Set the environment variable `CVDUPDATE_NAMESERVER`. Eg:
```bash
CVDUPDATE_NAMESERVER="208.67.222.222" cvd update
```
The environment variable will take precedence over the nameserver config setting.
### Using a proxy
Depending on your type of proxy, you may be able to use CVD-Update with your proxy by running CVD-Update like this:
First, set a custom domain name server to use the proxy:
```bash
cvd config set --nameserver
```
Then run CVD-Update like this:
```bash
http_proxy=http://: https_proxy=http://: cvd update -V
```
Or create a script to wrap the CVD-Update call. Something like:
```bash
#!/bin/bash
http_proxy=http://:
export http_proxy
https_proxy=http://:
export https_proxy
cvd update -V
```
> _Disclaimer_: CVD-Update doesn't support proxies that require authentication at this time. If your network admin allows it, you may be able to work around it by updating your proxy to allow HTTP requests through unauthenticated if the User-Agent matches your specific CVD-Update user agent. The CVD-Update User-Agent follows the form `CVDUPDATE/ ()` where the `uuid` is unique to your installation and can be found in the `~/.cvdupdate/config.json` file. See https://github.com/Cisco-Talos/cvdupdate/issues/9 for more details.
>
> Adding support for proxy authentication is a ripe opportunity for a community contribution to the project.
## Files and directories created by CVD-Update
This tool is to creates the following directories:
- `~/.cvdupdate`
- `~/.cvdupdate/logs`
- `~/.cvdupdate/databases`
This tool creates the following files:
- `~/.cvdupdate/config.json`
- `~/.cvdupdate/databases/.cvd`
- `~/.cvdupdate/databases/-.cdiff`
- `~/.cvdupdate/logs/.log`
> _Tip_: You can set custom `database` and `logs` directories with the `cvd config set` command. It is likely you will want to customize the `database` directory to point to your HTTP server's `www` directory (or equivalent). Bare in mind that if you already downloaded the databases to the old directory, you may want to move them to the new directory.
> _Important_: If you want to use a custom config path, you'll have to use it in every command. If you're fine with having it go in `~/.cvdupdate/config.json`, don't worry about it.
## Additional Usage
### Get familiar with the tool
Familiarize yourself with the various commands using the `--help` option.
```bash
cvd --help
cvd config --help
cvd update --help
cvd clean --help
```
Print out the current list of databases.
```bash
cvd list -V
```
Print out the config to see what it looks like.
```bash
cvd config show
```
### Do an update
Do an update, use "verbose mode" to so you can get a feel for how it works.
```bash
cvd update -V
```
List out the databases again:
```bash
cvd list -V
```
The print out the config again so you can see what's changed.
```bash
cvd config show
```
And maybe take a peek in the database directory as well to see it for yourself.
```bash
ls ~/.cvdupdate/database
```
Have a look at the logs if you wish.
```bash
ls ~/.cvdupdate/logs
cat ~/.cvdupdate/logs/*
```
### Serve it up, Test out FreshClam
Test out your mirror with FreshClam on the same computer.
This tool includes a `--serve` feature that will host the current database directory on http://localhost (default port: 8000).
You can test it by running `freshclam` or `freshclam.exe` locally, where you've configured `freshclam.conf` with:
```
DatabaseMirror http://localhost:8000
```
## Contribute
We'd love your help. There are many ways to contribute!
### Community
Join the ClamAV community on the [ClamAV Discord chat server](https://discord.gg/sGaxA5Q).
### Report issues
If you find an issue with CVD-Update or the CVD-Update documentation, please submit an issue to our [GitHub issue tracker](https://github.com/Cisco-Talos/cvdupdate/issues). Before you submit, please check to if someone else has already reported the issue.
### Development
If you find a bug and you're able to craft a fix yourself, consider submitting the fix in a [pull request](https://github.com/Cisco-Talos/cvdupdate/pulls). Your help will be greatly appreciated.
If you want to contribute to the project and don't have anything specific in mind, please check out our [issue tracker](https://github.com/Cisco-Talos/cvdupdate/issues). Perhaps you'll be able to fix a bug or add a cool new feature.
_By submitting a contribution to the project, you acknowledge and agree to assign Cisco Systems, Inc the copyright for the contribution. If you submit a significant contribution such as a new feature or capability or a large amount of code, you may be asked to sign a contributors license agreement comfirming that Cisco will have copyright license and patent license and that you are authorized to contribute the code._
#### Development Set-up
The following steps are intended to help users that wish to contribute to development of the CVD-Update project get started.
1. Create a fork of the [CVD-Update git repository](https://github.com/Cisco-Talos/cvdupdate), and then clone your fork to a local directory.
For example:
```bash
git clone https://github.com//cvdupdate.git
```
2. Make sure CVD-Update is not already installed. If it is, remove it.
```bash
python3 -m pip uninstall cvdupdate
```
3. Use pip to install CVD-Update in "edit" mode.
```bash
python3 -m pip install -e --user ./cvdupdate
```
Once installed in "edit" mode, any changes you make to your clone of the CVD-Update code will be immediately usable simply by running the `cvdupdate` / `cvd` commands.
### Conduct
This project has not selected a specific Code-of-Conduct document at this time. However, contributors are expected to behave in professional and respectful manner. Disrespectful or inappropriate behavior will not be tolerated.
## License
CVD-Update is licensed under the Apache License, Version 2.0 (the "License"). You may not use the CVD-Update project except in compliance with the License.
A copy of the license is located [here](LICENSE), and is also available online at [apache.org](http://www.apache.org/licenses/LICENSE-2.0).
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Description-Content-Type: text/markdown
cvdupdate-1.0.2/README.md 0000644 0001751 0000164 00000030004 14037577371 015564 0 ustar runner docker 0000000 0000000 A tool to download and update clamav databases and database patch files
for the purposes of hosting your own database mirror.
Copyright (C) 2021 Cisco Systems, Inc. and/or its affiliates. All rights reserved.
## About
This tool downloads the latest ClamAV databases along with the latest database patch files.
This project replaces the `clamdownloader.pl` Perl script by Frederic Vanden Poel, formerly provided here: https://www.clamav.net/documents/private-local-mirrors
Run this tool as often as you like, but it will only download new content if there is new content to download. If you somehow manage to download too frequently (eg: by using `cvd clean all` and `cvd update` repeatedly), then the official database server may refuse your download request, and one or more databases may go on cool-down until it's safe to try again.
## Requirements
- Python 3.6 or newer.
- An internet connection with DNS enabled.
## Installation
You may install `cvdupdate` from PyPI using `pip`, or you may clone the project Git repository and use `pip` to install it locally.
Install `cvdupdate` from PyPI:
```bash
python3 -m pip install --user cvdupdate
```
## Basic Usage
Use the `--help` option with any `cvd` command to get help.
```bash
cvd --help
```
> _Tip_: You may not be able to run the `cvd` or `cvdupdate` shortcut directly if your Python Scripts directory is not in your `PATH` environment variable. If you run into this issue, and do not wish to add the Python Scripts directory to your path, you can run CVD-Update like this:
>
> ```bash
> python -m cvdupdate --help
> ```
(optional) You may wish to customize where the databases are stored:
```bash
cvd config set --dbdir
```
Run this to download the latest database and associated CDIFF patch files:
```bash
cvd update
```
Downloaded databases will be placed in `~/.cvdupdate/database` unless you customized it to use a different directory.
Newly downloaded databases will replace the previous database version, but the CDIFF patch files will accumulate up to a configured maximum before it starts deleting old CDIFFs (default: 30 CDIFFs). You can configure it to keep more CDIFFs by manually editing the config (default: `~/.cvdupdate/config.json`). The same behavior applies for CVD-Update log rotation.
Run this to serve up the database directory on `http://localhost:8000` so you can test it with FreshClam.
```bash
cvd serve
```
> _Disclaimer_: The `cvd serve` feature is not intended for production use, just for testing. You probably want to use a more robust HTTP server for production work.
Install ClamAV if you don't already have it and, in another terminal window, modify your `freshclam.conf` file. Replace:
```
DatabaseMirror database.clamav.net
```
... with:
```
DatabaseMirror http://localhost:8000
```
> _Tip_: A default install on Linux/Unix places `freshclam.conf` in `/usr/local/etc/freshclam.conf`. If one does not exist, you may need to create it using `freshclam.conf.sample` as a template.
Now, run `freshclam -v` or `freshclam.exe -v` to see what happens. You should see FreshClam successfully update it's own database directory from your private database server.
Run `cvd update` as often as you need. Maybe put it in a `cron` job.
> _Tip_: Each command supports a `--verbose` (`-V`) mode, which often provides more details about what's going on under the hood.
### Cron Example
Cron is a popular choice to automate frequent tasks on Linux / Unix systems.
1. Open a terminal running as the user which you want CVD-Update to run under, do the following:
```bash
crontab -e
```
2. Press `i` to insert new text, and add this line:
```bash
30 */4 * * * /bin/sh -c "~/.local/bin/cvd update &> /dev/null"
```
Or instead of `~/`, you can do this, replacing `username` with your user name:
```bash
30 */4 * * * /bin/sh -c "/home/username/.local/bin/cvd update &> /dev/null"
```
3. Press , then type `:wq` and press to write the file to disk and quit.
**About these settings**:
I selected `30 */4 * * *` to run at minute 30 past every 4th hour. CVD-Update uses a DNS check to do version checks before it attempts to download any files, just like FreshClam. Running CVD-Update more than once a day should not be an issue.
CVD-Update will write logs to the `~/.cvdupdate/logs` directory, which is why I directed `stdout` and `stderr` to `/dev/null` instead of a log file. You can use the `cvd config set` command to customize the log directory if you like, or redirect `stdout` and `stderr` to a log file if you prefer everything in one log instead of separate daily logs.
## Optional Functionality
### Using a custom DNS server
DNS is required for CVD-Update to function properly (to gather the TXT record containing the current definition database version). You can select a specific nameserver to ensure said nameserver is used when querying the TXT record containing the current database definition version available
1. Set the nameserver in the config. Eg:
```bash
cvd config set --nameserver 208.67.222.222
```
2. Set the environment variable `CVDUPDATE_NAMESERVER`. Eg:
```bash
CVDUPDATE_NAMESERVER="208.67.222.222" cvd update
```
The environment variable will take precedence over the nameserver config setting.
### Using a proxy
Depending on your type of proxy, you may be able to use CVD-Update with your proxy by running CVD-Update like this:
First, set a custom domain name server to use the proxy:
```bash
cvd config set --nameserver
```
Then run CVD-Update like this:
```bash
http_proxy=http://: https_proxy=http://: cvd update -V
```
Or create a script to wrap the CVD-Update call. Something like:
```bash
#!/bin/bash
http_proxy=http://:
export http_proxy
https_proxy=http://:
export https_proxy
cvd update -V
```
> _Disclaimer_: CVD-Update doesn't support proxies that require authentication at this time. If your network admin allows it, you may be able to work around it by updating your proxy to allow HTTP requests through unauthenticated if the User-Agent matches your specific CVD-Update user agent. The CVD-Update User-Agent follows the form `CVDUPDATE/ ()` where the `uuid` is unique to your installation and can be found in the `~/.cvdupdate/config.json` file. See https://github.com/Cisco-Talos/cvdupdate/issues/9 for more details.
>
> Adding support for proxy authentication is a ripe opportunity for a community contribution to the project.
## Files and directories created by CVD-Update
This tool is to creates the following directories:
- `~/.cvdupdate`
- `~/.cvdupdate/logs`
- `~/.cvdupdate/databases`
This tool creates the following files:
- `~/.cvdupdate/config.json`
- `~/.cvdupdate/databases/.cvd`
- `~/.cvdupdate/databases/-.cdiff`
- `~/.cvdupdate/logs/.log`
> _Tip_: You can set custom `database` and `logs` directories with the `cvd config set` command. It is likely you will want to customize the `database` directory to point to your HTTP server's `www` directory (or equivalent). Bare in mind that if you already downloaded the databases to the old directory, you may want to move them to the new directory.
> _Important_: If you want to use a custom config path, you'll have to use it in every command. If you're fine with having it go in `~/.cvdupdate/config.json`, don't worry about it.
## Additional Usage
### Get familiar with the tool
Familiarize yourself with the various commands using the `--help` option.
```bash
cvd --help
cvd config --help
cvd update --help
cvd clean --help
```
Print out the current list of databases.
```bash
cvd list -V
```
Print out the config to see what it looks like.
```bash
cvd config show
```
### Do an update
Do an update, use "verbose mode" to so you can get a feel for how it works.
```bash
cvd update -V
```
List out the databases again:
```bash
cvd list -V
```
The print out the config again so you can see what's changed.
```bash
cvd config show
```
And maybe take a peek in the database directory as well to see it for yourself.
```bash
ls ~/.cvdupdate/database
```
Have a look at the logs if you wish.
```bash
ls ~/.cvdupdate/logs
cat ~/.cvdupdate/logs/*
```
### Serve it up, Test out FreshClam
Test out your mirror with FreshClam on the same computer.
This tool includes a `--serve` feature that will host the current database directory on http://localhost (default port: 8000).
You can test it by running `freshclam` or `freshclam.exe` locally, where you've configured `freshclam.conf` with:
```
DatabaseMirror http://localhost:8000
```
## Contribute
We'd love your help. There are many ways to contribute!
### Community
Join the ClamAV community on the [ClamAV Discord chat server](https://discord.gg/sGaxA5Q).
### Report issues
If you find an issue with CVD-Update or the CVD-Update documentation, please submit an issue to our [GitHub issue tracker](https://github.com/Cisco-Talos/cvdupdate/issues). Before you submit, please check to if someone else has already reported the issue.
### Development
If you find a bug and you're able to craft a fix yourself, consider submitting the fix in a [pull request](https://github.com/Cisco-Talos/cvdupdate/pulls). Your help will be greatly appreciated.
If you want to contribute to the project and don't have anything specific in mind, please check out our [issue tracker](https://github.com/Cisco-Talos/cvdupdate/issues). Perhaps you'll be able to fix a bug or add a cool new feature.
_By submitting a contribution to the project, you acknowledge and agree to assign Cisco Systems, Inc the copyright for the contribution. If you submit a significant contribution such as a new feature or capability or a large amount of code, you may be asked to sign a contributors license agreement comfirming that Cisco will have copyright license and patent license and that you are authorized to contribute the code._
#### Development Set-up
The following steps are intended to help users that wish to contribute to development of the CVD-Update project get started.
1. Create a fork of the [CVD-Update git repository](https://github.com/Cisco-Talos/cvdupdate), and then clone your fork to a local directory.
For example:
```bash
git clone https://github.com//cvdupdate.git
```
2. Make sure CVD-Update is not already installed. If it is, remove it.
```bash
python3 -m pip uninstall cvdupdate
```
3. Use pip to install CVD-Update in "edit" mode.
```bash
python3 -m pip install -e --user ./cvdupdate
```
Once installed in "edit" mode, any changes you make to your clone of the CVD-Update code will be immediately usable simply by running the `cvdupdate` / `cvd` commands.
### Conduct
This project has not selected a specific Code-of-Conduct document at this time. However, contributors are expected to behave in professional and respectful manner. Disrespectful or inappropriate behavior will not be tolerated.
## License
CVD-Update is licensed under the Apache License, Version 2.0 (the "License"). You may not use the CVD-Update project except in compliance with the License.
A copy of the license is located [here](LICENSE), and is also available online at [apache.org](http://www.apache.org/licenses/LICENSE-2.0).
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
cvdupdate-1.0.2/cvdupdate/ 0000755 0001751 0000164 00000000000 14037577401 016261 5 ustar runner docker 0000000 0000000 cvdupdate-1.0.2/cvdupdate/__init__.py 0000644 0001751 0000164 00000000000 14037577371 020366 0 ustar runner docker 0000000 0000000 cvdupdate-1.0.2/cvdupdate/__main__.py 0000644 0001751 0000164 00000022206 14037577371 020363 0 ustar runner docker 0000000 0000000 #!/usr/bin/env python3
"""
CVD-Update: ClamAV Database Updater
"""
_description = """
A tool to download and update clamav databases and database patch files
for the purposes of hosting your own database mirror.
"""
_copyright = """
Copyright (C) 2021 Cisco Systems, Inc. and/or its affiliates. All rights reserved.
"""
"""
Author: Micah Snyder
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
"""
import logging
import os
from pathlib import Path
import sys
import click
import coloredlogs
from http.server import HTTPServer
from RangeHTTPServer import RangeRequestHandler
import pkg_resources
from cvdupdate.cvdupdate import CVDUpdate
logging.basicConfig()
module_logger = logging.getLogger("cvdupdate")
coloredlogs.install(level="DEBUG", fmt="%(asctime)s %(name)s %(levelname)s %(message)s")
module_logger.setLevel(logging.DEBUG)
from colorama import Fore, Back, Style
#
# CLI Interface
#
@click.group(
epilog=Fore.BLUE
+ __doc__ + "\n"
+ Fore.GREEN
+ _description + "\n"
+ f"\nVersion {pkg_resources.get_distribution('cvdupdate').version}\n"
+ Style.RESET_ALL
+ _copyright,
)
def cli():
pass
@cli.command("list")
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
def db_list(config: str, verbose: bool):
"""
List the DBs found in the database directory.
"""
m = CVDUpdate(config=config, verbose=verbose)
m.db_list()
@cli.command("show")
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
@click.argument("db", required=True)
def db_show(config: str, verbose: bool, db: str):
"""
Show details about a specific database.
"""
m = CVDUpdate(config=config, verbose=verbose)
if not m.db_show(db):
sys.exit(1)
@cli.command("update")
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
@click.option("--debug-mode", "-D", is_flag=True, default=False, help="Print out HTTP headers for debugging purposes. [optional]")
@click.argument("db", required=False, default="")
def db_update(config: str, verbose: bool, db: str, debug_mode: bool):
"""
Update the DBs from the internet. Will update all DBs if DB not specified.
"""
m = CVDUpdate(config=config, verbose=verbose)
errors = m.db_update(db, debug_mode)
if errors > 0:
sys.exit(errors)
@cli.command("add")
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
@click.argument("db", required=True)
@click.argument("url", required=True)
def db_add(config: str, verbose: bool, db: str, url: str):
"""
Add a db to the list of known DBs.
"""
m = CVDUpdate(config=config, verbose=verbose)
if not m.config_add_db(db, url=url):
sys.exit(1)
@cli.command("remove")
@click.option("--config", "-c", type=str, required=False, default="")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
@click.argument("db", required=True)
def db_remove(config: str, verbose: bool, db: str):
"""
Remove a db from the list of known DBs and delete local copies of the DB.
"""
m = CVDUpdate(config=config, verbose=verbose)
if not m.config_remove_db(db):
sys.exit(1)
@cli.group(help="Commands to configure.")
def config():
pass
@config.command("set")
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
@click.option("--logdir", "-l", type=click.Path(), required=False, default="", help="Set a custom log directory. [optional]")
@click.option("--dbdir", "-d", type=click.Path(), required=False, default="", help="Set a custom database directory. [optional]")
@click.option("--nameserver", "-n", type=click.STRING, required=False, default="", help="Set a custom DNS nameserver. [optional]")
def config_set(config: str, verbose: bool, logdir: str, dbdir: str, nameserver: str):
"""
Set up first time configuration.
The default directories will be in ~/.cvdupdate
"""
CVDUpdate(
config=config,
verbose=verbose,
log_dir=logdir,
db_dir=dbdir,
nameserver=nameserver)
@config.command("show")
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
def config_show(config: str, verbose: bool):
"""
Print out the current configuration.
"""
m = CVDUpdate(config=config, verbose=verbose)
m.config_show()
@cli.group(help="Commands to clean up.")
def clean():
pass
@clean.command("dbs")
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
def clean_dbs(config: str, verbose: bool):
"""
Delete all files in the database directory.
"""
m = CVDUpdate(config=config, verbose=verbose)
m.clean_dbs()
@clean.command("logs")
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
def clean_logs(config: str, verbose: bool):
"""
Delete all files in the logs directory
"""
m = CVDUpdate(config=config, verbose=verbose)
m.clean_logs()
@clean.command("all")
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
def clean_all(config: str, verbose: bool):
"""
Delete the logs, databases, and config file.
"""
m = CVDUpdate(config=config, verbose=verbose)
m.clean_all()
@cli.command("serve")
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
@click.argument("port", type=int, required=False, default=8000)
def serve(port: int, config: str, verbose: bool):
"""
Serve up the database directory. Not a production quality server.
Intended for testing purposes.
"""
m = CVDUpdate(config=config, verbose=verbose)
os.chdir(str(m.db_dir))
m.logger.info(f"Serving up {m.db_dir} on localhost:{port}...")
RangeRequestHandler.protocol_version = 'HTTP/1.0'
# TODO(danvk): pick a random, available port
httpd = HTTPServer(('', port), RangeRequestHandler)
httpd.serve_forever()
#
# Command Aliases
#
@cli.command("list")
@click.pass_context
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
def list_alias(ctx, config: str, verbose: bool):
"""
List the DBs found in the database directory.
This is just an alias for `db list`.
"""
ctx.forward(db_list)
@cli.command("show")
@click.pass_context
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
@click.argument("db", required=True)
def show_alias(ctx, config: str, verbose: bool, db: str):
"""
Show details about a specific database.
This is just an alias for `db show`.
"""
ctx.forward(db_show)
@cli.command("update")
@click.pass_context
@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]")
@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]")
@click.option("--debug-mode", "-D", is_flag=True, default=False, help="Print out HTTP headers for debugging purposes. [optional]")
@click.argument("db", required=False, default="")
def update_alias(ctx, config: str, verbose: bool, db: str, debug_mode: bool):
"""
Update local copy of DBs.
This is just an alias for `db show`.
"""
ctx.forward(db_update)
if __name__ == "__main__":
sys.argv[0] = "cvdupdate"
cli(sys.argv[1:])
cvdupdate-1.0.2/cvdupdate/cvdupdate.py 0000644 0001751 0000164 00000121250 14037577371 020621 0 ustar runner docker 0000000 0000000 """
Copyright (C) 2021 Cisco Systems, Inc. and/or its affiliates. All rights reserved.
This module provides a tool to download and update clamav databases and database
patch files (CDIFFs) for the purposes of hosting your own database mirror.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
"""
from pathlib import Path
import copy
import datetime
from enum import Enum
import json
import logging
import os
import pkg_resources
import re
import subprocess
import sys
import time
from typing import *
import uuid
import http.client as http_client
from dns import resolver
import requests
from requests import status_codes
class CvdStatus(Enum):
NO_UPDATE = 0
UPDATED = 1
ERROR = 2
class CVDUpdate:
default_config_path: Path = Path.home() / ".cvdupdate" / "config.json"
default_config: dict = {
"nameserver" : "",
"max retry" : 3, # No `cvd config set` option to set this, because we don't
# _really_ want people hammering the CDN with partial downloads.
"log directory" : str(Path.home() / ".cvdupdate" / "logs"),
"rotate logs" : True,
"# logs to keep" : 30,
"db directory" : str(Path.home() / ".cvdupdate" / "database"),
"rotate cdiffs" : True,
"# cdiffs to keep" : 30,
"dbs" : {
"main.cvd" : {
"url" : "https://database.clamav.net/main.cvd",
"retry after" : 0,
"last modified" : 0,
"last checked" : 0,
"DNS field" : 1, # only for CVDs
"local version" : 0, # only for CVDs
"CDIFFs" : [] # only for CVDs
},
"daily.cvd" : {
"url" : "https://database.clamav.net/daily.cvd",
"retry after" : 0,
"last modified" : 0,
"last checked" : 0,
"DNS field" : 2,
"local version" : 0,
"CDIFFs" : []
},
"bytecode.cvd" : {
"url" : "https://database.clamav.net/bytecode.cvd",
"retry after" : 0,
"last modified" : 0,
"last checked" : 0,
"DNS field" : 7,
"local version" : 0,
"CDIFFs" : []
},
},
}
config_path: Path
config: dict
db_dir: Path
log_dir: Path
version: str
def __init__(
self,
config: str = "",
log_dir: str = "",
db_dir: str = "",
nameserver: str = "",
verbose: bool = False,
) -> None:
"""
CVDUpdate class.
Args:
log_dir: path output log.
db_dir: path where databases will be downloaded.
verbose: Enable DEBUG-level logs and other verbose messages.
"""
self.version = pkg_resources.get_distribution('cvdupdate').version
self.verbose = verbose
self._read_config(
config,
db_dir,
log_dir,
nameserver)
self._init_logging()
def _init_logging(self) -> None:
"""
Initializes the logging parameters.
"""
self.logger = logging.getLogger(f"cvdupdate-{self.version}")
if self.verbose:
self.logger.setLevel(logging.DEBUG)
else:
self.logger.setLevel(logging.INFO)
formatter = logging.Formatter(
fmt="%(asctime)s - %(levelname)s: %(message)s",
datefmt="%Y-%m-%d %I:%M:%S %p",
)
today = datetime.datetime.now()
self.log_file = self.log_dir / f"{today.strftime('%Y-%m-%d')}.log"
if not self.log_dir.exists():
# Make a new log directory
os.makedirs(os.path.split(self.log_file)[0])
else:
# Log dir already exists, lets check if we need to prune old logs
logs = self.log_dir.glob('*.log')
for log in logs:
log_date_str = str(log.stem)
log_date = datetime.datetime.strptime(log_date_str, "%Y-%m-%d")
if log_date + datetime.timedelta(days=self.config["# logs to keep"]) < today:
# Log is too old, delete!
os.remove(str(log))
self.filehandler = logging.FileHandler(filename=self.log_file)
self.filehandler.setLevel(self.logger.level)
self.filehandler.setFormatter(formatter)
self.logger.addHandler(self.filehandler)
# Also set the log level for urllib3, because it's "DEBUG" by default,
# and we may not like that.
urllib3_logger = logging.getLogger("urllib3.connectionpool")
urllib3_logger.setLevel(self.logger.level)
def _read_config(self,
config: str,
db_dir: str,
log_dir: str,
nameserver: str) -> None:
"""
Read in the config file.
Create a new one if one does not already exist.
"""
need_save = False
if config == "":
self.config_path = self.default_config_path
else:
self.config_path = Path(config)
if self.config_path.exists():
# Config already exists, load it.
with self.config_path.open('r') as config_file:
self.config = json.load(config_file)
else:
# Config does not exist, use default
self.config = self.default_config
need_save = True
if 'uuid' not in self.config:
# Create a UUID to put in our User-Agent for better (anonymous) metrics
self.config['uuid'] = str(uuid.uuid4())
need_save = True
if db_dir != "":
self.config["db directory"] = db_dir
need_save = True
self.db_dir = Path(self.config["db directory"])
if log_dir != "":
self.config["log directory"] = log_dir
need_save = True
self.log_dir = Path(self.config["log directory"])
if nameserver != "":
self.config['nameserver'] = nameserver
need_save = True
# For backwards compatibility with older configs.
if 'nameserver' not in self.config:
self.config['nameserver'] = ""
need_save = True
if 'max retry' not in self.config:
self.config['max retry'] = 3
need_save = True
if need_save:
self._save_config()
def _save_config(self) -> None:
"""
Save the current configuration.
"""
if not self.config_path.parent.exists():
# config directory doesn't exist yet
try:
os.makedirs(str(self.config_path.parent))
except Exception as exc:
print("Failed to create config directory!")
raise exc
try:
with self.config_path.open('w') as config_file:
json.dump(self.config, config_file, indent=4)
except Exception as exc:
print("Failed to create config file!")
raise exc
if self.verbose:
print(f"Saved: {self.config_path}\n")
def config_show(self):
"""
Print out the config
"""
print(f"Config file: {self.config_path}\n")
print(f"Config:\n{json.dumps(self.config, indent=4)}\n")
def update(self, db: str = "") -> bool:
"""
Update a specific database or all the databases.
"""
def clean_dbs(self):
"""
Delete all files in the database directory.
"""
self.logger.info(f"Deleting databases...")
dbs = self.db_dir.glob('*')
for db in dbs:
os.remove(str(db))
self.logger.info(f"Deleted: {db}")
def clean_logs(self):
"""
Delete all files in the log directory.
"""
self.logger.info(f"Deleting log files...")
logs = self.log_dir.glob('*')
for log in logs:
os.remove(str(log))
print(f"Deleted: {log}")
def clean_all(self):
"""
Delete all logs and databases and the config.
"""
self.clean_dbs()
self.clean_logs()
os.remove(str(self.config_path))
print(f"Deleted: {self.config_path}")
def _index_local_databases(self) -> dict:
need_save = False
dbs = copy.deepcopy(self.config['dbs'])
db_paths = self.db_dir.glob('*')
for db in db_paths:
if db.name.endswith('.cdiff'):
# Ignore CDIFFs, they'll get printed later.
continue
if db.name not in dbs:
version = 0
# Found a file in here that ISN'T a part of the config
if db.name.endswith('.cvd'):
# Found a CVD in here that ISN'T a part of the config!
# Very odd BTW.
self.logger.warning(f"Found a CVD in the DB directory that isn't in the config: {db.name}")
try:
version = self._get_cvd_version_from_file(db)
except Exception as exc:
self.logger.debug(f"EXCEPTION OCCURRED: {exc}")
self.logger.error(f"Failed to determine version for {db.name}")
dbs[db.name] = {
"url" : "n/a",
"retry after" : 0,
"last modified" : os.path.getmtime(str(db)),
"last checked" : 0,
"DNS field" : 0,
"local version" : version,
"CDIFFs" : []
}
else:
# DB on disk is from our config
if db.name.endswith(".cvd") and self.config['dbs'][db.name]['local version'] == 0:
# Seems like we somehow got a (config'd) CVD file in our database directory without
# saving the CVD info to the config. Let's just update the version field.
self.logger.info(f"Found {db.name} in the DB directory, though it wasn't downloaded using this tool.")
try:
dbs[db.name]['local version'] = self._get_cvd_version_from_file(self.db_dir / db.name)
self.logger.info(f"Identified mysterious {db.name} version: {dbs[db.name]['local version']}")
# Add the version info for this mysteriously deposited CVD to our config.
self.config['dbs'][db.name]['local version'] = dbs[db.name]['local version']
need_save = True
except Exception as exc:
self.logger.debug(f"EXCEPTION OCCURRED: {exc}")
self.logger.error(f"Failed to determine version # of mysterious {db.name} file. Perhaps it is corrupted?")
if need_save:
self._save_config()
return dbs
def db_list(self) -> None:
"""
Print list of databases
"""
dbs = self._index_local_databases()
for db in dbs:
updated = datetime.datetime.fromtimestamp(dbs[db]['last modified']).strftime('%Y-%m-%d %H:%M:%S')
checked = datetime.datetime.fromtimestamp(dbs[db]['last checked']).strftime('%Y-%m-%d %H:%M:%S')
self.logger.info(f"Database: {db}")
if dbs[db]['last modified'] == 0:
self.logger.info(" last modified: not downloaded")
else:
self.logger.info(f" last modified: {updated}")
if dbs[db]['last checked'] == 0:
self.logger.debug(" last checked: n/a")
else:
self.logger.debug(f" last checked: {checked}")
self.logger.debug(f" url: {dbs[db]['url']}")
if db.endswith(".cvd"):
# Only CVD's have versions.
self.logger.debug(f" local version: {dbs[db]['local version']}")
if len(dbs[db]['CDIFFs']) > 0:
self.logger.debug(f" CDIFFs:")
for cdiff in dbs[db]['CDIFFs']:
self.logger.debug(f" {cdiff}")
def db_show(self, name) -> bool:
"""
Show details for a specific database
"""
found = False
dbs = self._index_local_databases()
for db in dbs:
if db == name:
found = True;
updated = datetime.datetime.fromtimestamp(dbs[db]['last modified']).strftime('%Y-%m-%d %H:%M:%S')
checked = datetime.datetime.fromtimestamp(dbs[db]['last checked']).strftime('%Y-%m-%d %H:%M:%S')
self.logger.info(f"Database: {db}")
if dbs[db]['last modified'] == 0:
self.logger.info(" last modified: not downloaded")
else:
self.logger.info(f" last modified: {updated}")
if dbs[db]['last checked'] == 0:
self.logger.info(" last checked: n/a")
else:
self.logger.info(f" last checked: {checked}")
self.logger.info(f" url: {dbs[db]['url']}")
if db.endswith(".cvd"):
self.logger.info(f" local version: {dbs[db]['local version']}")
if len(dbs[db]['CDIFFs']) > 0:
self.logger.info(f" CDIFFs: \n{json.dumps(dbs[db]['CDIFFs'], indent=4)}")
return True
if not found:
self.logger.error(f"No such database: {name}")
return found
def _query_dns_txt_entry(self) -> bool:
'''
Attempt to get version from current.cvd.clamav.net DNS TXT entry
'''
got_it = False
self.logger.debug(f"Checking available versions via DNS TXT entry query of current.cvd.clamav.net")
try:
our_resolver = resolver.Resolver()
our_resolver.timeout = 5 # Explicitly setting query timeout to mitigate https://github.com/Cisco-Talos/cvdupdate/issues/17
nameserver = os.environ.get("CVDUPDATE_NAMESERVER")
if nameserver != None:
# Override the default nameserver using the CVDUPDATE_NAMESERVER environment variable.
our_resolver.nameservers = [nameserver]
self.logger.info(f"Using nameserver specified in CVDUPDATE_NAMESERVER: {nameserver}")
elif self.config['nameserver'] != "":
# Override the default nameserver using a config setting.
our_resolver.nameservers = [self.config['nameserver']]
self.logger.info(f"Using nameserver specified in the config: {self.config['nameserver']}")
answer = str(our_resolver.resolve("current.cvd.clamav.net","TXT").response.answer[0])
versions = re.search('".*"', answer).group().strip('"')
self.dns_version_tokens = versions.split(':')
got_it = True
except Exception as exc:
self.logger.debug(f"EXCEPTION OCCURRED: {exc}")
self.logger.warning(f"Failed to determine available version via DNS TXT query!")
return got_it
def _query_cvd_version_dns(self, db: str) -> int:
'''
This is a faux query.
Try to look up the version # from the DNS TXT entry we already have.
'''
version = 0
if self.dns_version_tokens == []:
# Query DNS if we haven't already
self._query_dns_txt_entry()
if self.dns_version_tokens == []:
# Query failed. Bail out.
return version
self.logger.debug(f"Checking {db} version via DNS TXT advertisement.")
if self.config['dbs'][db]['DNS field'] == 0:
# Invalid DNS field value for database version.
self.logger.warning(f"Failed to get DB version from DNS TXT entry: Invalid DNS field value for database version.")
return version
try:
version = int(self.dns_version_tokens[self.config['dbs'][db]['DNS field']])
self.logger.debug(f"{db} version advertised by DNS: {version}")
# Update the "last checked" time.
self.config['dbs'][db]['last checked'] = time.time()
except Exception as exc:
self.logger.debug(f"EXCEPTION OCCURRED: {exc}")
self.logger.warning(f"Failed to get DB version from DNS TXT entry!")
return version
def _query_cvd_version_http(self, db: str) -> int:
'''
Download the CVD header and read the CVD version
Return 1+ if queried version.
Return 0 if failed.
'''
version = 0
retry = 0
url = self.config['dbs'][db]['url']
self.logger.debug(f"Checking {db} version via HTTP download of CVD header.")
ims = datetime.datetime.utcfromtimestamp(self.config['dbs'][db]['last modified']).strftime('%a, %d %b %Y %H:%M:%S GMT')
while retry < self.config['max retry']:
response = requests.get(url, headers = {
'User-Agent': f'CVDUPDATE/{self.version} ({self.config["uuid"]})',
'Range': 'bytes=0-95',
'If-Modified-Since': ims,
})
if ((response.status_code == 200 or response.status_code == 206) and
('content-length' in response.headers) and
(int(response.headers['content-length']) > len(response.content))):
self.logger.warning(f"Response was truncated somehow...")
self.logger.warning(f" Expected {response.headers['content-length']}")
self.logger.warning(f" Received {response.content}, let's retry.")
retry += 1
else:
break
if response.status_code == 200 or response.status_code == 206:
# Looks like we downloaded something...
if (('content-length' in response.headers) and int(response.headers['content-length']) > len(response.content)):
self.logger.error(f"Failed to download {db} header to check the version #.")
return 0
# Successfully downloaded the header.
# We used the IMS header so this means it's probably newer, but we'll check just in case.
cvd_header = response.content
version = self._get_version_from_cvd_header(cvd_header)
self.logger.debug(f"{db} version available by HTTP download: {version}")
elif response.status_code == 304:
# HTTP Not-Modified, it's not newer.than what we already have.
# Just return the current local version.
version = self.config['dbs'][db]['local version']
self.logger.debug(f"{db} not-modified since: {ims} (local version {version})")
elif response.status_code == 429:
# Rejected because downloading the same file too frequently.
self.logger.warning(f"Failed to download {db} header to check the version #.")
self.logger.warning(f"Download request rejected because we've downloaded the same file too frequently.")
try_again_seconds = 60 * 60 * 12 # 12 hours
if 'Retry-After' in response.headers.keys():
try_again_seconds = int(response.headers['Retry-After'])
self.config['dbs'][db]['retry after'] = time.time() + float(try_again_seconds)
try_again_string = str(datetime.timedelta(seconds=try_again_seconds))
self.logger.warning(f"We won't try {db} again for {try_again_string} hours.")
else:
# Check failed!
self.logger.error(f"Failed to download {db} header to check the version #. Url: {url}")
if version > 0:
# Update the "last checked" time.
self.config['dbs'][db]['last checked'] = time.time()
return version
def _download_db_from_url(self, db: str, url: str, last_modified: int, version=0) -> CvdStatus:
'''
Download contents from a url and save to a filename in the database directory.
Will use If-Modified-Since
If Not-Modified, it will not replace the current database.
'''
retry = 0
ims: str = datetime.datetime.utcfromtimestamp(last_modified).strftime('%a, %d %b %Y %H:%M:%S GMT')
while retry < self.config['max retry']:
response = requests.get(url, headers = {
'User-Agent': f'CVDUPDATE/{self.version} ({self.config["uuid"]})',
'If-Modified-Since': ims,
})
if ((response.status_code == 200 or response.status_code == 206) and
('content-length' in response.headers) and
(int(response.headers['content-length']) > len(response.content))):
self.logger.warning(f"Response was truncated somehow...")
self.logger.warning(f" Expected {response.headers['content-length']}")
self.logger.warning(f" Received {response.content}, let's retry.")
retry += 1
else:
break
if response.status_code == 200:
# Looks like we downloaded something...
if (('content-length' in response.headers) and int(response.headers['content-length']) > len(response.content)):
self.logger.error(f"Failed to download {db}")
return CvdStatus.ERROR
# Download Success
if version > 0:
self.logger.info(f"Downloaded {db}. Version: {version}")
else:
self.logger.info(f"Downloaded {db}")
try:
with (self.db_dir / db).open('wb') as new_db:
new_db.write(response.content)
# Update config w/ new db info
self.config['dbs'][db]['last modified'] = time.time()
if db.endswith('.cvd'):
self.config['dbs'][db]['local version'] = self._get_version_from_cvd_header(response.content[:96])
except Exception as exc:
self.logger.debug(f"EXCEPTION OCCURRED: {exc}")
self.logger.error(f"Failed to save {db} to {self.db_dir}")
return CvdStatus.ERROR
elif response.status_code == 304:
# Not modified since IMS. We have the latest version.
version = self.config['dbs'][db]['local version']
self.logger.info(f"{db} not-modified since: {ims} (local version {version})")
return CvdStatus.NO_UPDATE
elif response.status_code == 429:
# Rejected because downloading the same file too frequently.
self.logger.warning(f"Failed to download {db}.")
self.logger.warning(f"Download request rejected because we've downloaded the same file too frequently.")
try_again_seconds = 60 * 60 * 12 # 12 hours
if 'Retry-After' in response.headers.keys():
try_again_seconds = int(response.headers['Retry-After'])
self.config['dbs'][db]['retry after'] = time.time() + float(try_again_seconds)
try_again_string = str(datetime.timedelta(seconds=try_again_seconds))
self.logger.warning(f"We won't try {db} again for {try_again_string} hours.")
# We'll have to retry after the cooldown.
return CvdStatus.ERROR
else:
# HTTP Get failed.
self.logger.error(f"Failed to download {db} from {url}")
return CvdStatus.ERROR
return CvdStatus.UPDATED
def _download_cvd(self, db: str, available_version: int) -> CvdStatus:
'''
Download the latest available version
If we already have some version of the database, attempt to download all CDIFFs in between.
If we don't, just get the last two CDIFFs.
'''
local_version = self.config['dbs'][db]['local version']
desired_version = local_version + 1
if local_version >= available_version:
# Oh! We're already up to date, don't worry about it.
self.logger.info(f"{db} is up-to-date. Version: {local_version}")
return CvdStatus.NO_UPDATE
elif local_version == 0:
# We don't have any version of the DB, let's just get the newest version + the last CDIFF
desired_version = available_version
# First try to get CDIFFs
self.logger.debug(f"Downloading CDIFFs first...")
while desired_version <= available_version:
# Attempt to download each CDIFF betwen our local version and the available version
# The url for CVDs should be https://database.clamav.net/
# Eg:
# https://database.clamav.net/daily.cvd
# For the daily cdiffs, we would want:
# https://database.clamav.net/daily-.cdiff
retry = 0
cdiff_filename = f"{db[:-len('.cvd')]}-{desired_version}.cdiff"
original_url = self.config['dbs'][db]['url']
url = f"{original_url[:-len(db)]}{cdiff_filename}"
if (self.db_dir / cdiff_filename).exists():
self.logger.debug(f"We already have {cdiff_filename}. Skipping...")
self.logger.debug(f"Checking for {cdiff_filename}")
while retry < self.config['max retry']:
response = requests.get(url, headers = {
'User-Agent': f'CVDUPDATE/{self.version} ({self.config["uuid"]})',
})
if ((response.status_code == 200 or response.status_code == 206) and
('content-length' in response.headers) and
(int(response.headers['content-length']) > len(response.content))):
self.logger.warning(f"Response was truncated somehow...")
self.logger.warning(f" Expected {response.headers['content-length']}")
self.logger.warning(f" Received {response.content}, let's retry.")
retry += 1
else:
break
if response.status_code == 200:
# Looks like we downloaded something...
if (('content-length' in response.headers) and int(response.headers['content-length']) > len(response.content)):
self.logger.error(f"Failed to download {db} header to check the version #.")
return CvdStatus.ERROR
# Download Success
self.logger.info(f"Downloaded {cdiff_filename}")
try:
with (self.db_dir / f"{cdiff_filename}").open('wb') as new_db:
new_db.write(response.content)
except Exception as exc:
self.logger.debug(f"EXCEPTION OCCURRED: {exc}")
self.logger.error(f"Failed to save {cdiff_filename} to {self.db_dir}.")
# Update config with CDIFF, for posterity
self.config['dbs'][db]['CDIFFs'].append(cdiff_filename)
# Prune old CDIFFs if needed
if len(self.config['dbs'][db]['CDIFFs']) > self.config['# cdiffs to keep']:
try:
os.remove(self.db_dir / self.config['dbs'][db]['CDIFFs'][0])
except Exception as exc:
self.logger.debug(f"EXCEPTION OCCURRED: {exc}")
self.logger.debug(f"Tried to prune old cdiffs, but they weren't found, maybe someone else removed them already.")
self.config['dbs'][db]['CDIFFs'] = self.config['dbs'][db]['CDIFFs'][1:]
elif response.status_code == 429:
# Rejected because downloading the same file too frequently.
self.logger.warning(f"Failed to download {cdiff_filename}")
self.logger.warning(f"Download request rejected because we've downloaded the same file too frequently.")
try_again_seconds = 60 * 60 * 12 # 12 hours
if 'Retry-After' in response.headers.keys():
try_again_seconds = int(response.headers['Retry-After'])
self.config['dbs'][db]['retry after'] = time.time() + float(try_again_seconds)
try_again_string = str(datetime.timedelta(seconds=try_again_seconds))
self.logger.warning(f"We won't try {db} again for {try_again_string} hours.")
# Sure only a CDIFF failed, but if we want any chance of trying the CDIFF again
# in the future, let's bail out now and retry the CVD + CDIFFs after the cooldown.
return CvdStatus.ERROR
else:
# HTTP Get failed.
self.logger.info(f"No CDIFF found for {db} version # {desired_version}")
if desired_version < available_version:
desired_version = available_version - 1
self.logger.info(f"Will just skip to the last CDIFF instead.")
else:
self.logger.info(f"Giving up on CDIFFs for {db}")
break
desired_version += 1
# Now download the available version.
desired_version = available_version
url = f"{self.config['dbs'][db]['url']}?version={desired_version}"
return self._download_db_from_url(db, url, last_modified=0, version=desired_version)
def _get_version_from_cvd_header(self, cvd_header: bytes) -> int:
'''
Parse a CVD header to read the database version.
'''
header_fields = cvd_header.decode('utf-8', 'ignore').strip().split(':')
version_found = 0
try:
version_found = int(header_fields[2])
except Exception as exc:
self.logger.debug(f"EXCEPTION OCCURRED: {exc}")
self.logger.error(f"Failed to determine version from CVD header!")
return version_found
def _get_cvd_version_from_file(self, path: Path) -> int:
cvd_header: bytes = b''
version_found = 0
try:
with path.open('rb') as cvd_fd:
cvd_header = cvd_fd.read(96)
if len(cvd_header) < 96:
# Most likely a corrupted CVD. Delete.
self.logger.debug(f"Failed to read CVD header, perhaps {path.name} is corrupted.")
self.logger.debug(f"Will delete {path.name} so it will not cause further problems.")
os.remove(str(path))
else:
# Got the header, lets parse out the version.
version_found = self._get_version_from_cvd_header(cvd_header)
except Exception as exc:
self.logger.debug(f"EXCEPTION OCCURRED: {exc}")
self.logger.error(f"Failed to read version from CVD header from {path}.")
if version_found == 0:
self.logger.error(f"Failed to determine version from CVD header.")
return version_found
def pypi_update_check(self):
def check(name):
'''
Check if there's a newer version of the cvdupdate package.
From https://stackoverflow.com/questions/58648739/how-to-check-if-python-package-is-latest-version-programmatically
'''
self.logger.debug(f'Checking for a newer version of cvdupdate.')
result = subprocess.run([sys.executable, '-m', 'pip', 'install', '{}==random'.format('cvdupdate')], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
latest_version = result.stderr.decode("utf-8")
latest_version = latest_version[latest_version.find('(from versions:')+15:]
latest_version = latest_version[:latest_version.find(')')]
latest_version = latest_version.replace(' ','').split(',')[-1].strip()
result = subprocess.run([sys.executable, '-m', 'pip', 'show', '{}'.format('cvdupdate')], stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
current_version = result.stdout.decode("utf-8")
current_version = current_version[current_version.find('Version:')+8:]
current_version = current_version[:current_version.find('\n')].replace(' ','').strip()
if 'ERROR' in latest_version:
self.logger.debug(f"Version check didn't work, didn't get back a list of package versions.")
return True
elif latest_version == current_version:
self.logger.debug(f'cvdupdate is up-to-date: {current_version}.')
return True
else:
self.logger.warning(f'You are running cvdupate version: {current_version}.')
self.logger.warning(f'There is a newer version on PyPI: {latest_version}. Please update!')
return False
return check('cvdupdate')
def db_update(self, db="", debug_mode=False) -> int:
"""
Update one or all of the databases.
Returns: Number of errors.
"""
self.update_errors = 0
self.dbs_updated = 0
self.dns_version_tokens = []
# Make sure we have a database directory to save files to
if not self.db_dir.exists():
os.makedirs(self.db_dir)
# Check if there is a newer version of CVD-Update
self.pypi_update_check()
# Query DNS so we can efficiently query CVD version #'s
self._query_dns_txt_entry()
if self.dns_version_tokens == []:
# Query failed. Bail out.
self.logger.error(f"Failed to update {db}. Missing or invalid URL: {self.config['dbs'][db]['url']}")
return 1
if debug_mode:
http_client.HTTPConnection.debuglevel = 1
def update(db) -> CvdStatus:
'''
Update a database
'''
if self.config['dbs'][db]['retry after'] > 0:
cooldown_date = datetime.datetime.fromtimestamp(self.config['dbs'][db]['retry after']).strftime('%Y-%m-%d %H:%M:%S')
if self.config['dbs'][db]['retry after'] > time.time():
self.logger.warning(f"Skipping {db} which is on cooldown until {cooldown_date}")
return CvdStatus.ERROR
else:
# Cooldown expired. Ok to try again.
self.config['dbs'][db]['retry after'] = 0
self.logger.info(f"{db} cooldown expired {cooldown_date}. OK to try again...")
if not self.config['dbs'][db]['url'].startswith('http'):
self.logger.error(f"Failed to update {db}. Missing or invalid URL: {self.config['dbs'][db]['url']}")
return CvdStatus.ERROR
self.logger.debug(f"Checking {db} for update from {self.config['dbs'][db]['url']}")
if db.endswith('.cvd'):
# It's a CVD (official signed clamav database)
advertised_version = 0
if self.config['dbs'][db]['local version'] == 0 and (self.db_dir / db).exists():
# Seems like we somehow got a CVD in our database directory without
# saving the CVD info to the config. Let's just update the version field.
self.config['dbs'][db]['local version'] = self._get_cvd_version_from_file(self.db_dir / db)
if self.config['dbs'][db]['DNS field'] > 0:
# We can use the DNS TXT fields to check if our version is old.
advertised_version = self._query_cvd_version_dns(db)
else:
# We can't use DNS to see if our version is old.
# Use HTTP to pull just the CVD header to check.
# First, make sure no one tampered with the DNS field for
# main/daily/bytecode when using database.clamav.net
if (('database.clamav.net' in self.config['dbs'][db]['url']) and
(db == 'main.cvd' or db == 'daily.cvd' or db == 'bytecode.cvd')):
self.logger.error(f'It appears that the "DNS field" in {self.config_path} for "{db}" was modified from the default.')
self.logger.error(f'Updating {db} from database.clamav.net requires DNS for the version check in order to conserve bandwidth.')
self.logger.error(f'Please restore the default settings for the "DNS field" and try again.')
return CvdStatus.ERROR
advertised_version = self._query_cvd_version_http(db)
if advertised_version == 0:
self.logger.error(f"Failed to update {db}. Failed to query available CVD version")
return CvdStatus.ERROR
return self._download_cvd(db, advertised_version)
else:
# Try the download.
# Will use If-Modified-Since
# If Not-Modified, it will not replace the current database.
return self._download_db_from_url(
db,
self.config['dbs'][db]['url'],
self.config['dbs'][db]['last modified'])
if db == "":
# Update every DB.
for db in self.config['dbs']:
status = update(db)
if status == CvdStatus.ERROR:
self.update_errors += 1
elif status == CvdStatus.UPDATED:
self.dbs_updated += 1
else:
# Update a specific DB.
if db not in self.config['dbs']:
self.logger.error(f"Update failed. Unknown database: {db}")
else:
status = update(db)
if status == CvdStatus.ERROR:
self.update_errors += 1
elif status == CvdStatus.UPDATED:
self.dbs_updated += 1
self._save_config()
if self.update_errors == 0 and self.dbs_updated > 0:
with (self.db_dir / 'dns.txt').open('w') as dns_file:
dns_file.write(':'.join(self.dns_version_tokens))
self.logger.debug(f"Updated {self.db_dir / 'dns.txt'}")
return self.update_errors
def config_add_db(self, db: str, url: str) -> bool:
"""
Add another database + url to check when we update.
"""
extension = db.split('.')[-1]
if extension not in [
'cvd', 'cld', 'cud',
'cfg', 'cat', 'crb',
'ftm',
'ndb', 'ndu',
'ldb', 'ldu', 'idb',
'ydb', 'yar', 'yara',
'cdb',
'cbc',
'pdb', 'gdb', 'wdb',
'hdb', 'hsb', 'hdu', 'hsu',
'mdb', 'msb', 'mdu', 'msu',
'ign', 'ign2',
'info',
]:
self.logger.warning(f"{db} does not have valid clamav database file extension.")
if db in self.config['dbs']:
self.logger.info(f"Cannot add {db}, it is already in our list.")
self.logger.info(f"Hint: Try `db list -V` or `db show {db}` for more information.")
return False
self.config['dbs'][db] = {
"url" : url,
"retry after" : 0,
"last modified" : 0,
"last checked" : 0,
"DNS field" : 0,
"local version" : 0,
"CDIFFs" : []
}
self.logger.info(f"Added {db} ({url}) to DB list.")
self.logger.info(f"{db} will be downloaded next time you run `cvd update` or `cvd update {db}`")
self._save_config()
return True
def config_remove_db(self, db: str) -> bool:
"""
Remove a database from our list, and delete copies of the DB from the database directory.
"""
if db not in self.config['dbs']:
self.logger.info(f"Cannot remove {db}, it is not in our list.")
self.logger.info(f"Hint: Try `db list -V` for more information.")
return False
try:
if (self.db_dir / db).exists():
os.remove(str(self.db_dir / db))
self.logger.info(f"Deleted {db} from database directory.")
except Exception as exc:
self.logger.debug(f"An exception occured: {exc}")
self.logger.error(f"Failed to delete {db} from databse directory!")
for cdiff in self.config['dbs'][db]['CDIFFs']:
try:
if (self.db_dir / cdiff).exists():
os.remove(str(self.db_dir / cdiff))
self.logger.info(f"Deleted {cdiff} from database directory.")
except Exception as exc:
self.logger.debug(f"An exception occured: {exc}")
self.logger.error(f"Failed to delete {cdiff} from databse directory!")
self.config['dbs'].pop(db)
self.logger.info(f"Removed {db} from DB list.")
self._save_config()
return True
cvdupdate-1.0.2/cvdupdate.egg-info/ 0000755 0001751 0000164 00000000000 14037577401 017753 5 ustar runner docker 0000000 0000000 cvdupdate-1.0.2/cvdupdate.egg-info/PKG-INFO 0000644 0001751 0000164 00000035756 14037577401 021070 0 ustar runner docker 0000000 0000000 Metadata-Version: 2.1
Name: cvdupdate
Version: 1.0.2
Summary: ClamAV Private Database Mirror Updater Tool
Home-page: https://github.com/Cisco-Talos/cvdupdate
Author: The ClamAV Team
Author-email: clamav-bugs@external.cisco.com
License: UNKNOWN
Description: A tool to download and update clamav databases and database patch files
for the purposes of hosting your own database mirror.
Copyright (C) 2021 Cisco Systems, Inc. and/or its affiliates. All rights reserved.
## About
This tool downloads the latest ClamAV databases along with the latest database patch files.
This project replaces the `clamdownloader.pl` Perl script by Frederic Vanden Poel, formerly provided here: https://www.clamav.net/documents/private-local-mirrors
Run this tool as often as you like, but it will only download new content if there is new content to download. If you somehow manage to download too frequently (eg: by using `cvd clean all` and `cvd update` repeatedly), then the official database server may refuse your download request, and one or more databases may go on cool-down until it's safe to try again.
## Requirements
- Python 3.6 or newer.
- An internet connection with DNS enabled.
## Installation
You may install `cvdupdate` from PyPI using `pip`, or you may clone the project Git repository and use `pip` to install it locally.
Install `cvdupdate` from PyPI:
```bash
python3 -m pip install --user cvdupdate
```
## Basic Usage
Use the `--help` option with any `cvd` command to get help.
```bash
cvd --help
```
> _Tip_: You may not be able to run the `cvd` or `cvdupdate` shortcut directly if your Python Scripts directory is not in your `PATH` environment variable. If you run into this issue, and do not wish to add the Python Scripts directory to your path, you can run CVD-Update like this:
>
> ```bash
> python -m cvdupdate --help
> ```
(optional) You may wish to customize where the databases are stored:
```bash
cvd config set --dbdir
```
Run this to download the latest database and associated CDIFF patch files:
```bash
cvd update
```
Downloaded databases will be placed in `~/.cvdupdate/database` unless you customized it to use a different directory.
Newly downloaded databases will replace the previous database version, but the CDIFF patch files will accumulate up to a configured maximum before it starts deleting old CDIFFs (default: 30 CDIFFs). You can configure it to keep more CDIFFs by manually editing the config (default: `~/.cvdupdate/config.json`). The same behavior applies for CVD-Update log rotation.
Run this to serve up the database directory on `http://localhost:8000` so you can test it with FreshClam.
```bash
cvd serve
```
> _Disclaimer_: The `cvd serve` feature is not intended for production use, just for testing. You probably want to use a more robust HTTP server for production work.
Install ClamAV if you don't already have it and, in another terminal window, modify your `freshclam.conf` file. Replace:
```
DatabaseMirror database.clamav.net
```
... with:
```
DatabaseMirror http://localhost:8000
```
> _Tip_: A default install on Linux/Unix places `freshclam.conf` in `/usr/local/etc/freshclam.conf`. If one does not exist, you may need to create it using `freshclam.conf.sample` as a template.
Now, run `freshclam -v` or `freshclam.exe -v` to see what happens. You should see FreshClam successfully update it's own database directory from your private database server.
Run `cvd update` as often as you need. Maybe put it in a `cron` job.
> _Tip_: Each command supports a `--verbose` (`-V`) mode, which often provides more details about what's going on under the hood.
### Cron Example
Cron is a popular choice to automate frequent tasks on Linux / Unix systems.
1. Open a terminal running as the user which you want CVD-Update to run under, do the following:
```bash
crontab -e
```
2. Press `i` to insert new text, and add this line:
```bash
30 */4 * * * /bin/sh -c "~/.local/bin/cvd update &> /dev/null"
```
Or instead of `~/`, you can do this, replacing `username` with your user name:
```bash
30 */4 * * * /bin/sh -c "/home/username/.local/bin/cvd update &> /dev/null"
```
3. Press , then type `:wq` and press to write the file to disk and quit.
**About these settings**:
I selected `30 */4 * * *` to run at minute 30 past every 4th hour. CVD-Update uses a DNS check to do version checks before it attempts to download any files, just like FreshClam. Running CVD-Update more than once a day should not be an issue.
CVD-Update will write logs to the `~/.cvdupdate/logs` directory, which is why I directed `stdout` and `stderr` to `/dev/null` instead of a log file. You can use the `cvd config set` command to customize the log directory if you like, or redirect `stdout` and `stderr` to a log file if you prefer everything in one log instead of separate daily logs.
## Optional Functionality
### Using a custom DNS server
DNS is required for CVD-Update to function properly (to gather the TXT record containing the current definition database version). You can select a specific nameserver to ensure said nameserver is used when querying the TXT record containing the current database definition version available
1. Set the nameserver in the config. Eg:
```bash
cvd config set --nameserver 208.67.222.222
```
2. Set the environment variable `CVDUPDATE_NAMESERVER`. Eg:
```bash
CVDUPDATE_NAMESERVER="208.67.222.222" cvd update
```
The environment variable will take precedence over the nameserver config setting.
### Using a proxy
Depending on your type of proxy, you may be able to use CVD-Update with your proxy by running CVD-Update like this:
First, set a custom domain name server to use the proxy:
```bash
cvd config set --nameserver
```
Then run CVD-Update like this:
```bash
http_proxy=http://: https_proxy=http://: cvd update -V
```
Or create a script to wrap the CVD-Update call. Something like:
```bash
#!/bin/bash
http_proxy=http://:
export http_proxy
https_proxy=http://:
export https_proxy
cvd update -V
```
> _Disclaimer_: CVD-Update doesn't support proxies that require authentication at this time. If your network admin allows it, you may be able to work around it by updating your proxy to allow HTTP requests through unauthenticated if the User-Agent matches your specific CVD-Update user agent. The CVD-Update User-Agent follows the form `CVDUPDATE/ ()` where the `uuid` is unique to your installation and can be found in the `~/.cvdupdate/config.json` file. See https://github.com/Cisco-Talos/cvdupdate/issues/9 for more details.
>
> Adding support for proxy authentication is a ripe opportunity for a community contribution to the project.
## Files and directories created by CVD-Update
This tool is to creates the following directories:
- `~/.cvdupdate`
- `~/.cvdupdate/logs`
- `~/.cvdupdate/databases`
This tool creates the following files:
- `~/.cvdupdate/config.json`
- `~/.cvdupdate/databases/.cvd`
- `~/.cvdupdate/databases/-.cdiff`
- `~/.cvdupdate/logs/.log`
> _Tip_: You can set custom `database` and `logs` directories with the `cvd config set` command. It is likely you will want to customize the `database` directory to point to your HTTP server's `www` directory (or equivalent). Bare in mind that if you already downloaded the databases to the old directory, you may want to move them to the new directory.
> _Important_: If you want to use a custom config path, you'll have to use it in every command. If you're fine with having it go in `~/.cvdupdate/config.json`, don't worry about it.
## Additional Usage
### Get familiar with the tool
Familiarize yourself with the various commands using the `--help` option.
```bash
cvd --help
cvd config --help
cvd update --help
cvd clean --help
```
Print out the current list of databases.
```bash
cvd list -V
```
Print out the config to see what it looks like.
```bash
cvd config show
```
### Do an update
Do an update, use "verbose mode" to so you can get a feel for how it works.
```bash
cvd update -V
```
List out the databases again:
```bash
cvd list -V
```
The print out the config again so you can see what's changed.
```bash
cvd config show
```
And maybe take a peek in the database directory as well to see it for yourself.
```bash
ls ~/.cvdupdate/database
```
Have a look at the logs if you wish.
```bash
ls ~/.cvdupdate/logs
cat ~/.cvdupdate/logs/*
```
### Serve it up, Test out FreshClam
Test out your mirror with FreshClam on the same computer.
This tool includes a `--serve` feature that will host the current database directory on http://localhost (default port: 8000).
You can test it by running `freshclam` or `freshclam.exe` locally, where you've configured `freshclam.conf` with:
```
DatabaseMirror http://localhost:8000
```
## Contribute
We'd love your help. There are many ways to contribute!
### Community
Join the ClamAV community on the [ClamAV Discord chat server](https://discord.gg/sGaxA5Q).
### Report issues
If you find an issue with CVD-Update or the CVD-Update documentation, please submit an issue to our [GitHub issue tracker](https://github.com/Cisco-Talos/cvdupdate/issues). Before you submit, please check to if someone else has already reported the issue.
### Development
If you find a bug and you're able to craft a fix yourself, consider submitting the fix in a [pull request](https://github.com/Cisco-Talos/cvdupdate/pulls). Your help will be greatly appreciated.
If you want to contribute to the project and don't have anything specific in mind, please check out our [issue tracker](https://github.com/Cisco-Talos/cvdupdate/issues). Perhaps you'll be able to fix a bug or add a cool new feature.
_By submitting a contribution to the project, you acknowledge and agree to assign Cisco Systems, Inc the copyright for the contribution. If you submit a significant contribution such as a new feature or capability or a large amount of code, you may be asked to sign a contributors license agreement comfirming that Cisco will have copyright license and patent license and that you are authorized to contribute the code._
#### Development Set-up
The following steps are intended to help users that wish to contribute to development of the CVD-Update project get started.
1. Create a fork of the [CVD-Update git repository](https://github.com/Cisco-Talos/cvdupdate), and then clone your fork to a local directory.
For example:
```bash
git clone https://github.com//cvdupdate.git
```
2. Make sure CVD-Update is not already installed. If it is, remove it.
```bash
python3 -m pip uninstall cvdupdate
```
3. Use pip to install CVD-Update in "edit" mode.
```bash
python3 -m pip install -e --user ./cvdupdate
```
Once installed in "edit" mode, any changes you make to your clone of the CVD-Update code will be immediately usable simply by running the `cvdupdate` / `cvd` commands.
### Conduct
This project has not selected a specific Code-of-Conduct document at this time. However, contributors are expected to behave in professional and respectful manner. Disrespectful or inappropriate behavior will not be tolerated.
## License
CVD-Update is licensed under the Apache License, Version 2.0 (the "License"). You may not use the CVD-Update project except in compliance with the License.
A copy of the license is located [here](LICENSE), and is also available online at [apache.org](http://www.apache.org/licenses/LICENSE-2.0).
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Description-Content-Type: text/markdown
cvdupdate-1.0.2/cvdupdate.egg-info/SOURCES.txt 0000644 0001751 0000164 00000000445 14037577401 021642 0 ustar runner docker 0000000 0000000 LICENSE
README.md
setup.py
cvdupdate/__init__.py
cvdupdate/__main__.py
cvdupdate/cvdupdate.py
cvdupdate.egg-info/PKG-INFO
cvdupdate.egg-info/SOURCES.txt
cvdupdate.egg-info/dependency_links.txt
cvdupdate.egg-info/entry_points.txt
cvdupdate.egg-info/requires.txt
cvdupdate.egg-info/top_level.txt cvdupdate-1.0.2/cvdupdate.egg-info/dependency_links.txt 0000644 0001751 0000164 00000000001 14037577401 024021 0 ustar runner docker 0000000 0000000
cvdupdate-1.0.2/cvdupdate.egg-info/entry_points.txt 0000644 0001751 0000164 00000000123 14037577401 023245 0 ustar runner docker 0000000 0000000 [console_scripts]
cvd = cvdupdate.__main__:cli
cvdupdate = cvdupdate.__main__:cli
cvdupdate-1.0.2/cvdupdate.egg-info/requires.txt 0000644 0001751 0000164 00000000120 14037577401 022344 0 ustar runner docker 0000000 0000000 click>=7.0
coloredlogs>=10.0
colorama
requests
dnspython>=2.1.0
rangehttpserver
cvdupdate-1.0.2/cvdupdate.egg-info/top_level.txt 0000644 0001751 0000164 00000000012 14037577401 022476 0 ustar runner docker 0000000 0000000 cvdupdate
cvdupdate-1.0.2/setup.cfg 0000644 0001751 0000164 00000000046 14037577401 016123 0 ustar runner docker 0000000 0000000 [egg_info]
tag_build =
tag_date = 0
cvdupdate-1.0.2/setup.py 0000644 0001751 0000164 00000002105 14037577371 016020 0 ustar runner docker 0000000 0000000 import setuptools
with open("README.md", "r") as fh:
long_description = fh.read()
setuptools.setup(
name="cvdupdate",
version="1.0.2",
author="The ClamAV Team",
author_email="clamav-bugs@external.cisco.com",
copyright="Copyright (C) 2021 Cisco Systems, Inc. and/or its affiliates. All rights reserved.",
description="ClamAV Private Database Mirror Updater Tool",
long_description=long_description,
long_description_content_type="text/markdown",
url="https://github.com/Cisco-Talos/cvdupdate",
packages=setuptools.find_packages(),
entry_points={
"console_scripts": [
"cvdupdate = cvdupdate.__main__:cli",
"cvd = cvdupdate.__main__:cli",
]
},
install_requires=[
"click>=7.0",
"coloredlogs>=10.0",
"colorama",
"requests",
"dnspython>=2.1.0",
"rangehttpserver",
],
classifiers=[
"Programming Language :: Python :: 3",
"License :: OSI Approved :: Apache Software License",
"Operating System :: OS Independent",
],
)