:dancer: This is a community driven project, we need your feedback.
##
CrowdSec is a free, modern & collaborative behavior detection engine, coupled with a global IP reputation network. It stacks on fail2ban's philosophy but is IPV6 compatible and 60x faster (Go vs Python), uses Grok patterns to parse logs and YAML scenario to identify behaviors. CrowdSec is engineered for modern Cloud / Containers / VM based infrastructures (by decoupling detection and remediation). Once detected you can remedy threats with various bouncers (firewall block, nginx http 403, Captchas, etc.) while the aggressive IP can be sent to CrowdSec for curation before being shared among all users to further improve everyone's security. See [FAQ](https://doc.crowdsec.net/docs/faq) or read below for more.
## 2 mins install
Installing it through the [Package system](https://doc.crowdsec.net/docs/getting_started/install_crowdsec) of your OS is the easiest way to proceed.
Otherwise, you can install it from source.
### From package (Debian)
```sh
curl -s https://packagecloud.io/install/repositories/crowdsec/crowdsec/script.deb.sh | sudo bash
sudo apt-get update
sudo apt-get install crowdsec
```
### From package (rhel/centos/amazon linux)
```sh
curl -s https://packagecloud.io/install/repositories/crowdsec/crowdsec/script.rpm.sh | sudo bash
sudo yum install crowdsec
```
### From package (FreeBSD)
```
sudo pkg update
sudo pkg install crowdsec
```
### From source
```sh
wget https://github.com/crowdsecurity/crowdsec/releases/latest/download/crowdsec-release.tgz
tar xzvf crowdsec-release.tgz
cd crowdsec-v* && sudo ./wizard.sh -i
```
## :information_source: About the CrowdSec project
Crowdsec is an open-source, lightweight software, detecting peers with aggressive behaviors to prevent them from accessing your systems. Its user friendly design and assistance offers a low technical barrier of entry and nevertheless a high security gain.
The architecture is as follows :
Once an unwanted behavior is detected, deal with it through a [bouncer](https://hub.crowdsec.net/browse/#bouncers). The aggressive IP, scenario triggered and timestamp are sent for curation, to avoid poisoning & false positives. (This can be disabled). If verified, this IP is then redistributed to all CrowdSec users running the same scenario.
## Outnumbering hackers all together
By sharing the threat they faced, all users are protecting each-others (hence the name Crowd-Security). Crowdsec is designed for modern infrastructures, with its "*Detect Here, Remedy There*" approach, letting you analyse logs coming from several sources in one place and block threats at various levels (applicative, system, infrastructural) of your stack.
CrowdSec ships by default with scenarios (brute force, port scan, web scan, etc.) adapted for most context, but you can easily extend it by picking more of them from the **[HUB](https://hub.crowdsec.net)**. It is also easy to adapt an existing one or create one yourself.
## :point_right: What it is not
CrowdSec is not a SIEM, storing your logs (neither locally nor remotely). Your data are analyzed locally and forgotten.
Signals sent to the curation platform are limited to the very strict minimum: IP, Scenario, Timestamp. They are only used to allow the system to spot new rogue IPs, rule out false positives or poisoning attempts.
## :arrow_down: Install it !
Crowdsec is available for various platforms :
- [Use our debian repositories](https://doc.crowdsec.net/docs/getting_started/install_crowdsec) or the [official debian packages](https://packages.debian.org/search?keywords=crowdsec&searchon=names&suite=stable§ion=all)
- An [image](https://hub.docker.com/r/crowdsecurity/crowdsec) is available for docker
- [Prebuilt release packages](https://github.com/crowdsecurity/crowdsec/releases) are also available (suitable for `amd64`)
- You can as well [build it from source](https://doc.crowdsec.net/docs/user_guides/building)
Or look directly at [installation documentation](https://doc.crowdsec.net/docs/getting_started/install_crowdsec) for other methods and platforms.
## :tada: Key benefits
### Fast assisted installation, no technical barrier
Initial configuration is automated, providing functional out-of-the-box setup
### Out of the box detection
Baseline detection is effective out-of-the-box, no fine-tuning required (click to expand)
### Easy bouncer deployment
It's trivial to add bouncers to enforce decisions of crowdsec (click to expand)
### Easy dashboard access
It's easy to deploy a metabase interface to view your data simply with cscli (click to expand)
### Hot & Cold logs
Process cold logs, for forensic, tests and chasing false-positives & false negatives (click to expand)
## đĻ About this repository
This repository contains the code for the two main components of crowdsec :
- `crowdsec` : the daemon a-la-fail2ban that can read, parse, enrich and apply heuristics to logs. This is the component in charge of "detecting" the attacks
- `cscli` : the cli tool mainly used to interact with crowdsec : ban/unban/view current bans, enable/disable parsers and scenarios.
## Contributing
If you wish to contribute to the core of crowdsec, you are welcome to open a PR in this repository.
If you wish to add a new parser, scenario or collection, please open a PR in the [hub repository](https://github.com/crowdsecurity/hub).
If you wish to contribute to the documentation, please open a PR in the [documentation repository](http://github.com/crowdsecurity/crowdsec-docs).
crowdsec-1.4.6/SECURITY.md 0000664 0000000 0000000 00000003257 14371172257 0015144 0 ustar 00root root 0000000 0000000 # Security Policy
## Scope
This security policy applies to :
- Crowdsec agent
- Crowdsec Local API
- Crowdsec bouncers **developed and maintained** by the Crowdsec team [1]
Reports regarding developements of community members that are not part of the crowdsecurity organization will be thoroughly investigated nonetheless.
[1] Projects developed and maintained by the Crowdsec team are under the **crowdsecurity** github organization. Bouncers developed by community members that are not part of the Crowdsec organization are explictely excluded.
## Reporting a Vulnerability
We are extremely grateful to security researchers and users that report vulnerabilities regarding the Crowdsec project. All reports are thoroughly investigated by members of the Crowdsec organization.
You can email the private [security@crowdsec.net](mailto:security@crowdsec.net) list with the security details and the details expected for [all Crowdsec bug reports](https://github.com/crowdsecurity/crowdsec/blob/master/.github/ISSUE_TEMPLATE/bug_report.md).
You may encrypt your email to this list using the GPG key of the [Security team](https://doc.crowdsec.net/docs/next/contact_team). Encryption using GPG is NOT required to make a disclosure.
## When Should I Report a Vulnerability?
- You think you discovered a potential security vulnerability in Crowdsec
- You are unsure how a vulnerability affects Crowdsec
- You think you discovered a vulnerability in another project that Crowdsec depends on
For projects with their own vulnerability reporting and disclosure process, please report it directly there.
crowdsec-1.4.6/azure-pipelines.yml 0000664 0000000 0000000 00000012031 14371172257 0017200 0 ustar 00root root 0000000 0000000 trigger:
tags:
include:
- "v*"
exclude:
- "v*freebsd"
branches:
exclude:
- "*"
pr: none
pool:
vmImage: windows-latest
stages:
- stage: Build
jobs:
- job:
displayName: "Build"
steps:
- task: DotNetCoreCLI@2
displayName: "Install SignClient"
inputs:
command: 'custom'
custom: 'tool'
arguments: 'install --global SignClient --version 1.3.155'
- task: GoTool@0
displayName: "Install Go 1.19"
inputs:
version: '1.19'
- pwsh: |
choco install -y jq
choco install -y make
displayName: "Install builds deps"
- task: PowerShell@2
inputs:
targetType: 'inline'
pwsh: true
#we are not calling make windows_installer because we want to sign the binaries before they are added to the MSI
script: |
make build
- task: AzureKeyVault@2
inputs:
azureSubscription: 'Azure subscription 1(8a93ab40-7e99-445e-ad47-0f6a3e2ef546)'
KeyVaultName: 'CodeSigningSecrets'
SecretsFilter: 'CodeSigningUser,CodeSigningPassword'
RunAsPreJob: false
- task: DownloadSecureFile@1
inputs:
secureFile: appsettings.json
- pwsh: |
SignClient.exe Sign --name "crowdsec-binaries" `
--input "**/*.exe" --config (Join-Path -Path $(Agent.TempDirectory) -ChildPath "appsettings.json") `
--user $(CodeSigningUser) --secret '$(CodeSigningPassword)'
displayName: "Sign Crowdsec binaries + plugins"
- pwsh: |
$build_version=$env:BUILD_SOURCEBRANCHNAME
if ($build_version.StartsWith("v"))
{
$build_version = $build_version.Substring(1)
}
if ($build_version.Contains("-"))
{
$build_version = $build_version.Substring(0, $build_version.IndexOf("-"))
}
Write-Host "##vso[task.setvariable variable=BuildVersion;isOutput=true]$build_version"
displayName: GetCrowdsecVersion
name: GetCrowdsecVersion
- pwsh: |
.\make_installer.ps1 -version '$(GetCrowdsecVersion.BuildVersion)'
displayName: "Build Crowdsec MSI"
name: BuildMSI
- pwsh: |
.\make_chocolatey.ps1 -version '$(GetCrowdsecVersion.BuildVersion)'
displayName: "Build Chocolatey nupkg"
- pwsh: |
SignClient.exe Sign --name "crowdsec-msi" `
--input "*.msi" --config (Join-Path -Path $(Agent.TempDirectory) -ChildPath "appsettings.json") `
--user $(CodeSigningUser) --secret '$(CodeSigningPassword)'
displayName: "Sign Crowdsec MSI"
- task: PublishBuildArtifacts@1
inputs:
PathtoPublish: '$(Build.Repository.LocalPath)\\crowdsec_$(GetCrowdsecVersion.BuildVersion).msi'
ArtifactName: 'crowdsec.msi'
publishLocation: 'Container'
displayName: "Upload MSI artifact"
- task: PublishBuildArtifacts@1
inputs:
PathtoPublish: '$(Build.Repository.LocalPath)\\windows\\Chocolatey\\crowdsec\\crowdsec.$(GetCrowdsecVersion.BuildVersion).nupkg'
ArtifactName: 'crowdsec.nupkg'
publishLocation: 'Container'
displayName: "Upload nupkg artifact"
- stage: Publish
dependsOn: Build
jobs:
- deployment: "Publish"
displayName: "Publish to GitHub"
environment: github
strategy:
runOnce:
deploy:
steps:
- bash: |
tag=$(curl -H "Accept: application/vnd.github.v3+json" https://api.github.com/repos/crowdsecurity/crowdsec/releases | jq -r '. | map(select(.prerelease==true)) | sort_by(.created_at) | reverse | .[0].tag_name')
echo "##vso[task.setvariable variable=LatestPreRelease;isOutput=true]$tag"
name: GetLatestPrelease
- task: GitHubRelease@1
inputs:
gitHubConnection: "github.com_blotus"
repositoryName: '$(Build.Repository.Name)'
action: 'edit'
tag: '$(GetLatestPrelease.LatestPreRelease)'
assetUploadMode: 'replace'
addChangeLog: false
isPreRelease: true #we force prerelease because the pipeline is invoked on tag creation, which happens when we do a prerelease
#the .. is an ugly hack, but I can't find the var that gives D:\a\1 ...
assets: |
$(Build.ArtifactStagingDirectory)\..\crowdsec.msi/*.msi
$(Build.ArtifactStagingDirectory)\..\crowdsec.nupkg/*.nupkg
condition: ne(variables['GetLatestPrelease.LatestPreRelease'], '')
crowdsec-1.4.6/cmd/ 0000775 0000000 0000000 00000000000 14371172257 0014107 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/cmd/crowdsec-cli/ 0000775 0000000 0000000 00000000000 14371172257 0016465 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/cmd/crowdsec-cli/Makefile 0000664 0000000 0000000 00000002144 14371172257 0020126 0 ustar 00root root 0000000 0000000 ifeq ($(OS), Windows_NT)
SHELL := pwsh.exe
.SHELLFLAGS := -NoProfile -Command
EXT = .exe
endif
# Go parameters
GOCMD = go
GOBUILD = $(GOCMD) build
GOCLEAN = $(GOCMD) clean
GOTEST = $(GOCMD) test
GOGET = $(GOCMD) get
BINARY_NAME = cscli$(EXT)
# names longer than 15 chars break 'pgrep'
BINARY_NAME_COVER = $(BINARY_NAME).cover
PREFIX ?= "/"
BIN_PREFIX = $(PREFIX)"/usr/local/bin/"
.PHONY: all
all: clean build
build: clean
$(GOBUILD) $(LD_OPTS) $(BUILD_VENDOR_FLAGS) -o $(BINARY_NAME)
build-bincover: clean
$(GOTEST) . -tags testrunmain -coverpkg=$(go list github.com/crowdsecurity/crowdsec/... | grep -v -e 'pkg/database' -e 'plugins/notifications' -e 'pkg/protobufs' -e 'pkg/cwversions' -e 'pkg/cstest' -e 'pkg/models') -covermode=atomic $(LD_OPTS) -c -o $(BINARY_NAME_COVER)
.PHONY: install
install: install-conf install-bin
install-conf:
install-bin:
@install -v -m 755 -D "$(BINARY_NAME)" "$(BIN_PREFIX)/$(BINARY_NAME)" || exit
uninstall:
@$(RM) $(CSCLI_CONFIG) $(WIN_IGNORE_ERR)
@$(RM) $(BIN_PREFIX)$(BINARY_NAME) $(WIN_IGNORE_ERR)
clean:
@$(RM) $(BINARY_NAME) $(BINARY_NAME_COVER) $(WIN_IGNORE_ERR)
crowdsec-1.4.6/cmd/crowdsec-cli/alerts.go 0000664 0000000 0000000 00000035741 14371172257 0020320 0 ustar 00root root 0000000 0000000 package main
import (
"context"
"encoding/csv"
"encoding/json"
"fmt"
"net/url"
"os"
"strconv"
"strings"
"github.com/fatih/color"
"github.com/go-openapi/strfmt"
"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"gopkg.in/yaml.v2"
"github.com/crowdsecurity/crowdsec/pkg/apiclient"
"github.com/crowdsecurity/crowdsec/pkg/cwversion"
"github.com/crowdsecurity/crowdsec/pkg/database"
"github.com/crowdsecurity/crowdsec/pkg/models"
)
var printMachine bool
var limit *int
func DecisionsFromAlert(alert *models.Alert) string {
ret := ""
var decMap = make(map[string]int)
for _, decision := range alert.Decisions {
k := *decision.Type
if *decision.Simulated {
k = fmt.Sprintf("(simul)%s", k)
}
v := decMap[k]
decMap[k] = v + 1
}
for k, v := range decMap {
if len(ret) > 0 {
ret += " "
}
ret += fmt.Sprintf("%s:%d", k, v)
}
return ret
}
func AlertsToTable(alerts *models.GetAlertsResponse, printMachine bool) error {
if csConfig.Cscli.Output == "raw" {
csvwriter := csv.NewWriter(os.Stdout)
header := []string{"id", "scope", "value", "reason", "country", "as", "decisions", "created_at"}
if printMachine {
header = append(header, "machine")
}
err := csvwriter.Write(header)
if err != nil {
return err
}
for _, alertItem := range *alerts {
row := []string{
fmt.Sprintf("%d", alertItem.ID),
*alertItem.Source.Scope,
*alertItem.Source.Value,
*alertItem.Scenario,
alertItem.Source.Cn,
alertItem.Source.GetAsNumberName(),
DecisionsFromAlert(alertItem),
*alertItem.StartAt,
}
if printMachine {
row = append(row, alertItem.MachineID)
}
err := csvwriter.Write(row)
if err != nil {
return err
}
}
csvwriter.Flush()
} else if csConfig.Cscli.Output == "json" {
x, _ := json.MarshalIndent(alerts, "", " ")
fmt.Printf("%s", string(x))
} else if csConfig.Cscli.Output == "human" {
if len(*alerts) == 0 {
fmt.Println("No active alerts")
return nil
}
alertsTable(color.Output, alerts, printMachine)
}
return nil
}
func DisplayOneAlert(alert *models.Alert, withDetail bool) error {
if csConfig.Cscli.Output == "human" {
fmt.Printf("\n################################################################################################\n\n")
scopeAndValue := *alert.Source.Scope
if *alert.Source.Value != "" {
scopeAndValue += ":" + *alert.Source.Value
}
fmt.Printf(" - ID : %d\n", alert.ID)
fmt.Printf(" - Date : %s\n", alert.CreatedAt)
fmt.Printf(" - Machine : %s\n", alert.MachineID)
fmt.Printf(" - Simulation : %v\n", *alert.Simulated)
fmt.Printf(" - Reason : %s\n", *alert.Scenario)
fmt.Printf(" - Events Count : %d\n", *alert.EventsCount)
fmt.Printf(" - Scope:Value: %s\n", scopeAndValue)
fmt.Printf(" - Country : %s\n", alert.Source.Cn)
fmt.Printf(" - AS : %s\n", alert.Source.AsName)
fmt.Printf(" - Begin : %s\n", *alert.StartAt)
fmt.Printf(" - End : %s\n\n", *alert.StopAt)
alertDecisionsTable(color.Output, alert)
if withDetail {
fmt.Printf("\n - Events :\n")
for _, event := range alert.Events {
alertEventTable(color.Output, event)
}
}
}
return nil
}
func NewAlertsCmd() *cobra.Command {
/* ---- ALERTS COMMAND */
var cmdAlerts = &cobra.Command{
Use: "alerts [action]",
Short: "Manage alerts",
Args: cobra.MinimumNArgs(1),
DisableAutoGenTag: true,
PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
var err error
if err := csConfig.LoadAPIClient(); err != nil {
return errors.Wrap(err, "loading api client")
}
apiURL, err := url.Parse(csConfig.API.Client.Credentials.URL)
if err != nil {
return errors.Wrapf(err, "parsing api url %s", apiURL)
}
Client, err = apiclient.NewClient(&apiclient.Config{
MachineID: csConfig.API.Client.Credentials.Login,
Password: strfmt.Password(csConfig.API.Client.Credentials.Password),
UserAgent: fmt.Sprintf("crowdsec/%s", cwversion.VersionStr()),
URL: apiURL,
VersionPrefix: "v1",
})
if err != nil {
return errors.Wrap(err, "new api client")
}
return nil
},
}
var alertListFilter = apiclient.AlertsListOpts{
ScopeEquals: new(string),
ValueEquals: new(string),
ScenarioEquals: new(string),
IPEquals: new(string),
RangeEquals: new(string),
Since: new(string),
Until: new(string),
TypeEquals: new(string),
IncludeCAPI: new(bool),
}
limit = new(int)
contained := new(bool)
var cmdAlertsList = &cobra.Command{
Use: "list [filters]",
Short: "List alerts",
Example: `cscli alerts list
cscli alerts list --ip 1.2.3.4
cscli alerts list --range 1.2.3.0/24
cscli alerts list -s crowdsecurity/ssh-bf
cscli alerts list --type ban`,
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
var err error
if err := manageCliDecisionAlerts(alertListFilter.IPEquals, alertListFilter.RangeEquals,
alertListFilter.ScopeEquals, alertListFilter.ValueEquals); err != nil {
printHelp(cmd)
log.Fatalf("%s", err)
}
if limit != nil {
alertListFilter.Limit = limit
}
if *alertListFilter.Until == "" {
alertListFilter.Until = nil
} else if strings.HasSuffix(*alertListFilter.Until, "d") {
/*time.ParseDuration support hours 'h' as bigger unit, let's make the user's life easier*/
realDuration := strings.TrimSuffix(*alertListFilter.Until, "d")
days, err := strconv.Atoi(realDuration)
if err != nil {
printHelp(cmd)
log.Fatalf("Can't parse duration %s, valid durations format: 1d, 4h, 4h15m", *alertListFilter.Until)
}
*alertListFilter.Until = fmt.Sprintf("%d%s", days*24, "h")
}
if *alertListFilter.Since == "" {
alertListFilter.Since = nil
} else if strings.HasSuffix(*alertListFilter.Since, "d") {
/*time.ParseDuration support hours 'h' as bigger unit, let's make the user's life easier*/
realDuration := strings.TrimSuffix(*alertListFilter.Since, "d")
days, err := strconv.Atoi(realDuration)
if err != nil {
printHelp(cmd)
log.Fatalf("Can't parse duration %s, valid durations format: 1d, 4h, 4h15m", *alertListFilter.Since)
}
*alertListFilter.Since = fmt.Sprintf("%d%s", days*24, "h")
}
if *alertListFilter.IncludeCAPI {
*alertListFilter.Limit = 0
}
if *alertListFilter.TypeEquals == "" {
alertListFilter.TypeEquals = nil
}
if *alertListFilter.ScopeEquals == "" {
alertListFilter.ScopeEquals = nil
}
if *alertListFilter.ValueEquals == "" {
alertListFilter.ValueEquals = nil
}
if *alertListFilter.ScenarioEquals == "" {
alertListFilter.ScenarioEquals = nil
}
if *alertListFilter.IPEquals == "" {
alertListFilter.IPEquals = nil
}
if *alertListFilter.RangeEquals == "" {
alertListFilter.RangeEquals = nil
}
if contained != nil && *contained {
alertListFilter.Contains = new(bool)
}
alerts, _, err := Client.Alerts.List(context.Background(), alertListFilter)
if err != nil {
log.Fatalf("Unable to list alerts : %v", err)
}
err = AlertsToTable(alerts, printMachine)
if err != nil {
log.Fatalf("unable to list alerts : %v", err)
}
},
}
cmdAlertsList.Flags().SortFlags = false
cmdAlertsList.Flags().BoolVarP(alertListFilter.IncludeCAPI, "all", "a", false, "Include decisions from Central API")
cmdAlertsList.Flags().StringVar(alertListFilter.Until, "until", "", "restrict to alerts older than until (ie. 4h, 30d)")
cmdAlertsList.Flags().StringVar(alertListFilter.Since, "since", "", "restrict to alerts newer than since (ie. 4h, 30d)")
cmdAlertsList.Flags().StringVarP(alertListFilter.IPEquals, "ip", "i", "", "restrict to alerts from this source ip (shorthand for --scope ip --value )")
cmdAlertsList.Flags().StringVarP(alertListFilter.ScenarioEquals, "scenario", "s", "", "the scenario (ie. crowdsecurity/ssh-bf)")
cmdAlertsList.Flags().StringVarP(alertListFilter.RangeEquals, "range", "r", "", "restrict to alerts from this range (shorthand for --scope range --value )")
cmdAlertsList.Flags().StringVar(alertListFilter.TypeEquals, "type", "", "restrict to alerts with given decision type (ie. ban, captcha)")
cmdAlertsList.Flags().StringVar(alertListFilter.ScopeEquals, "scope", "", "restrict to alerts of this scope (ie. ip,range)")
cmdAlertsList.Flags().StringVarP(alertListFilter.ValueEquals, "value", "v", "", "the value to match for in the specified scope")
cmdAlertsList.Flags().BoolVar(contained, "contained", false, "query decisions contained by range")
cmdAlertsList.Flags().BoolVarP(&printMachine, "machine", "m", false, "print machines that sent alerts")
cmdAlertsList.Flags().IntVarP(limit, "limit", "l", 50, "limit size of alerts list table (0 to view all alerts)")
cmdAlerts.AddCommand(cmdAlertsList)
var ActiveDecision *bool
var AlertDeleteAll bool
var delAlertByID string
var alertDeleteFilter = apiclient.AlertsDeleteOpts{
ScopeEquals: new(string),
ValueEquals: new(string),
ScenarioEquals: new(string),
IPEquals: new(string),
RangeEquals: new(string),
}
var cmdAlertsDelete = &cobra.Command{
Use: "delete [filters] [--all]",
Short: `Delete alerts
/!\ This command can be use only on the same machine than the local API.`,
Example: `cscli alerts delete --ip 1.2.3.4
cscli alerts delete --range 1.2.3.0/24
cscli alerts delete -s crowdsecurity/ssh-bf"`,
DisableAutoGenTag: true,
Aliases: []string{"remove"},
Args: cobra.ExactArgs(0),
PreRun: func(cmd *cobra.Command, args []string) {
if AlertDeleteAll {
return
}
if *alertDeleteFilter.ScopeEquals == "" && *alertDeleteFilter.ValueEquals == "" &&
*alertDeleteFilter.ScenarioEquals == "" && *alertDeleteFilter.IPEquals == "" &&
*alertDeleteFilter.RangeEquals == "" && delAlertByID == "" {
_ = cmd.Usage()
log.Fatalln("At least one filter or --all must be specified")
}
},
Run: func(cmd *cobra.Command, args []string) {
var err error
if !AlertDeleteAll {
if err := manageCliDecisionAlerts(alertDeleteFilter.IPEquals, alertDeleteFilter.RangeEquals,
alertDeleteFilter.ScopeEquals, alertDeleteFilter.ValueEquals); err != nil {
printHelp(cmd)
log.Fatalf("%s", err)
}
if ActiveDecision != nil {
alertDeleteFilter.ActiveDecisionEquals = ActiveDecision
}
if *alertDeleteFilter.ScopeEquals == "" {
alertDeleteFilter.ScopeEquals = nil
}
if *alertDeleteFilter.ValueEquals == "" {
alertDeleteFilter.ValueEquals = nil
}
if *alertDeleteFilter.ScenarioEquals == "" {
alertDeleteFilter.ScenarioEquals = nil
}
if *alertDeleteFilter.IPEquals == "" {
alertDeleteFilter.IPEquals = nil
}
if *alertDeleteFilter.RangeEquals == "" {
alertDeleteFilter.RangeEquals = nil
}
if contained != nil && *contained {
alertDeleteFilter.Contains = new(bool)
}
limit := 0
alertDeleteFilter.Limit = &limit
} else {
limit := 0
alertDeleteFilter = apiclient.AlertsDeleteOpts{Limit: &limit}
}
var alerts *models.DeleteAlertsResponse
if delAlertByID == "" {
alerts, _, err = Client.Alerts.Delete(context.Background(), alertDeleteFilter)
if err != nil {
log.Fatalf("Unable to delete alerts : %v", err)
}
} else {
alerts, _, err = Client.Alerts.DeleteOne(context.Background(), delAlertByID)
if err != nil {
log.Fatalf("Unable to delete alert : %v", err)
}
}
log.Infof("%s alert(s) deleted", alerts.NbDeleted)
},
}
cmdAlertsDelete.Flags().SortFlags = false
cmdAlertsDelete.Flags().StringVar(alertDeleteFilter.ScopeEquals, "scope", "", "the scope (ie. ip,range)")
cmdAlertsDelete.Flags().StringVarP(alertDeleteFilter.ValueEquals, "value", "v", "", "the value to match for in the specified scope")
cmdAlertsDelete.Flags().StringVarP(alertDeleteFilter.ScenarioEquals, "scenario", "s", "", "the scenario (ie. crowdsecurity/ssh-bf)")
cmdAlertsDelete.Flags().StringVarP(alertDeleteFilter.IPEquals, "ip", "i", "", "Source ip (shorthand for --scope ip --value )")
cmdAlertsDelete.Flags().StringVarP(alertDeleteFilter.RangeEquals, "range", "r", "", "Range source ip (shorthand for --scope range --value )")
cmdAlertsDelete.Flags().StringVar(&delAlertByID, "id", "", "alert ID")
cmdAlertsDelete.Flags().BoolVarP(&AlertDeleteAll, "all", "a", false, "delete all alerts")
cmdAlertsDelete.Flags().BoolVar(contained, "contained", false, "query decisions contained by range")
cmdAlerts.AddCommand(cmdAlertsDelete)
var details bool
var cmdAlertsInspect = &cobra.Command{
Use: `inspect "alert_id"`,
Short: `Show info about an alert`,
Example: `cscli alerts inspect 123`,
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
if len(args) == 0 {
printHelp(cmd)
return
}
for _, alertID := range args {
id, err := strconv.Atoi(alertID)
if err != nil {
log.Fatalf("bad alert id %s", alertID)
continue
}
alert, _, err := Client.Alerts.GetByID(context.Background(), id)
if err != nil {
log.Fatalf("can't find alert with id %s: %s", alertID, err)
}
switch csConfig.Cscli.Output {
case "human":
if err := DisplayOneAlert(alert, details); err != nil {
continue
}
case "json":
data, err := json.MarshalIndent(alert, "", " ")
if err != nil {
log.Fatalf("unable to marshal alert with id %s: %s", alertID, err)
}
fmt.Printf("%s\n", string(data))
case "raw":
data, err := yaml.Marshal(alert)
if err != nil {
log.Fatalf("unable to marshal alert with id %s: %s", alertID, err)
}
fmt.Printf("%s\n", string(data))
}
}
},
}
cmdAlertsInspect.Flags().SortFlags = false
cmdAlertsInspect.Flags().BoolVarP(&details, "details", "d", false, "show alerts with events")
cmdAlerts.AddCommand(cmdAlertsInspect)
var maxItems int
var maxAge string
var cmdAlertsFlush = &cobra.Command{
Use: `flush`,
Short: `Flush alerts
/!\ This command can be used only on the same machine than the local API`,
Example: `cscli alerts flush --max-items 1000 --max-age 7d`,
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
var err error
if err := csConfig.LoadAPIServer(); err != nil || csConfig.DisableAPI {
log.Fatal("Local API is disabled, please run this command on the local API machine")
}
if err := csConfig.LoadDBConfig(); err != nil {
log.Fatal(err)
}
dbClient, err = database.NewClient(csConfig.DbConfig)
if err != nil {
log.Fatalf("unable to create new database client: %s", err)
}
log.Info("Flushing alerts. !! This may take a long time !!")
err = dbClient.FlushAlerts(maxAge, maxItems)
if err != nil {
log.Fatalf("unable to flush alerts: %s", err)
}
log.Info("Alerts flushed")
},
}
cmdAlertsFlush.Flags().SortFlags = false
cmdAlertsFlush.Flags().IntVar(&maxItems, "max-items", 5000, "Maximum number of alert items to keep in the database")
cmdAlertsFlush.Flags().StringVar(&maxAge, "max-age", "7d", "Maximum age of alert items to keep in the database")
cmdAlerts.AddCommand(cmdAlertsFlush)
return cmdAlerts
}
crowdsec-1.4.6/cmd/crowdsec-cli/alerts_table.go 0000664 0000000 0000000 00000004116 14371172257 0021457 0 ustar 00root root 0000000 0000000 package main
import (
"fmt"
"io"
"sort"
"strconv"
"time"
log "github.com/sirupsen/logrus"
"github.com/crowdsecurity/crowdsec/pkg/models"
)
func alertsTable(out io.Writer, alerts *models.GetAlertsResponse, printMachine bool) {
t := newTable(out)
t.SetRowLines(false)
header := []string{"ID", "value", "reason", "country", "as", "decisions", "created_at"}
if printMachine {
header = append(header, "machine")
}
t.SetHeaders(header...)
for _, alertItem := range *alerts {
displayVal := *alertItem.Source.Scope
if *alertItem.Source.Value != "" {
displayVal += ":" + *alertItem.Source.Value
}
row := []string{
strconv.Itoa(int(alertItem.ID)),
displayVal,
*alertItem.Scenario,
alertItem.Source.Cn,
alertItem.Source.GetAsNumberName(),
DecisionsFromAlert(alertItem),
*alertItem.StartAt,
}
if printMachine {
row = append(row, alertItem.MachineID)
}
t.AddRow(row...)
}
t.Render()
}
func alertDecisionsTable(out io.Writer, alert *models.Alert) {
foundActive := false
t := newTable(out)
t.SetRowLines(false)
t.SetHeaders("ID", "scope:value", "action", "expiration", "created_at")
for _, decision := range alert.Decisions {
parsedDuration, err := time.ParseDuration(*decision.Duration)
if err != nil {
log.Error(err)
}
expire := time.Now().UTC().Add(parsedDuration)
if time.Now().UTC().After(expire) {
continue
}
foundActive = true
scopeAndValue := *decision.Scope
if *decision.Value != "" {
scopeAndValue += ":" + *decision.Value
}
t.AddRow(
strconv.Itoa(int(decision.ID)),
scopeAndValue,
*decision.Type,
*decision.Duration,
alert.CreatedAt,
)
}
if foundActive {
fmt.Printf(" - Active Decisions :\n")
t.Render() // Send output
}
}
func alertEventTable(out io.Writer, event *models.Event) {
fmt.Fprintf(out, "\n- Date: %s\n", *event.Timestamp)
t := newTable(out)
t.SetHeaders("Key", "Value")
sort.Slice(event.Meta, func(i, j int) bool {
return event.Meta[i].Key < event.Meta[j].Key
})
for _, meta := range event.Meta {
t.AddRow(
meta.Key,
meta.Value,
)
}
t.Render() // Send output
}
crowdsec-1.4.6/cmd/crowdsec-cli/bouncers.go 0000664 0000000 0000000 00000012007 14371172257 0020634 0 ustar 00root root 0000000 0000000 package main
import (
"encoding/csv"
"encoding/json"
"fmt"
"io"
"time"
"github.com/fatih/color"
"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
middlewares "github.com/crowdsecurity/crowdsec/pkg/apiserver/middlewares/v1"
"github.com/crowdsecurity/crowdsec/pkg/database"
"github.com/crowdsecurity/crowdsec/pkg/types"
)
var keyIP string
var keyLength int
var key string
func getBouncers(out io.Writer, dbClient *database.Client) error {
bouncers, err := dbClient.ListBouncers()
if err != nil {
return fmt.Errorf("unable to list bouncers: %s", err)
}
if csConfig.Cscli.Output == "human" {
getBouncersTable(out, bouncers)
} else if csConfig.Cscli.Output == "json" {
enc := json.NewEncoder(out)
enc.SetIndent("", " ")
if err := enc.Encode(bouncers); err != nil {
return errors.Wrap(err, "failed to unmarshal")
}
return nil
} else if csConfig.Cscli.Output == "raw" {
csvwriter := csv.NewWriter(out)
err := csvwriter.Write([]string{"name", "ip", "revoked", "last_pull", "type", "version", "auth_type"})
if err != nil {
return errors.Wrap(err, "failed to write raw header")
}
for _, b := range bouncers {
var revoked string
if !b.Revoked {
revoked = "validated"
} else {
revoked = "pending"
}
err := csvwriter.Write([]string{b.Name, b.IPAddress, revoked, b.LastPull.Format(time.RFC3339), b.Type, b.Version, b.AuthType})
if err != nil {
return errors.Wrap(err, "failed to write raw")
}
}
csvwriter.Flush()
}
return nil
}
func NewBouncersCmd() *cobra.Command {
/* ---- DECISIONS COMMAND */
var cmdBouncers = &cobra.Command{
Use: "bouncers [action]",
Short: "Manage bouncers [requires local API]",
Long: `To list/add/delete bouncers.
Note: This command requires database direct access, so is intended to be run on Local API/master.
`,
Args: cobra.MinimumNArgs(1),
Aliases: []string{"bouncer"},
DisableAutoGenTag: true,
PersistentPreRun: func(cmd *cobra.Command, args []string) {
var err error
if err := csConfig.LoadAPIServer(); err != nil || csConfig.DisableAPI {
log.Fatal("Local API is disabled, please run this command on the local API machine")
}
if err := csConfig.LoadDBConfig(); err != nil {
log.Fatal(err)
}
dbClient, err = database.NewClient(csConfig.DbConfig)
if err != nil {
log.Fatalf("unable to create new database client: %s", err)
}
},
}
var cmdBouncersList = &cobra.Command{
Use: "list",
Short: "List bouncers",
Long: `List bouncers`,
Example: `cscli bouncers list`,
Args: cobra.ExactArgs(0),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, arg []string) {
err := getBouncers(color.Output, dbClient)
if err != nil {
log.Fatalf("unable to list bouncers: %s", err)
}
},
}
cmdBouncers.AddCommand(cmdBouncersList)
var cmdBouncersAdd = &cobra.Command{
Use: "add MyBouncerName [--length 16]",
Short: "add bouncer",
Long: `add bouncer`,
Example: fmt.Sprintf(`cscli bouncers add MyBouncerName
cscli bouncers add MyBouncerName -l 24
cscli bouncers add MyBouncerName -k %s`, generatePassword(32)),
Args: cobra.ExactArgs(1),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, arg []string) {
keyName := arg[0]
var apiKey string
var err error
if keyName == "" {
log.Fatalf("Please provide a name for the api key")
}
apiKey = key
if key == "" {
apiKey, err = middlewares.GenerateAPIKey(keyLength)
}
if err != nil {
log.Fatalf("unable to generate api key: %s", err)
}
_, err = dbClient.CreateBouncer(keyName, keyIP, middlewares.HashSHA512(apiKey), types.ApiKeyAuthType)
if err != nil {
log.Fatalf("unable to create bouncer: %s", err)
}
if csConfig.Cscli.Output == "human" {
fmt.Printf("Api key for '%s':\n\n", keyName)
fmt.Printf(" %s\n\n", apiKey)
fmt.Print("Please keep this key since you will not be able to retrieve it!\n")
} else if csConfig.Cscli.Output == "raw" {
fmt.Printf("%s", apiKey)
} else if csConfig.Cscli.Output == "json" {
j, err := json.Marshal(apiKey)
if err != nil {
log.Fatalf("unable to marshal api key")
}
fmt.Printf("%s", string(j))
}
},
}
cmdBouncersAdd.Flags().IntVarP(&keyLength, "length", "l", 16, "length of the api key")
cmdBouncersAdd.Flags().StringVarP(&key, "key", "k", "", "api key for the bouncer")
cmdBouncers.AddCommand(cmdBouncersAdd)
var cmdBouncersDelete = &cobra.Command{
Use: "delete MyBouncerName",
Short: "delete bouncer",
Args: cobra.MinimumNArgs(1),
Aliases: []string{"remove"},
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
for _, bouncerID := range args {
err := dbClient.DeleteBouncer(bouncerID)
if err != nil {
log.Fatalf("unable to delete bouncer '%s': %s", bouncerID, err)
}
log.Infof("bouncer '%s' deleted successfully", bouncerID)
}
},
}
cmdBouncers.AddCommand(cmdBouncersDelete)
return cmdBouncers
}
crowdsec-1.4.6/cmd/crowdsec-cli/bouncers_table.go 0000664 0000000 0000000 00000001546 14371172257 0022011 0 ustar 00root root 0000000 0000000 package main
import (
"io"
"time"
"github.com/aquasecurity/table"
"github.com/enescakir/emoji"
"github.com/crowdsecurity/crowdsec/pkg/database/ent"
)
func getBouncersTable(out io.Writer, bouncers []*ent.Bouncer) {
t := newLightTable(out)
t.SetHeaders("Name", "IP Address", "Valid", "Last API pull", "Type", "Version", "Auth Type")
t.SetHeaderAlignment(table.AlignLeft, table.AlignLeft, table.AlignLeft, table.AlignLeft, table.AlignLeft, table.AlignLeft)
t.SetAlignment(table.AlignLeft, table.AlignLeft, table.AlignLeft, table.AlignLeft, table.AlignLeft, table.AlignLeft)
for _, b := range bouncers {
var revoked string
if !b.Revoked {
revoked = emoji.CheckMark.String()
} else {
revoked = emoji.Prohibited.String()
}
t.AddRow(b.Name, b.IPAddress, revoked, b.LastPull.Format(time.RFC3339), b.Type, b.Version, b.AuthType)
}
t.Render()
}
crowdsec-1.4.6/cmd/crowdsec-cli/capi.go 0000664 0000000 0000000 00000013265 14371172257 0017737 0 ustar 00root root 0000000 0000000 package main
import (
"context"
"fmt"
"net/url"
"os"
"github.com/go-openapi/strfmt"
"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"gopkg.in/yaml.v2"
"github.com/crowdsecurity/crowdsec/pkg/apiclient"
"github.com/crowdsecurity/crowdsec/pkg/csconfig"
"github.com/crowdsecurity/crowdsec/pkg/cwhub"
"github.com/crowdsecurity/crowdsec/pkg/cwversion"
"github.com/crowdsecurity/crowdsec/pkg/models"
)
var CAPIURLPrefix string = "v2"
var CAPIBaseURL string = "https://api.crowdsec.net/"
var capiUserPrefix string
func NewCapiCmd() *cobra.Command {
var cmdCapi = &cobra.Command{
Use: "capi [action]",
Short: "Manage interaction with Central API (CAPI)",
Args: cobra.MinimumNArgs(1),
DisableAutoGenTag: true,
PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
if err := csConfig.LoadAPIServer(); err != nil || csConfig.DisableAPI {
return errors.Wrap(err, "Local API is disabled, please run this command on the local API machine")
}
if csConfig.API.Server.OnlineClient == nil {
log.Fatalf("no configuration for Central API in '%s'", *csConfig.FilePath)
}
return nil
},
}
var cmdCapiRegister = &cobra.Command{
Use: "register",
Short: "Register to Central API (CAPI)",
Args: cobra.MinimumNArgs(0),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
var err error
capiUser, err := generateID(capiUserPrefix)
if err != nil {
log.Fatalf("unable to generate machine id: %s", err)
}
password := strfmt.Password(generatePassword(passwordLength))
apiurl, err := url.Parse(CAPIBaseURL)
if err != nil {
log.Fatalf("unable to parse api url %s : %s", CAPIBaseURL, err)
}
_, err = apiclient.RegisterClient(&apiclient.Config{
MachineID: capiUser,
Password: password,
UserAgent: fmt.Sprintf("crowdsec/%s", cwversion.VersionStr()),
URL: apiurl,
VersionPrefix: CAPIURLPrefix,
}, nil)
if err != nil {
log.Fatalf("api client register ('%s'): %s", CAPIBaseURL, err)
}
log.Printf("Successfully registered to Central API (CAPI)")
var dumpFile string
if outputFile != "" {
dumpFile = outputFile
} else if csConfig.API.Server.OnlineClient.CredentialsFilePath != "" {
dumpFile = csConfig.API.Server.OnlineClient.CredentialsFilePath
} else {
dumpFile = ""
}
apiCfg := csconfig.ApiCredentialsCfg{
Login: capiUser,
Password: password.String(),
URL: CAPIBaseURL,
}
apiConfigDump, err := yaml.Marshal(apiCfg)
if err != nil {
log.Fatalf("unable to marshal api credentials: %s", err)
}
if dumpFile != "" {
err = os.WriteFile(dumpFile, apiConfigDump, 0600)
if err != nil {
log.Fatalf("write api credentials in '%s' failed: %s", dumpFile, err)
}
log.Printf("Central API credentials dumped to '%s'", dumpFile)
} else {
fmt.Printf("%s\n", string(apiConfigDump))
}
log.Warning(ReloadMessage())
},
}
cmdCapiRegister.Flags().StringVarP(&outputFile, "file", "f", "", "output file destination")
cmdCapiRegister.Flags().StringVar(&capiUserPrefix, "schmilblick", "", "set a schmilblick (use in tests only)")
if err := cmdCapiRegister.Flags().MarkHidden("schmilblick"); err != nil {
log.Fatalf("failed to hide flag: %s", err)
}
cmdCapi.AddCommand(cmdCapiRegister)
var cmdCapiStatus = &cobra.Command{
Use: "status",
Short: "Check status with the Central API (CAPI)",
Args: cobra.MinimumNArgs(0),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
var err error
if csConfig.API.Server == nil {
log.Fatalln("There is no configuration on 'api.server:'")
}
if csConfig.API.Server.OnlineClient == nil {
log.Fatalf("Please provide credentials for the Central API (CAPI) in '%s'", csConfig.API.Server.OnlineClient.CredentialsFilePath)
}
if csConfig.API.Server.OnlineClient.Credentials == nil {
log.Fatalf("no credentials for Central API (CAPI) in '%s'", csConfig.API.Server.OnlineClient.CredentialsFilePath)
}
password := strfmt.Password(csConfig.API.Server.OnlineClient.Credentials.Password)
apiurl, err := url.Parse(csConfig.API.Server.OnlineClient.Credentials.URL)
if err != nil {
log.Fatalf("parsing api url ('%s'): %s", csConfig.API.Server.OnlineClient.Credentials.URL, err)
}
if err := csConfig.LoadHub(); err != nil {
log.Fatal(err)
}
if err := cwhub.GetHubIdx(csConfig.Hub); err != nil {
log.Info("Run 'sudo cscli hub update' to get the hub index")
log.Fatalf("Failed to load hub index : %s", err)
}
scenarios, err := cwhub.GetInstalledScenariosAsString()
if err != nil {
log.Fatalf("failed to get scenarios : %s", err)
}
if len(scenarios) == 0 {
log.Fatalf("no scenarios installed, abort")
}
Client, err = apiclient.NewDefaultClient(apiurl, CAPIURLPrefix, fmt.Sprintf("crowdsec/%s", cwversion.VersionStr()), nil)
if err != nil {
log.Fatalf("init default client: %s", err)
}
t := models.WatcherAuthRequest{
MachineID: &csConfig.API.Server.OnlineClient.Credentials.Login,
Password: &password,
Scenarios: scenarios,
}
log.Infof("Loaded credentials from %s", csConfig.API.Server.OnlineClient.CredentialsFilePath)
log.Infof("Trying to authenticate with username %s on %s", csConfig.API.Server.OnlineClient.Credentials.Login, apiurl)
_, err = Client.Auth.AuthenticateWatcher(context.Background(), t)
if err != nil {
log.Fatalf("Failed to authenticate to Central API (CAPI) : %s", err)
}
log.Infof("You can successfully interact with Central API (CAPI)")
},
}
cmdCapi.AddCommand(cmdCapiStatus)
return cmdCapi
}
crowdsec-1.4.6/cmd/crowdsec-cli/collections.go 0000664 0000000 0000000 00000015730 14371172257 0021340 0 ustar 00root root 0000000 0000000 package main
import (
"fmt"
"github.com/fatih/color"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"github.com/crowdsecurity/crowdsec/pkg/cwhub"
)
func NewCollectionsCmd() *cobra.Command {
var cmdCollections = &cobra.Command{
Use: "collections [action]",
Short: "Manage collections from hub",
Long: `Install/Remove/Upgrade/Inspect collections from the CrowdSec Hub.`,
/*TBD fix help*/
Args: cobra.MinimumNArgs(1),
Aliases: []string{"collection"},
DisableAutoGenTag: true,
PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
if err := csConfig.LoadHub(); err != nil {
log.Fatal(err)
}
if csConfig.Hub == nil {
return fmt.Errorf("you must configure cli before interacting with hub")
}
if err := cwhub.SetHubBranch(); err != nil {
return fmt.Errorf("error while setting hub branch: %s", err)
}
if err := cwhub.GetHubIdx(csConfig.Hub); err != nil {
log.Info("Run 'sudo cscli hub update' to get the hub index")
log.Fatalf("Failed to get Hub index : %v", err)
}
return nil
},
PersistentPostRun: func(cmd *cobra.Command, args []string) {
if cmd.Name() == "inspect" || cmd.Name() == "list" {
return
}
log.Infof(ReloadMessage())
},
}
var ignoreError bool
var cmdCollectionsInstall = &cobra.Command{
Use: "install collection",
Short: "Install given collection(s)",
Long: `Fetch and install given collection(s) from hub`,
Example: `cscli collections install crowdsec/xxx crowdsec/xyz`,
Args: cobra.MinimumNArgs(1),
ValidArgsFunction: func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
return compAllItems(cwhub.COLLECTIONS, args, toComplete)
},
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
for _, name := range args {
t := cwhub.GetItem(cwhub.COLLECTIONS, name)
if t == nil {
nearestItem, score := GetDistance(cwhub.COLLECTIONS, name)
Suggest(cwhub.COLLECTIONS, name, nearestItem.Name, score, ignoreError)
continue
}
if err := cwhub.InstallItem(csConfig, name, cwhub.COLLECTIONS, forceAction, downloadOnly); err != nil {
if !ignoreError {
log.Fatalf("Error while installing '%s': %s", name, err)
}
log.Errorf("Error while installing '%s': %s", name, err)
}
}
},
}
cmdCollectionsInstall.PersistentFlags().BoolVarP(&downloadOnly, "download-only", "d", false, "Only download packages, don't enable")
cmdCollectionsInstall.PersistentFlags().BoolVar(&forceAction, "force", false, "Force install : Overwrite tainted and outdated files")
cmdCollectionsInstall.PersistentFlags().BoolVar(&ignoreError, "ignore", false, "Ignore errors when installing multiple collections")
cmdCollections.AddCommand(cmdCollectionsInstall)
var cmdCollectionsRemove = &cobra.Command{
Use: "remove collection",
Short: "Remove given collection(s)",
Long: `Remove given collection(s) from hub`,
Example: `cscli collections remove crowdsec/xxx crowdsec/xyz`,
Aliases: []string{"delete"},
DisableAutoGenTag: true,
ValidArgsFunction: func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
return compInstalledItems(cwhub.COLLECTIONS, args, toComplete)
},
Run: func(cmd *cobra.Command, args []string) {
if all {
cwhub.RemoveMany(csConfig, cwhub.COLLECTIONS, "", all, purge, forceAction)
return
}
if len(args) == 0 {
log.Fatal("Specify at least one collection to remove or '--all' flag.")
}
for _, name := range args {
if !forceAction {
item := cwhub.GetItem(cwhub.COLLECTIONS, name)
if item == nil {
log.Fatalf("unable to retrieve: %s\n", name)
}
if len(item.BelongsToCollections) > 0 {
log.Warningf("%s belongs to other collections :\n%s\n", name, item.BelongsToCollections)
log.Printf("Run 'sudo cscli collections remove %s --force' if you want to force remove this sub collection\n", name)
continue
}
}
cwhub.RemoveMany(csConfig, cwhub.COLLECTIONS, name, all, purge, forceAction)
}
},
}
cmdCollectionsRemove.PersistentFlags().BoolVar(&purge, "purge", false, "Delete source file too")
cmdCollectionsRemove.PersistentFlags().BoolVar(&forceAction, "force", false, "Force remove : Remove tainted and outdated files")
cmdCollectionsRemove.PersistentFlags().BoolVar(&all, "all", false, "Delete all the collections")
cmdCollections.AddCommand(cmdCollectionsRemove)
var cmdCollectionsUpgrade = &cobra.Command{
Use: "upgrade collection",
Short: "Upgrade given collection(s)",
Long: `Fetch and upgrade given collection(s) from hub`,
Example: `cscli collections upgrade crowdsec/xxx crowdsec/xyz`,
DisableAutoGenTag: true,
ValidArgsFunction: func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
return compInstalledItems(cwhub.COLLECTIONS, args, toComplete)
},
Run: func(cmd *cobra.Command, args []string) {
if all {
cwhub.UpgradeConfig(csConfig, cwhub.COLLECTIONS, "", forceAction)
} else {
if len(args) == 0 {
log.Fatalf("no target collection to upgrade")
}
for _, name := range args {
cwhub.UpgradeConfig(csConfig, cwhub.COLLECTIONS, name, forceAction)
}
}
},
}
cmdCollectionsUpgrade.PersistentFlags().BoolVarP(&all, "all", "a", false, "Upgrade all the collections")
cmdCollectionsUpgrade.PersistentFlags().BoolVar(&forceAction, "force", false, "Force upgrade : Overwrite tainted and outdated files")
cmdCollections.AddCommand(cmdCollectionsUpgrade)
var cmdCollectionsInspect = &cobra.Command{
Use: "inspect collection",
Short: "Inspect given collection",
Long: `Inspect given collection`,
Example: `cscli collections inspect crowdsec/xxx crowdsec/xyz`,
Args: cobra.MinimumNArgs(1),
DisableAutoGenTag: true,
ValidArgsFunction: func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
return compInstalledItems(cwhub.COLLECTIONS, args, toComplete)
},
Run: func(cmd *cobra.Command, args []string) {
for _, name := range args {
InspectItem(name, cwhub.COLLECTIONS)
}
},
}
cmdCollectionsInspect.PersistentFlags().StringVarP(&prometheusURL, "url", "u", "", "Prometheus url")
cmdCollections.AddCommand(cmdCollectionsInspect)
var cmdCollectionsList = &cobra.Command{
Use: "list collection [-a]",
Short: "List all collections",
Long: `List all collections`,
Example: `cscli collections list`,
Args: cobra.ExactArgs(0),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
ListItems(color.Output, []string{cwhub.COLLECTIONS}, args, false, true, all)
},
}
cmdCollectionsList.PersistentFlags().BoolVarP(&all, "all", "a", false, "List disabled items as well")
cmdCollections.AddCommand(cmdCollectionsList)
return cmdCollections
}
crowdsec-1.4.6/cmd/crowdsec-cli/completion.go 0000664 0000000 0000000 00000004310 14371172257 0021163 0 ustar 00root root 0000000 0000000 package main
import (
"os"
"github.com/spf13/cobra"
)
func NewCompletionCmd() *cobra.Command {
var completionCmd = &cobra.Command{
Use: "completion [bash|zsh|powershell|fish]",
Short: "Generate completion script",
Long: `To load completions:
### Bash:
` + "```shell" + `
$ source <(cscli completion bash)
# To load completions for each session, execute once:
# Linux:
$ cscli completion bash | sudo tee /etc/bash_completion.d/cscli
$ source ~/.bashrc
# macOS:
$ cscli completion bash | sudo tee /usr/local/etc/bash_completion.d/cscli
# Troubleshoot:
If you have this error (bash: _get_comp_words_by_ref: command not found), it seems that you need "bash-completion" dependency :
* Install bash-completion package
$ source /etc/profile
$ source <(cscli completion bash)
` + "```" + `
### Zsh:
` + "```shell" + `
# If shell completion is not already enabled in your environment,
# you will need to enable it. You can execute the following once:
$ echo "autoload -U compinit; compinit" >> ~/.zshrc
# To load completions for each session, execute once:
$ cscli completion zsh > "${fpath[1]}/_cscli"
# You will need to start a new shell for this setup to take effect.
### fish:
` + "```shell" + `
$ cscli completion fish | source
# To load completions for each session, execute once:
$ cscli completion fish > ~/.config/fish/completions/cscli.fish
` + "```" + `
### PowerShell:
` + "```powershell" + `
PS> cscli completion powershell | Out-String | Invoke-Expression
# To load completions for every new session, run:
PS> cscli completion powershell > cscli.ps1
# and source this file from your PowerShell profile.
` + "```",
DisableFlagsInUseLine: true,
DisableAutoGenTag: true,
ValidArgs: []string{"bash", "zsh", "powershell", "fish"},
Args: cobra.ExactValidArgs(1),
Run: func(cmd *cobra.Command, args []string) {
switch args[0] {
case "bash":
cmd.Root().GenBashCompletion(os.Stdout)
case "zsh":
cmd.Root().GenZshCompletion(os.Stdout)
case "powershell":
cmd.Root().GenPowerShellCompletion(os.Stdout)
case "fish":
cmd.Root().GenFishCompletion(os.Stdout, true)
}
},
}
return completionCmd
}
crowdsec-1.4.6/cmd/crowdsec-cli/config.go 0000664 0000000 0000000 00000047502 14371172257 0020271 0 ustar 00root root 0000000 0000000 package main
import (
"encoding/json"
"fmt"
"io"
"os"
"path/filepath"
"github.com/antonmedv/expr"
"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"gopkg.in/yaml.v2"
"github.com/crowdsecurity/crowdsec/pkg/csconfig"
"github.com/crowdsecurity/crowdsec/pkg/cwhub"
"github.com/crowdsecurity/crowdsec/pkg/types"
)
type OldAPICfg struct {
MachineID string `json:"machine_id"`
Password string `json:"password"`
}
/* Backup crowdsec configurations to directory :
- Main config (config.yaml)
- Profiles config (profiles.yaml)
- Simulation config (simulation.yaml)
- Backup of API credentials (local API and online API)
- List of scenarios, parsers, postoverflows and collections that are up-to-date
- Tainted/local/out-of-date scenarios, parsers, postoverflows and collections
*/
func backupConfigToDirectory(dirPath string) error {
var err error
if dirPath == "" {
return fmt.Errorf("directory path can't be empty")
}
log.Infof("Starting configuration backup")
/*if parent directory doesn't exist, bail out. create final dir with Mkdir*/
parentDir := filepath.Dir(dirPath)
if _, err := os.Stat(parentDir); err != nil {
return errors.Wrapf(err, "while checking parent directory %s existence", parentDir)
}
if err = os.Mkdir(dirPath, 0700); err != nil {
return errors.Wrapf(err, "while creating %s", dirPath)
}
if csConfig.ConfigPaths.SimulationFilePath != "" {
backupSimulation := filepath.Join(dirPath, "simulation.yaml")
if err = types.CopyFile(csConfig.ConfigPaths.SimulationFilePath, backupSimulation); err != nil {
return errors.Wrapf(err, "failed copy %s to %s", csConfig.ConfigPaths.SimulationFilePath, backupSimulation)
}
log.Infof("Saved simulation to %s", backupSimulation)
}
/*
- backup AcquisitionFilePath
- backup the other files of acquisition directory
*/
if csConfig.Crowdsec != nil && csConfig.Crowdsec.AcquisitionFilePath != "" {
backupAcquisition := filepath.Join(dirPath, "acquis.yaml")
if err = types.CopyFile(csConfig.Crowdsec.AcquisitionFilePath, backupAcquisition); err != nil {
return fmt.Errorf("failed copy %s to %s : %s", csConfig.Crowdsec.AcquisitionFilePath, backupAcquisition, err)
}
}
acquisBackupDir := filepath.Join(dirPath, "acquis")
if err = os.Mkdir(acquisBackupDir, 0700); err != nil {
return fmt.Errorf("error while creating %s : %s", acquisBackupDir, err)
}
if csConfig.Crowdsec != nil && len(csConfig.Crowdsec.AcquisitionFiles) > 0 {
for _, acquisFile := range csConfig.Crowdsec.AcquisitionFiles {
/*if it was the default one, it was already backup'ed*/
if csConfig.Crowdsec.AcquisitionFilePath == acquisFile {
continue
}
targetFname, err := filepath.Abs(filepath.Join(acquisBackupDir, filepath.Base(acquisFile)))
if err != nil {
return errors.Wrapf(err, "while saving %s to %s", acquisFile, acquisBackupDir)
}
if err = types.CopyFile(acquisFile, targetFname); err != nil {
return fmt.Errorf("failed copy %s to %s : %s", acquisFile, targetFname, err)
}
log.Infof("Saved acquis %s to %s", acquisFile, targetFname)
}
}
if ConfigFilePath != "" {
backupMain := fmt.Sprintf("%s/config.yaml", dirPath)
if err = types.CopyFile(ConfigFilePath, backupMain); err != nil {
return fmt.Errorf("failed copy %s to %s : %s", ConfigFilePath, backupMain, err)
}
log.Infof("Saved default yaml to %s", backupMain)
}
if csConfig.API != nil && csConfig.API.Server != nil && csConfig.API.Server.OnlineClient != nil && csConfig.API.Server.OnlineClient.CredentialsFilePath != "" {
backupCAPICreds := fmt.Sprintf("%s/online_api_credentials.yaml", dirPath)
if err = types.CopyFile(csConfig.API.Server.OnlineClient.CredentialsFilePath, backupCAPICreds); err != nil {
return fmt.Errorf("failed copy %s to %s : %s", csConfig.API.Server.OnlineClient.CredentialsFilePath, backupCAPICreds, err)
}
log.Infof("Saved online API credentials to %s", backupCAPICreds)
}
if csConfig.API != nil && csConfig.API.Client != nil && csConfig.API.Client.CredentialsFilePath != "" {
backupLAPICreds := fmt.Sprintf("%s/local_api_credentials.yaml", dirPath)
if err = types.CopyFile(csConfig.API.Client.CredentialsFilePath, backupLAPICreds); err != nil {
return fmt.Errorf("failed copy %s to %s : %s", csConfig.API.Client.CredentialsFilePath, backupLAPICreds, err)
}
log.Infof("Saved local API credentials to %s", backupLAPICreds)
}
if csConfig.API != nil && csConfig.API.Server != nil && csConfig.API.Server.ProfilesPath != "" {
backupProfiles := fmt.Sprintf("%s/profiles.yaml", dirPath)
if err = types.CopyFile(csConfig.API.Server.ProfilesPath, backupProfiles); err != nil {
return fmt.Errorf("failed copy %s to %s : %s", csConfig.API.Server.ProfilesPath, backupProfiles, err)
}
log.Infof("Saved profiles to %s", backupProfiles)
}
if err = BackupHub(dirPath); err != nil {
return fmt.Errorf("failed to backup hub config : %s", err)
}
return nil
}
/* Restore crowdsec configurations to directory :
- Main config (config.yaml)
- Profiles config (profiles.yaml)
- Simulation config (simulation.yaml)
- Backup of API credentials (local API and online API)
- List of scenarios, parsers, postoverflows and collections that are up-to-date
- Tainted/local/out-of-date scenarios, parsers, postoverflows and collections
*/
func restoreConfigFromDirectory(dirPath string) error {
var err error
if !restoreOldBackup {
backupMain := fmt.Sprintf("%s/config.yaml", dirPath)
if _, err = os.Stat(backupMain); err == nil {
if csConfig.ConfigPaths != nil && csConfig.ConfigPaths.ConfigDir != "" {
if err = types.CopyFile(backupMain, fmt.Sprintf("%s/config.yaml", csConfig.ConfigPaths.ConfigDir)); err != nil {
return fmt.Errorf("failed copy %s to %s : %s", backupMain, csConfig.ConfigPaths.ConfigDir, err)
}
}
}
// Now we have config.yaml, we should regenerate config struct to have rights paths etc
ConfigFilePath = fmt.Sprintf("%s/config.yaml", csConfig.ConfigPaths.ConfigDir)
initConfig()
backupCAPICreds := fmt.Sprintf("%s/online_api_credentials.yaml", dirPath)
if _, err = os.Stat(backupCAPICreds); err == nil {
if err = types.CopyFile(backupCAPICreds, csConfig.API.Server.OnlineClient.CredentialsFilePath); err != nil {
return fmt.Errorf("failed copy %s to %s : %s", backupCAPICreds, csConfig.API.Server.OnlineClient.CredentialsFilePath, err)
}
}
backupLAPICreds := fmt.Sprintf("%s/local_api_credentials.yaml", dirPath)
if _, err = os.Stat(backupLAPICreds); err == nil {
if err = types.CopyFile(backupLAPICreds, csConfig.API.Client.CredentialsFilePath); err != nil {
return fmt.Errorf("failed copy %s to %s : %s", backupLAPICreds, csConfig.API.Client.CredentialsFilePath, err)
}
}
backupProfiles := fmt.Sprintf("%s/profiles.yaml", dirPath)
if _, err = os.Stat(backupProfiles); err == nil {
if err = types.CopyFile(backupProfiles, csConfig.API.Server.ProfilesPath); err != nil {
return fmt.Errorf("failed copy %s to %s : %s", backupProfiles, csConfig.API.Server.ProfilesPath, err)
}
}
} else {
var oldAPICfg OldAPICfg
backupOldAPICfg := fmt.Sprintf("%s/api_creds.json", dirPath)
jsonFile, err := os.Open(backupOldAPICfg)
if err != nil {
log.Warningf("failed to open %s : %s", backupOldAPICfg, err)
} else {
byteValue, _ := io.ReadAll(jsonFile)
err = json.Unmarshal(byteValue, &oldAPICfg)
if err != nil {
return fmt.Errorf("failed to load json file %s : %s", backupOldAPICfg, err)
}
apiCfg := csconfig.ApiCredentialsCfg{
Login: oldAPICfg.MachineID,
Password: oldAPICfg.Password,
URL: CAPIBaseURL,
}
apiConfigDump, err := yaml.Marshal(apiCfg)
if err != nil {
return fmt.Errorf("unable to dump api credentials: %s", err)
}
apiConfigDumpFile := fmt.Sprintf("%s/online_api_credentials.yaml", csConfig.ConfigPaths.ConfigDir)
if csConfig.API.Server.OnlineClient != nil && csConfig.API.Server.OnlineClient.CredentialsFilePath != "" {
apiConfigDumpFile = csConfig.API.Server.OnlineClient.CredentialsFilePath
}
err = os.WriteFile(apiConfigDumpFile, apiConfigDump, 0644)
if err != nil {
return fmt.Errorf("write api credentials in '%s' failed: %s", apiConfigDumpFile, err)
}
log.Infof("Saved API credentials to %s", apiConfigDumpFile)
}
}
backupSimulation := fmt.Sprintf("%s/simulation.yaml", dirPath)
if _, err = os.Stat(backupSimulation); err == nil {
if err = types.CopyFile(backupSimulation, csConfig.ConfigPaths.SimulationFilePath); err != nil {
return fmt.Errorf("failed copy %s to %s : %s", backupSimulation, csConfig.ConfigPaths.SimulationFilePath, err)
}
}
/*if there is a acquisition dir, restore its content*/
if csConfig.Crowdsec.AcquisitionDirPath != "" {
if err = os.Mkdir(csConfig.Crowdsec.AcquisitionDirPath, 0700); err != nil {
return fmt.Errorf("error while creating %s : %s", csConfig.Crowdsec.AcquisitionDirPath, err)
}
}
//if there was a single one
backupAcquisition := fmt.Sprintf("%s/acquis.yaml", dirPath)
if _, err = os.Stat(backupAcquisition); err == nil {
log.Debugf("restoring backup'ed %s", backupAcquisition)
if err = types.CopyFile(backupAcquisition, csConfig.Crowdsec.AcquisitionFilePath); err != nil {
return fmt.Errorf("failed copy %s to %s : %s", backupAcquisition, csConfig.Crowdsec.AcquisitionFilePath, err)
}
}
//if there is files in the acquis backup dir, restore them
acquisBackupDir := filepath.Join(dirPath, "acquis", "*.yaml")
if acquisFiles, err := filepath.Glob(acquisBackupDir); err == nil {
for _, acquisFile := range acquisFiles {
targetFname, err := filepath.Abs(csConfig.Crowdsec.AcquisitionDirPath + "/" + filepath.Base(acquisFile))
if err != nil {
return errors.Wrapf(err, "while saving %s to %s", acquisFile, targetFname)
}
log.Debugf("restoring %s to %s", acquisFile, targetFname)
if err = types.CopyFile(acquisFile, targetFname); err != nil {
return fmt.Errorf("failed copy %s to %s : %s", acquisFile, targetFname, err)
}
}
}
if csConfig.Crowdsec != nil && len(csConfig.Crowdsec.AcquisitionFiles) > 0 {
for _, acquisFile := range csConfig.Crowdsec.AcquisitionFiles {
log.Infof("backup filepath from dir -> %s", acquisFile)
/*if it was the default one, it was already backup'ed*/
if csConfig.Crowdsec.AcquisitionFilePath == acquisFile {
log.Infof("skip this one")
continue
}
targetFname, err := filepath.Abs(filepath.Join(acquisBackupDir, filepath.Base(acquisFile)))
if err != nil {
return errors.Wrapf(err, "while saving %s to %s", acquisFile, acquisBackupDir)
}
if err = types.CopyFile(acquisFile, targetFname); err != nil {
return fmt.Errorf("failed copy %s to %s : %s", acquisFile, targetFname, err)
}
log.Infof("Saved acquis %s to %s", acquisFile, targetFname)
}
}
if err = RestoreHub(dirPath); err != nil {
return fmt.Errorf("failed to restore hub config : %s", err)
}
return nil
}
func NewConfigCmd() *cobra.Command {
var cmdConfig = &cobra.Command{
Use: "config [command]",
Short: "Allows to view current config",
Args: cobra.ExactArgs(0),
DisableAutoGenTag: true,
}
var key string
type Env struct {
Config *csconfig.Config
}
var cmdConfigShow = &cobra.Command{
Use: "show",
Short: "Displays current config",
Long: `Displays the current cli configuration.`,
Args: cobra.ExactArgs(0),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
if key != "" {
program, err := expr.Compile(key, expr.Env(Env{}))
if err != nil {
log.Fatal(err)
}
output, err := expr.Run(program, Env{Config: csConfig})
if err != nil {
log.Fatal(err)
}
switch csConfig.Cscli.Output {
case "human", "raw":
switch output.(type) {
case string:
fmt.Printf("%s\n", output)
case int:
fmt.Printf("%d\n", output)
default:
fmt.Printf("%v\n", output)
}
case "json":
data, err := json.MarshalIndent(output, "", " ")
if err != nil {
log.Fatalf("failed to marshal configuration: %s", err)
}
fmt.Printf("%s\n", string(data))
}
return
}
switch csConfig.Cscli.Output {
case "human":
fmt.Printf("Global:\n")
if csConfig.ConfigPaths != nil {
fmt.Printf(" - Configuration Folder : %s\n", csConfig.ConfigPaths.ConfigDir)
fmt.Printf(" - Data Folder : %s\n", csConfig.ConfigPaths.DataDir)
fmt.Printf(" - Hub Folder : %s\n", csConfig.ConfigPaths.HubDir)
fmt.Printf(" - Simulation File : %s\n", csConfig.ConfigPaths.SimulationFilePath)
}
if csConfig.Common != nil {
fmt.Printf(" - Log Folder : %s\n", csConfig.Common.LogDir)
fmt.Printf(" - Log level : %s\n", csConfig.Common.LogLevel)
fmt.Printf(" - Log Media : %s\n", csConfig.Common.LogMedia)
}
if csConfig.Crowdsec != nil {
fmt.Printf("Crowdsec:\n")
fmt.Printf(" - Acquisition File : %s\n", csConfig.Crowdsec.AcquisitionFilePath)
fmt.Printf(" - Parsers routines : %d\n", csConfig.Crowdsec.ParserRoutinesCount)
if csConfig.Crowdsec.AcquisitionDirPath != "" {
fmt.Printf(" - Acquisition Folder : %s\n", csConfig.Crowdsec.AcquisitionDirPath)
}
}
if csConfig.Cscli != nil {
fmt.Printf("cscli:\n")
fmt.Printf(" - Output : %s\n", csConfig.Cscli.Output)
fmt.Printf(" - Hub Branch : %s\n", csConfig.Cscli.HubBranch)
fmt.Printf(" - Hub Folder : %s\n", csConfig.Cscli.HubDir)
}
if csConfig.API != nil {
if csConfig.API.Client != nil && csConfig.API.Client.Credentials != nil {
fmt.Printf("API Client:\n")
fmt.Printf(" - URL : %s\n", csConfig.API.Client.Credentials.URL)
fmt.Printf(" - Login : %s\n", csConfig.API.Client.Credentials.Login)
fmt.Printf(" - Credentials File : %s\n", csConfig.API.Client.CredentialsFilePath)
}
if csConfig.API.Server != nil {
fmt.Printf("Local API Server:\n")
fmt.Printf(" - Listen URL : %s\n", csConfig.API.Server.ListenURI)
fmt.Printf(" - Profile File : %s\n", csConfig.API.Server.ProfilesPath)
if csConfig.API.Server.TLS != nil {
if csConfig.API.Server.TLS.CertFilePath != "" {
fmt.Printf(" - Cert File : %s\n", csConfig.API.Server.TLS.CertFilePath)
}
if csConfig.API.Server.TLS.KeyFilePath != "" {
fmt.Printf(" - Key File : %s\n", csConfig.API.Server.TLS.KeyFilePath)
}
if csConfig.API.Server.TLS.CACertPath != "" {
fmt.Printf(" - CA Cert : %s\n", csConfig.API.Server.TLS.CACertPath)
}
if csConfig.API.Server.TLS.CRLPath != "" {
fmt.Printf(" - CRL : %s\n", csConfig.API.Server.TLS.CRLPath)
}
if csConfig.API.Server.TLS.CacheExpiration != nil {
fmt.Printf(" - Cache Expiration : %s\n", csConfig.API.Server.TLS.CacheExpiration)
}
if csConfig.API.Server.TLS.ClientVerification != "" {
fmt.Printf(" - Client Verification : %s\n", csConfig.API.Server.TLS.ClientVerification)
}
if csConfig.API.Server.TLS.AllowedAgentsOU != nil {
for _, ou := range csConfig.API.Server.TLS.AllowedAgentsOU {
fmt.Printf(" - Allowed Agents OU : %s\n", ou)
}
}
if csConfig.API.Server.TLS.AllowedBouncersOU != nil {
for _, ou := range csConfig.API.Server.TLS.AllowedBouncersOU {
fmt.Printf(" - Allowed Bouncers OU : %s\n", ou)
}
}
}
fmt.Printf(" - Trusted IPs: \n")
for _, ip := range csConfig.API.Server.TrustedIPs {
fmt.Printf(" - %s\n", ip)
}
if csConfig.API.Server.OnlineClient != nil && csConfig.API.Server.OnlineClient.Credentials != nil {
fmt.Printf("Central API:\n")
fmt.Printf(" - URL : %s\n", csConfig.API.Server.OnlineClient.Credentials.URL)
fmt.Printf(" - Login : %s\n", csConfig.API.Server.OnlineClient.Credentials.Login)
fmt.Printf(" - Credentials File : %s\n", csConfig.API.Server.OnlineClient.CredentialsFilePath)
}
}
}
if csConfig.DbConfig != nil {
fmt.Printf(" - Database:\n")
fmt.Printf(" - Type : %s\n", csConfig.DbConfig.Type)
switch csConfig.DbConfig.Type {
case "sqlite":
fmt.Printf(" - Path : %s\n", csConfig.DbConfig.DbPath)
default:
fmt.Printf(" - Host : %s\n", csConfig.DbConfig.Host)
fmt.Printf(" - Port : %d\n", csConfig.DbConfig.Port)
fmt.Printf(" - User : %s\n", csConfig.DbConfig.User)
fmt.Printf(" - DB Name : %s\n", csConfig.DbConfig.DbName)
}
if csConfig.DbConfig.Flush != nil {
if *csConfig.DbConfig.Flush.MaxAge != "" {
fmt.Printf(" - Flush age : %s\n", *csConfig.DbConfig.Flush.MaxAge)
}
if *csConfig.DbConfig.Flush.MaxItems != 0 {
fmt.Printf(" - Flush size : %d\n", *csConfig.DbConfig.Flush.MaxItems)
}
}
}
case "json":
data, err := json.MarshalIndent(csConfig, "", " ")
if err != nil {
log.Fatalf("failed to marshal configuration: %s", err)
}
fmt.Printf("%s\n", string(data))
case "raw":
data, err := yaml.Marshal(csConfig)
if err != nil {
log.Fatalf("failed to marshal configuration: %s", err)
}
fmt.Printf("%s\n", string(data))
}
},
}
cmdConfigShow.Flags().StringVar(&key, "key", "", "Display only this value (Config.API.Server.ListenURI)")
cmdConfig.AddCommand(cmdConfigShow)
var cmdConfigBackup = &cobra.Command{
Use: `backup "directory"`,
Short: "Backup current config",
Long: `Backup the current crowdsec configuration including :
- Main config (config.yaml)
- Simulation config (simulation.yaml)
- Profiles config (profiles.yaml)
- List of scenarios, parsers, postoverflows and collections that are up-to-date
- Tainted/local/out-of-date scenarios, parsers, postoverflows and collections
- Backup of API credentials (local API and online API)`,
Example: `cscli config backup ./my-backup`,
Args: cobra.ExactArgs(1),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
var err error
if err := csConfig.LoadHub(); err != nil {
log.Fatal(err)
}
if err = cwhub.GetHubIdx(csConfig.Hub); err != nil {
log.Info("Run 'sudo cscli hub update' to get the hub index")
log.Fatalf("Failed to get Hub index : %v", err)
}
if err = backupConfigToDirectory(args[0]); err != nil {
log.Fatalf("Failed to backup configurations: %s", err)
}
},
}
cmdConfig.AddCommand(cmdConfigBackup)
var cmdConfigRestore = &cobra.Command{
Use: `restore "directory"`,
Short: `Restore config in backup "directory"`,
Long: `Restore the crowdsec configuration from specified backup "directory" including:
- Main config (config.yaml)
- Simulation config (simulation.yaml)
- Profiles config (profiles.yaml)
- List of scenarios, parsers, postoverflows and collections that are up-to-date
- Tainted/local/out-of-date scenarios, parsers, postoverflows and collections
- Backup of API credentials (local API and online API)`,
Args: cobra.ExactArgs(1),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
var err error
if err := csConfig.LoadHub(); err != nil {
log.Fatal(err)
}
if err = cwhub.GetHubIdx(csConfig.Hub); err != nil {
log.Info("Run 'sudo cscli hub update' to get the hub index")
log.Fatalf("Failed to get Hub index : %v", err)
}
if err := restoreConfigFromDirectory(args[0]); err != nil {
log.Fatalf("failed restoring configurations from %s : %s", args[0], err)
}
},
}
cmdConfigRestore.PersistentFlags().BoolVar(&restoreOldBackup, "old-backup", false, "To use when you are upgrading crowdsec v0.X to v1.X and you need to restore backup from v0.X")
cmdConfig.AddCommand(cmdConfigRestore)
return cmdConfig
}
crowdsec-1.4.6/cmd/crowdsec-cli/console.go 0000664 0000000 0000000 00000023166 14371172257 0020466 0 ustar 00root root 0000000 0000000 package main
import (
"context"
"encoding/csv"
"encoding/json"
"errors"
"fmt"
"io/fs"
"net/url"
"os"
"github.com/fatih/color"
"github.com/go-openapi/strfmt"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"github.com/crowdsecurity/crowdsec/pkg/apiclient"
"github.com/crowdsecurity/crowdsec/pkg/csconfig"
"github.com/crowdsecurity/crowdsec/pkg/cwhub"
"github.com/crowdsecurity/crowdsec/pkg/cwversion"
"github.com/crowdsecurity/crowdsec/pkg/types"
)
func NewConsoleCmd() *cobra.Command {
var cmdConsole = &cobra.Command{
Use: "console [action]",
Short: "Manage interaction with Crowdsec console (https://app.crowdsec.net)",
Args: cobra.MinimumNArgs(1),
DisableAutoGenTag: true,
PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
if err := csConfig.LoadAPIServer(); err != nil || csConfig.DisableAPI {
var fdErr *fs.PathError
if errors.As(err, &fdErr) {
log.Fatalf("Unable to load Local API : %s", fdErr)
}
if err != nil {
log.Fatalf("Unable to load required Local API Configuration : %s", err)
}
log.Fatal("Local API is disabled, please run this command on the local API machine")
}
if csConfig.DisableAPI {
log.Fatal("Local API is disabled, please run this command on the local API machine")
}
if csConfig.API.Server.OnlineClient == nil {
log.Fatalf("No configuration for Central API (CAPI) in '%s'", *csConfig.FilePath)
}
if csConfig.API.Server.OnlineClient.Credentials == nil {
log.Fatal("You must configure Central API (CAPI) with `cscli capi register` before enrolling your instance")
}
return nil
},
}
name := ""
overwrite := false
tags := []string{}
cmdEnroll := &cobra.Command{
Use: "enroll [enroll-key]",
Short: "Enroll this instance to https://app.crowdsec.net [requires local API]",
Long: `
Enroll this instance to https://app.crowdsec.net
You can get your enrollment key by creating an account on https://app.crowdsec.net.
After running this command your will need to validate the enrollment in the webapp.`,
Example: `cscli console enroll YOUR-ENROLL-KEY
cscli console enroll --name [instance_name] YOUR-ENROLL-KEY
cscli console enroll --name [instance_name] --tags [tag_1] --tags [tag_2] YOUR-ENROLL-KEY
`,
Args: cobra.ExactArgs(1),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
password := strfmt.Password(csConfig.API.Server.OnlineClient.Credentials.Password)
apiURL, err := url.Parse(csConfig.API.Server.OnlineClient.Credentials.URL)
if err != nil {
log.Fatalf("Could not parse CAPI URL : %s", err)
}
if err := csConfig.LoadHub(); err != nil {
log.Fatal(err)
}
if err := cwhub.GetHubIdx(csConfig.Hub); err != nil {
log.Fatalf("Failed to load hub index : %s", err)
log.Info("Run 'sudo cscli hub update' to get the hub index")
}
scenarios, err := cwhub.GetInstalledScenariosAsString()
if err != nil {
log.Fatalf("failed to get scenarios : %s", err)
}
if len(scenarios) == 0 {
scenarios = make([]string, 0)
}
c, _ := apiclient.NewClient(&apiclient.Config{
MachineID: csConfig.API.Server.OnlineClient.Credentials.Login,
Password: password,
Scenarios: scenarios,
UserAgent: fmt.Sprintf("crowdsec/%s", cwversion.VersionStr()),
URL: apiURL,
VersionPrefix: "v2",
})
resp, err := c.Auth.EnrollWatcher(context.Background(), args[0], name, tags, overwrite)
if err != nil {
log.Fatalf("Could not enroll instance: %s", err)
}
if resp.Response.StatusCode == 200 && !overwrite {
log.Warning("Instance already enrolled. You can use '--overwrite' to force enroll")
return
}
SetConsoleOpts(csconfig.CONSOLE_CONFIGS, true)
if err := csConfig.API.Server.DumpConsoleConfig(); err != nil {
log.Fatalf("failed writing console config : %s", err)
}
log.Infof("Enabled tainted&manual alerts sharing, see 'cscli console status'.")
log.Infof("Watcher successfully enrolled. Visit https://app.crowdsec.net to accept it.")
log.Infof("Please restart crowdsec after accepting the enrollment.")
},
}
cmdEnroll.Flags().StringVarP(&name, "name", "n", "", "Name to display in the console")
cmdEnroll.Flags().BoolVarP(&overwrite, "overwrite", "", false, "Force enroll the instance")
cmdEnroll.Flags().StringSliceVarP(&tags, "tags", "t", tags, "Tags to display in the console")
cmdConsole.AddCommand(cmdEnroll)
var enableAll, disableAll bool
cmdEnable := &cobra.Command{
Use: "enable [feature-flag]",
Short: "Enable a feature flag",
Example: "enable tainted",
Long: `
Enable given information push to the central API. Allows to empower the console`,
ValidArgs: csconfig.CONSOLE_CONFIGS,
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
if enableAll {
SetConsoleOpts(csconfig.CONSOLE_CONFIGS, true)
log.Infof("All features have been enabled successfully")
} else {
if len(args) == 0 {
log.Fatalf("You must specify at least one feature to enable")
}
SetConsoleOpts(args, true)
log.Infof("%v have been enabled", args)
}
if err := csConfig.API.Server.DumpConsoleConfig(); err != nil {
log.Fatalf("failed writing console config : %s", err)
}
log.Infof(ReloadMessage())
},
}
cmdEnable.Flags().BoolVarP(&enableAll, "all", "a", false, "Enable all feature flags")
cmdConsole.AddCommand(cmdEnable)
cmdDisable := &cobra.Command{
Use: "disable [feature-flag]",
Short: "Disable a feature flag",
Example: "disable tainted",
Long: `
Disable given information push to the central API.`,
ValidArgs: csconfig.CONSOLE_CONFIGS,
Args: cobra.MinimumNArgs(1),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
if disableAll {
SetConsoleOpts(csconfig.CONSOLE_CONFIGS, false)
} else {
SetConsoleOpts(args, false)
}
if err := csConfig.API.Server.DumpConsoleConfig(); err != nil {
log.Fatalf("failed writing console config : %s", err)
}
if disableAll {
log.Infof("All features have been disabled")
} else {
log.Infof("%v have been disabled", args)
}
log.Infof(ReloadMessage())
},
}
cmdDisable.Flags().BoolVarP(&disableAll, "all", "a", false, "Enable all feature flags")
cmdConsole.AddCommand(cmdDisable)
cmdConsoleStatus := &cobra.Command{
Use: "status [feature-flag]",
Short: "Shows status of one or all feature flags",
Example: "status tainted",
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
switch csConfig.Cscli.Output {
case "human":
cmdConsoleStatusTable(color.Output, *csConfig)
case "json":
data, err := json.MarshalIndent(csConfig.API.Server.ConsoleConfig, "", " ")
if err != nil {
log.Fatalf("failed to marshal configuration: %s", err)
}
fmt.Printf("%s\n", string(data))
case "raw":
csvwriter := csv.NewWriter(os.Stdout)
err := csvwriter.Write([]string{"option", "enabled"})
if err != nil {
log.Fatal(err)
}
rows := [][]string{
{"share_manual_decisions", fmt.Sprintf("%t", *csConfig.API.Server.ConsoleConfig.ShareManualDecisions)},
{"share_custom", fmt.Sprintf("%t", *csConfig.API.Server.ConsoleConfig.ShareCustomScenarios)},
{"share_tainted", fmt.Sprintf("%t", *csConfig.API.Server.ConsoleConfig.ShareTaintedScenarios)},
}
for _, row := range rows {
err = csvwriter.Write(row)
if err != nil {
log.Fatal(err)
}
}
csvwriter.Flush()
}
},
}
cmdConsole.AddCommand(cmdConsoleStatus)
return cmdConsole
}
func SetConsoleOpts(args []string, wanted bool) {
for _, arg := range args {
switch arg {
case csconfig.SEND_CUSTOM_SCENARIOS:
/*for each flag check if it's already set before setting it*/
if csConfig.API.Server.ConsoleConfig.ShareCustomScenarios != nil {
if *csConfig.API.Server.ConsoleConfig.ShareCustomScenarios == wanted {
log.Infof("%s already set to %t", csconfig.SEND_CUSTOM_SCENARIOS, wanted)
} else {
log.Infof("%s set to %t", csconfig.SEND_CUSTOM_SCENARIOS, wanted)
*csConfig.API.Server.ConsoleConfig.ShareCustomScenarios = wanted
}
} else {
log.Infof("%s set to %t", csconfig.SEND_CUSTOM_SCENARIOS, wanted)
csConfig.API.Server.ConsoleConfig.ShareCustomScenarios = types.BoolPtr(wanted)
}
case csconfig.SEND_TAINTED_SCENARIOS:
/*for each flag check if it's already set before setting it*/
if csConfig.API.Server.ConsoleConfig.ShareTaintedScenarios != nil {
if *csConfig.API.Server.ConsoleConfig.ShareTaintedScenarios == wanted {
log.Infof("%s already set to %t", csconfig.SEND_TAINTED_SCENARIOS, wanted)
} else {
log.Infof("%s set to %t", csconfig.SEND_TAINTED_SCENARIOS, wanted)
*csConfig.API.Server.ConsoleConfig.ShareTaintedScenarios = wanted
}
} else {
log.Infof("%s set to %t", csconfig.SEND_TAINTED_SCENARIOS, wanted)
csConfig.API.Server.ConsoleConfig.ShareTaintedScenarios = types.BoolPtr(wanted)
}
case csconfig.SEND_MANUAL_SCENARIOS:
/*for each flag check if it's already set before setting it*/
if csConfig.API.Server.ConsoleConfig.ShareManualDecisions != nil {
if *csConfig.API.Server.ConsoleConfig.ShareManualDecisions == wanted {
log.Infof("%s already set to %t", csconfig.SEND_MANUAL_SCENARIOS, wanted)
} else {
log.Infof("%s set to %t", csconfig.SEND_MANUAL_SCENARIOS, wanted)
*csConfig.API.Server.ConsoleConfig.ShareManualDecisions = wanted
}
} else {
log.Infof("%s set to %t", csconfig.SEND_MANUAL_SCENARIOS, wanted)
csConfig.API.Server.ConsoleConfig.ShareManualDecisions = types.BoolPtr(wanted)
}
default:
log.Fatalf("unknown flag %s", arg)
}
}
}
crowdsec-1.4.6/cmd/crowdsec-cli/console_table.go 0000664 0000000 0000000 00000002435 14371172257 0021631 0 ustar 00root root 0000000 0000000 package main
import (
"io"
"github.com/aquasecurity/table"
"github.com/enescakir/emoji"
"github.com/crowdsecurity/crowdsec/pkg/csconfig"
)
func cmdConsoleStatusTable(out io.Writer, csConfig csconfig.Config) {
t := newTable(out)
t.SetRowLines(false)
t.SetHeaders("Option Name", "Activated", "Description")
t.SetHeaderAlignment(table.AlignLeft, table.AlignLeft, table.AlignLeft)
for _, option := range csconfig.CONSOLE_CONFIGS {
switch option {
case csconfig.SEND_CUSTOM_SCENARIOS:
activated := string(emoji.CrossMark)
if *csConfig.API.Server.ConsoleConfig.ShareCustomScenarios {
activated = string(emoji.CheckMarkButton)
}
t.AddRow(option, activated, "Send alerts from custom scenarios to the console")
case csconfig.SEND_MANUAL_SCENARIOS:
activated := string(emoji.CrossMark)
if *csConfig.API.Server.ConsoleConfig.ShareManualDecisions {
activated = string(emoji.CheckMarkButton)
}
t.AddRow(option, activated, "Send manual decisions to the console")
case csconfig.SEND_TAINTED_SCENARIOS:
activated := string(emoji.CrossMark)
if *csConfig.API.Server.ConsoleConfig.ShareTaintedScenarios {
activated = string(emoji.CheckMarkButton)
}
t.AddRow(option, activated, "Send alerts from tainted scenarios to the console")
}
}
t.Render()
}
crowdsec-1.4.6/cmd/crowdsec-cli/dashboard.go 0000664 0000000 0000000 00000026014 14371172257 0020746 0 ustar 00root root 0000000 0000000 package main
import (
"errors"
"fmt"
"math"
"os"
"os/exec"
"os/user"
"path/filepath"
"strconv"
"strings"
"unicode"
"github.com/AlecAivazis/survey/v2"
"github.com/pbnjay/memory"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"github.com/crowdsecurity/crowdsec/pkg/metabase"
)
var (
metabaseUser = "crowdsec@crowdsec.net"
metabasePassword string
metabaseDbPath string
metabaseConfigPath string
metabaseConfigFolder = "metabase/"
metabaseConfigFile = "metabase.yaml"
/**/
metabaseListenAddress = "127.0.0.1"
metabaseListenPort = "3000"
metabaseContainerID = "crowdsec-metabase"
crowdsecGroup = "crowdsec"
forceYes bool
/*informations needed to setup a random password on user's behalf*/
)
func NewDashboardCmd() *cobra.Command {
/* ---- UPDATE COMMAND */
var cmdDashboard = &cobra.Command{
Use: "dashboard [command]",
Short: "Manage your metabase dashboard container [requires local API]",
Long: `Install/Start/Stop/Remove a metabase container exposing dashboard and metrics.
Note: This command requires database direct access, so is intended to be run on Local API/master.
`,
Args: cobra.ExactArgs(1),
DisableAutoGenTag: true,
Example: `
cscli dashboard setup
cscli dashboard start
cscli dashboard stop
cscli dashboard remove
`,
PersistentPreRun: func(cmd *cobra.Command, args []string) {
if err := metabase.TestAvailability(); err != nil {
log.Fatalf("%s", err)
}
if err := csConfig.LoadAPIServer(); err != nil || csConfig.DisableAPI {
log.Fatal("Local API is disabled, please run this command on the local API machine")
}
metabaseConfigFolderPath := filepath.Join(csConfig.ConfigPaths.ConfigDir, metabaseConfigFolder)
metabaseConfigPath = filepath.Join(metabaseConfigFolderPath, metabaseConfigFile)
if err := os.MkdirAll(metabaseConfigFolderPath, os.ModePerm); err != nil {
log.Fatal(err)
}
if err := csConfig.LoadDBConfig(); err != nil {
log.Errorf("This command requires direct database access (must be run on the local API machine)")
log.Fatal(err)
}
/*
Old container name was "/crowdsec-metabase" but podman doesn't
allow '/' in container name. We do this check to not break
existing dashboard setup.
*/
if !metabase.IsContainerExist(metabaseContainerID) {
oldContainerID := fmt.Sprintf("/%s", metabaseContainerID)
if metabase.IsContainerExist(oldContainerID) {
metabaseContainerID = oldContainerID
}
}
},
}
var force bool
var cmdDashSetup = &cobra.Command{
Use: "setup",
Short: "Setup a metabase container.",
Long: `Perform a metabase docker setup, download standard dashboards, create a fresh user and start the container`,
Args: cobra.ExactArgs(0),
DisableAutoGenTag: true,
Example: `
cscli dashboard setup
cscli dashboard setup --listen 0.0.0.0
cscli dashboard setup -l 0.0.0.0 -p 443 --password
`,
Run: func(cmd *cobra.Command, args []string) {
if metabaseDbPath == "" {
metabaseDbPath = csConfig.ConfigPaths.DataDir
}
if metabasePassword == "" {
isValid := passwordIsValid(metabasePassword)
for !isValid {
metabasePassword = generatePassword(16)
isValid = passwordIsValid(metabasePassword)
}
}
var answer bool
if valid, err := checkSystemMemory(); err == nil && !valid {
if !forceYes {
prompt := &survey.Confirm{
Message: "Metabase requires 1-2GB of RAM, your system is below this requirement continue ?",
Default: true,
}
if err := survey.AskOne(prompt, &answer); err != nil {
log.Warnf("unable to ask about RAM check: %s", err)
}
if !answer {
log.Fatal("Unable to continue due to RAM requirement")
}
} else {
log.Warnf("Metabase requires 1-2GB of RAM, your system is below this requirement")
}
}
groupExist := false
dockerGroup, err := user.LookupGroup(crowdsecGroup)
if err == nil {
groupExist = true
}
if !forceYes && !groupExist {
prompt := &survey.Confirm{
Message: fmt.Sprintf("For metabase docker to be able to access SQLite file we need to add a new group called '%s' to the system, is it ok for you ?", crowdsecGroup),
Default: true,
}
if err := survey.AskOne(prompt, &answer); err != nil {
log.Fatalf("unable to ask to force: %s", err)
}
}
if !answer && !forceYes && !groupExist {
log.Fatalf("unable to continue without creating '%s' group", crowdsecGroup)
}
if !groupExist {
groupAddCmd, err := exec.LookPath("groupadd")
if err != nil {
log.Fatalf("unable to find 'groupadd' command, can't continue")
}
groupAdd := &exec.Cmd{Path: groupAddCmd, Args: []string{groupAddCmd, crowdsecGroup}}
if err := groupAdd.Run(); err != nil {
log.Fatalf("unable to add group '%s': %s", dockerGroup, err)
}
dockerGroup, err = user.LookupGroup(crowdsecGroup)
if err != nil {
log.Fatalf("unable to lookup '%s' group: %+v", dockerGroup, err)
}
}
intID, err := strconv.Atoi(dockerGroup.Gid)
if err != nil {
log.Fatalf("unable to convert group ID to int: %s", err)
}
if err := os.Chown(csConfig.DbConfig.DbPath, 0, intID); err != nil {
log.Fatalf("unable to chown sqlite db file '%s': %s", csConfig.DbConfig.DbPath, err)
}
mb, err := metabase.SetupMetabase(csConfig.API.Server.DbConfig, metabaseListenAddress, metabaseListenPort, metabaseUser, metabasePassword, metabaseDbPath, dockerGroup.Gid, metabaseContainerID)
if err != nil {
log.Fatal(err)
}
if err := mb.DumpConfig(metabaseConfigPath); err != nil {
log.Fatal(err)
}
log.Infof("Metabase is ready")
fmt.Println()
fmt.Printf("\tURL : '%s'\n", mb.Config.ListenURL)
fmt.Printf("\tusername : '%s'\n", mb.Config.Username)
fmt.Printf("\tpassword : '%s'\n", mb.Config.Password)
},
}
cmdDashSetup.Flags().BoolVarP(&force, "force", "f", false, "Force setup : override existing files.")
cmdDashSetup.Flags().StringVarP(&metabaseDbPath, "dir", "d", "", "Shared directory with metabase container.")
cmdDashSetup.Flags().StringVarP(&metabaseListenAddress, "listen", "l", metabaseListenAddress, "Listen address of container")
cmdDashSetup.Flags().StringVarP(&metabaseListenPort, "port", "p", metabaseListenPort, "Listen port of container")
cmdDashSetup.Flags().BoolVarP(&forceYes, "yes", "y", false, "force yes")
//cmdDashSetup.Flags().StringVarP(&metabaseUser, "user", "u", "crowdsec@crowdsec.net", "metabase user")
cmdDashSetup.Flags().StringVar(&metabasePassword, "password", "", "metabase password")
cmdDashboard.AddCommand(cmdDashSetup)
var cmdDashStart = &cobra.Command{
Use: "start",
Short: "Start the metabase container.",
Long: `Stats the metabase container using docker.`,
Args: cobra.ExactArgs(0),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
mb, err := metabase.NewMetabase(metabaseConfigPath, metabaseContainerID)
if err != nil {
log.Fatal(err)
}
if err := mb.Container.Start(); err != nil {
log.Fatalf("Failed to start metabase container : %s", err)
}
log.Infof("Started metabase")
log.Infof("url : http://%s:%s", metabaseListenAddress, metabaseListenPort)
},
}
cmdDashboard.AddCommand(cmdDashStart)
var cmdDashStop = &cobra.Command{
Use: "stop",
Short: "Stops the metabase container.",
Long: `Stops the metabase container using docker.`,
Args: cobra.ExactArgs(0),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
if err := metabase.StopContainer(metabaseContainerID); err != nil {
log.Fatalf("unable to stop container '%s': %s", metabaseContainerID, err)
}
},
}
cmdDashboard.AddCommand(cmdDashStop)
var cmdDashShowPassword = &cobra.Command{Use: "show-password",
Short: "displays password of metabase.",
Args: cobra.ExactArgs(0),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
m := metabase.Metabase{}
if err := m.LoadConfig(metabaseConfigPath); err != nil {
log.Fatal(err)
}
log.Printf("'%s'", m.Config.Password)
},
}
cmdDashboard.AddCommand(cmdDashShowPassword)
var cmdDashRemove = &cobra.Command{
Use: "remove",
Short: "removes the metabase container.",
Long: `removes the metabase container using docker.`,
Args: cobra.ExactArgs(0),
DisableAutoGenTag: true,
Example: `
cscli dashboard remove
cscli dashboard remove --force
`,
Run: func(cmd *cobra.Command, args []string) {
answer := true
if !forceYes {
prompt := &survey.Confirm{
Message: "Do you really want to remove crowdsec dashboard? (all your changes will be lost)",
Default: true,
}
if err := survey.AskOne(prompt, &answer); err != nil {
log.Fatalf("unable to ask to force: %s", err)
}
}
if answer {
if metabase.IsContainerExist(metabaseContainerID) {
log.Debugf("Stopping container %s", metabaseContainerID)
if err := metabase.StopContainer(metabaseContainerID); err != nil {
log.Warningf("unable to stop container '%s': %s", metabaseContainerID, err)
}
dockerGroup, err := user.LookupGroup(crowdsecGroup)
if err == nil { // if group exist, remove it
groupDelCmd, err := exec.LookPath("groupdel")
if err != nil {
log.Fatalf("unable to find 'groupdel' command, can't continue")
}
groupDel := &exec.Cmd{Path: groupDelCmd, Args: []string{groupDelCmd, crowdsecGroup}}
if err := groupDel.Run(); err != nil {
log.Errorf("unable to delete group '%s': %s", dockerGroup, err)
}
}
log.Debugf("Removing container %s", metabaseContainerID)
if err := metabase.RemoveContainer(metabaseContainerID); err != nil {
log.Warningf("unable to remove container '%s': %s", metabaseContainerID, err)
}
log.Infof("container %s stopped & removed", metabaseContainerID)
}
log.Debugf("Removing metabase db %s", csConfig.ConfigPaths.DataDir)
if err := metabase.RemoveDatabase(csConfig.ConfigPaths.DataDir); err != nil {
log.Warningf("failed to remove metabase internal db : %s", err)
}
if force {
if err := metabase.RemoveImageContainer(); err != nil {
if !strings.Contains(err.Error(), "No such image") {
log.Fatalf("removing docker image: %s", err)
}
}
}
}
},
}
cmdDashRemove.Flags().BoolVarP(&force, "force", "f", false, "Remove also the metabase image")
cmdDashRemove.Flags().BoolVarP(&forceYes, "yes", "y", false, "force yes")
cmdDashboard.AddCommand(cmdDashRemove)
return cmdDashboard
}
func passwordIsValid(password string) bool {
hasDigit := false
for _, j := range password {
if unicode.IsDigit(j) {
hasDigit = true
break
}
}
if !hasDigit || len(password) < 6 {
return false
}
return true
}
func checkSystemMemory() (bool, error) {
totMem := memory.TotalMemory()
if totMem == 0 {
return true, errors.New("Unable to get system total memory")
}
if uint64(math.Pow(2, 30)) >= totMem {
return false, nil
}
return true, nil
}
crowdsec-1.4.6/cmd/crowdsec-cli/decisions.go 0000664 0000000 0000000 00000052621 14371172257 0021002 0 ustar 00root root 0000000 0000000 package main
import (
"context"
"encoding/csv"
"encoding/json"
"fmt"
"net/url"
"os"
"path/filepath"
"strconv"
"strings"
"time"
"github.com/fatih/color"
"github.com/go-openapi/strfmt"
"github.com/jszwec/csvutil"
"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"github.com/crowdsecurity/crowdsec/pkg/apiclient"
"github.com/crowdsecurity/crowdsec/pkg/cwversion"
"github.com/crowdsecurity/crowdsec/pkg/models"
"github.com/crowdsecurity/crowdsec/pkg/types"
)
var Client *apiclient.ApiClient
var (
defaultDuration = "4h"
defaultScope = "ip"
defaultType = "ban"
defaultReason = "manual"
)
func DecisionsToTable(alerts *models.GetAlertsResponse, printMachine bool) error {
/*here we cheat a bit : to make it more readable for the user, we dedup some entries*/
var spamLimit map[string]bool = make(map[string]bool)
var skipped = 0
for aIdx := 0; aIdx < len(*alerts); aIdx++ {
alertItem := (*alerts)[aIdx]
newDecisions := make([]*models.Decision, 0)
for _, decisionItem := range alertItem.Decisions {
spamKey := fmt.Sprintf("%t:%s:%s:%s", *decisionItem.Simulated, *decisionItem.Type, *decisionItem.Scope, *decisionItem.Value)
if _, ok := spamLimit[spamKey]; ok {
skipped++
continue
}
spamLimit[spamKey] = true
newDecisions = append(newDecisions, decisionItem)
}
alertItem.Decisions = newDecisions
}
if csConfig.Cscli.Output == "raw" {
csvwriter := csv.NewWriter(os.Stdout)
header := []string{"id", "source", "ip", "reason", "action", "country", "as", "events_count", "expiration", "simulated", "alert_id"}
if printMachine {
header = append(header, "machine")
}
err := csvwriter.Write(header)
if err != nil {
return err
}
for _, alertItem := range *alerts {
for _, decisionItem := range alertItem.Decisions {
raw := []string{
fmt.Sprintf("%d", decisionItem.ID),
*decisionItem.Origin,
*decisionItem.Scope + ":" + *decisionItem.Value,
*decisionItem.Scenario,
*decisionItem.Type,
alertItem.Source.Cn,
alertItem.Source.GetAsNumberName(),
fmt.Sprintf("%d", *alertItem.EventsCount),
*decisionItem.Duration,
fmt.Sprintf("%t", *decisionItem.Simulated),
fmt.Sprintf("%d", alertItem.ID),
}
if printMachine {
raw = append(raw, alertItem.MachineID)
}
err := csvwriter.Write(raw)
if err != nil {
return err
}
}
}
csvwriter.Flush()
} else if csConfig.Cscli.Output == "json" {
x, _ := json.MarshalIndent(alerts, "", " ")
fmt.Printf("%s", string(x))
} else if csConfig.Cscli.Output == "human" {
if len(*alerts) == 0 {
fmt.Println("No active decisions")
return nil
}
decisionsTable(color.Output, alerts, printMachine)
if skipped > 0 {
fmt.Printf("%d duplicated entries skipped\n", skipped)
}
}
return nil
}
func NewDecisionsCmd() *cobra.Command {
/* ---- DECISIONS COMMAND */
var cmdDecisions = &cobra.Command{
Use: "decisions [action]",
Short: "Manage decisions",
Long: `Add/List/Delete/Import decisions from LAPI`,
Example: `cscli decisions [action] [filter]`,
Aliases: []string{"decision"},
/*TBD example*/
Args: cobra.MinimumNArgs(1),
DisableAutoGenTag: true,
PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
if err := csConfig.LoadAPIClient(); err != nil {
return errors.Wrap(err, "loading api client")
}
password := strfmt.Password(csConfig.API.Client.Credentials.Password)
apiurl, err := url.Parse(csConfig.API.Client.Credentials.URL)
if err != nil {
return errors.Wrapf(err, "parsing api url %s", csConfig.API.Client.Credentials.URL)
}
Client, err = apiclient.NewClient(&apiclient.Config{
MachineID: csConfig.API.Client.Credentials.Login,
Password: password,
UserAgent: fmt.Sprintf("crowdsec/%s", cwversion.VersionStr()),
URL: apiurl,
VersionPrefix: "v1",
})
if err != nil {
return errors.Wrap(err, "creating api client")
}
return nil
},
}
var filter = apiclient.AlertsListOpts{
ValueEquals: new(string),
ScopeEquals: new(string),
ScenarioEquals: new(string),
OriginEquals: new(string),
IPEquals: new(string),
RangeEquals: new(string),
Since: new(string),
Until: new(string),
TypeEquals: new(string),
IncludeCAPI: new(bool),
Limit: new(int),
}
NoSimu := new(bool)
contained := new(bool)
var printMachine bool
var cmdDecisionsList = &cobra.Command{
Use: "list [options]",
Short: "List decisions from LAPI",
Example: `cscli decisions list -i 1.2.3.4
cscli decisions list -r 1.2.3.0/24
cscli decisions list -s crowdsecurity/ssh-bf
cscli decisions list -t ban
`,
Args: cobra.ExactArgs(0),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
var err error
/*take care of shorthand options*/
if err := manageCliDecisionAlerts(filter.IPEquals, filter.RangeEquals, filter.ScopeEquals, filter.ValueEquals); err != nil {
log.Fatalf("%s", err)
}
filter.ActiveDecisionEquals = new(bool)
*filter.ActiveDecisionEquals = true
if NoSimu != nil && *NoSimu {
filter.IncludeSimulated = new(bool)
}
/* nullify the empty entries to avoid bad filter */
if *filter.Until == "" {
filter.Until = nil
} else if strings.HasSuffix(*filter.Until, "d") {
/*time.ParseDuration support hours 'h' as bigger unit, let's make the user's life easier*/
realDuration := strings.TrimSuffix(*filter.Until, "d")
days, err := strconv.Atoi(realDuration)
if err != nil {
printHelp(cmd)
log.Fatalf("Can't parse duration %s, valid durations format: 1d, 4h, 4h15m", *filter.Until)
}
*filter.Until = fmt.Sprintf("%d%s", days*24, "h")
}
if *filter.Since == "" {
filter.Since = nil
} else if strings.HasSuffix(*filter.Since, "d") {
/*time.ParseDuration support hours 'h' as bigger unit, let's make the user's life easier*/
realDuration := strings.TrimSuffix(*filter.Since, "d")
days, err := strconv.Atoi(realDuration)
if err != nil {
printHelp(cmd)
log.Fatalf("Can't parse duration %s, valid durations format: 1d, 4h, 4h15m", *filter.Until)
}
*filter.Since = fmt.Sprintf("%d%s", days*24, "h")
}
if *filter.IncludeCAPI {
*filter.Limit = 0
}
if *filter.TypeEquals == "" {
filter.TypeEquals = nil
}
if *filter.ValueEquals == "" {
filter.ValueEquals = nil
}
if *filter.ScopeEquals == "" {
filter.ScopeEquals = nil
}
if *filter.ScenarioEquals == "" {
filter.ScenarioEquals = nil
}
if *filter.IPEquals == "" {
filter.IPEquals = nil
}
if *filter.RangeEquals == "" {
filter.RangeEquals = nil
}
if *filter.OriginEquals == "" {
filter.OriginEquals = nil
}
if contained != nil && *contained {
filter.Contains = new(bool)
}
alerts, _, err := Client.Alerts.List(context.Background(), filter)
if err != nil {
log.Fatalf("Unable to list decisions : %v", err)
}
err = DecisionsToTable(alerts, printMachine)
if err != nil {
log.Fatalf("unable to list decisions : %v", err)
}
},
}
cmdDecisionsList.Flags().SortFlags = false
cmdDecisionsList.Flags().BoolVarP(filter.IncludeCAPI, "all", "a", false, "Include decisions from Central API")
cmdDecisionsList.Flags().StringVar(filter.Since, "since", "", "restrict to alerts newer than since (ie. 4h, 30d)")
cmdDecisionsList.Flags().StringVar(filter.Until, "until", "", "restrict to alerts older than until (ie. 4h, 30d)")
cmdDecisionsList.Flags().StringVarP(filter.TypeEquals, "type", "t", "", "restrict to this decision type (ie. ban,captcha)")
cmdDecisionsList.Flags().StringVar(filter.ScopeEquals, "scope", "", "restrict to this scope (ie. ip,range,session)")
cmdDecisionsList.Flags().StringVar(filter.OriginEquals, "origin", "", "restrict to this origin (ie. lists,CAPI,cscli,cscli-import,crowdsec)")
cmdDecisionsList.Flags().StringVarP(filter.ValueEquals, "value", "v", "", "restrict to this value (ie. 1.2.3.4,userName)")
cmdDecisionsList.Flags().StringVarP(filter.ScenarioEquals, "scenario", "s", "", "restrict to this scenario (ie. crowdsecurity/ssh-bf)")
cmdDecisionsList.Flags().StringVarP(filter.IPEquals, "ip", "i", "", "restrict to alerts from this source ip (shorthand for --scope ip --value )")
cmdDecisionsList.Flags().StringVarP(filter.RangeEquals, "range", "r", "", "restrict to alerts from this source range (shorthand for --scope range --value )")
cmdDecisionsList.Flags().IntVarP(filter.Limit, "limit", "l", 100, "number of alerts to get (use 0 to remove the limit)")
cmdDecisionsList.Flags().BoolVar(NoSimu, "no-simu", false, "exclude decisions in simulation mode")
cmdDecisionsList.Flags().BoolVarP(&printMachine, "machine", "m", false, "print machines that triggered decisions")
cmdDecisionsList.Flags().BoolVar(contained, "contained", false, "query decisions contained by range")
cmdDecisions.AddCommand(cmdDecisionsList)
var (
addIP string
addRange string
addDuration string
addValue string
addScope string
addReason string
addType string
)
var cmdDecisionsAdd = &cobra.Command{
Use: "add [options]",
Short: "Add decision to LAPI",
Example: `cscli decisions add --ip 1.2.3.4
cscli decisions add --range 1.2.3.0/24
cscli decisions add --ip 1.2.3.4 --duration 24h --type captcha
cscli decisions add --scope username --value foobar
`,
/*TBD : fix long and example*/
Args: cobra.ExactArgs(0),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
var err error
var ipRange string
alerts := models.AddAlertsRequest{}
origin := "cscli"
capacity := int32(0)
leakSpeed := "0"
eventsCount := int32(1)
empty := ""
simulated := false
startAt := time.Now().UTC().Format(time.RFC3339)
stopAt := time.Now().UTC().Format(time.RFC3339)
createdAt := time.Now().UTC().Format(time.RFC3339)
/*take care of shorthand options*/
if err := manageCliDecisionAlerts(&addIP, &addRange, &addScope, &addValue); err != nil {
log.Fatalf("%s", err)
}
if addIP != "" {
addValue = addIP
addScope = types.Ip
} else if addRange != "" {
addValue = addRange
addScope = types.Range
} else if addValue == "" {
printHelp(cmd)
log.Fatalf("Missing arguments, a value is required (--ip, --range or --scope and --value)")
}
if addReason == "" {
addReason = fmt.Sprintf("manual '%s' from '%s'", addType, csConfig.API.Client.Credentials.Login)
}
decision := models.Decision{
Duration: &addDuration,
Scope: &addScope,
Value: &addValue,
Type: &addType,
Scenario: &addReason,
Origin: &origin,
}
alert := models.Alert{
Capacity: &capacity,
Decisions: []*models.Decision{&decision},
Events: []*models.Event{},
EventsCount: &eventsCount,
Leakspeed: &leakSpeed,
Message: &addReason,
ScenarioHash: &empty,
Scenario: &addReason,
ScenarioVersion: &empty,
Simulated: &simulated,
Source: &models.Source{
AsName: empty,
AsNumber: empty,
Cn: empty,
IP: addValue,
Range: ipRange,
Scope: &addScope,
Value: &addValue,
},
StartAt: &startAt,
StopAt: &stopAt,
CreatedAt: createdAt,
}
alerts = append(alerts, &alert)
_, _, err = Client.Alerts.Add(context.Background(), alerts)
if err != nil {
log.Fatal(err)
}
log.Info("Decision successfully added")
},
}
cmdDecisionsAdd.Flags().SortFlags = false
cmdDecisionsAdd.Flags().StringVarP(&addIP, "ip", "i", "", "Source ip (shorthand for --scope ip --value )")
cmdDecisionsAdd.Flags().StringVarP(&addRange, "range", "r", "", "Range source ip (shorthand for --scope range --value )")
cmdDecisionsAdd.Flags().StringVarP(&addDuration, "duration", "d", "4h", "Decision duration (ie. 1h,4h,30m)")
cmdDecisionsAdd.Flags().StringVarP(&addValue, "value", "v", "", "The value (ie. --scope username --value foobar)")
cmdDecisionsAdd.Flags().StringVar(&addScope, "scope", types.Ip, "Decision scope (ie. ip,range,username)")
cmdDecisionsAdd.Flags().StringVarP(&addReason, "reason", "R", "", "Decision reason (ie. scenario-name)")
cmdDecisionsAdd.Flags().StringVarP(&addType, "type", "t", "ban", "Decision type (ie. ban,captcha,throttle)")
cmdDecisions.AddCommand(cmdDecisionsAdd)
var delFilter = apiclient.DecisionsDeleteOpts{
ScopeEquals: new(string),
ValueEquals: new(string),
TypeEquals: new(string),
IPEquals: new(string),
RangeEquals: new(string),
ScenarioEquals: new(string),
}
var delDecisionId string
var delDecisionAll bool
var cmdDecisionsDelete = &cobra.Command{
Use: "delete [options]",
Short: "Delete decisions",
DisableAutoGenTag: true,
Aliases: []string{"remove"},
Example: `cscli decisions delete -r 1.2.3.0/24
cscli decisions delete -i 1.2.3.4
cscli decisions delete --id 42
cscli decisions delete --type captcha
`,
/*TBD : refaire le Long/Example*/
PreRun: func(cmd *cobra.Command, args []string) {
if delDecisionAll {
return
}
if *delFilter.ScopeEquals == "" && *delFilter.ValueEquals == "" &&
*delFilter.TypeEquals == "" && *delFilter.IPEquals == "" &&
*delFilter.RangeEquals == "" && *delFilter.ScenarioEquals == "" && delDecisionId == "" {
cmd.Usage()
log.Fatalln("At least one filter or --all must be specified")
}
},
Run: func(cmd *cobra.Command, args []string) {
var err error
var decisions *models.DeleteDecisionResponse
/*take care of shorthand options*/
if err := manageCliDecisionAlerts(delFilter.IPEquals, delFilter.RangeEquals, delFilter.ScopeEquals, delFilter.ValueEquals); err != nil {
log.Fatalf("%s", err)
}
if *delFilter.ScopeEquals == "" {
delFilter.ScopeEquals = nil
}
if *delFilter.ValueEquals == "" {
delFilter.ValueEquals = nil
}
if *delFilter.ScenarioEquals == "" {
delFilter.ScenarioEquals = nil
}
if *delFilter.TypeEquals == "" {
delFilter.TypeEquals = nil
}
if *delFilter.IPEquals == "" {
delFilter.IPEquals = nil
}
if *delFilter.RangeEquals == "" {
delFilter.RangeEquals = nil
}
if contained != nil && *contained {
delFilter.Contains = new(bool)
}
if delDecisionId == "" {
decisions, _, err = Client.Decisions.Delete(context.Background(), delFilter)
if err != nil {
log.Fatalf("Unable to delete decisions : %v", err)
}
} else {
if _, err = strconv.Atoi(delDecisionId); err != nil {
log.Fatalf("id '%s' is not an integer: %v", delDecisionId, err)
}
decisions, _, err = Client.Decisions.DeleteOne(context.Background(), delDecisionId)
if err != nil {
log.Fatalf("Unable to delete decision : %v", err)
}
}
log.Infof("%s decision(s) deleted", decisions.NbDeleted)
},
}
cmdDecisionsDelete.Flags().SortFlags = false
cmdDecisionsDelete.Flags().StringVarP(delFilter.IPEquals, "ip", "i", "", "Source ip (shorthand for --scope ip --value )")
cmdDecisionsDelete.Flags().StringVarP(delFilter.RangeEquals, "range", "r", "", "Range source ip (shorthand for --scope range --value )")
cmdDecisionsDelete.Flags().StringVarP(delFilter.TypeEquals, "type", "t", "", "the decision type (ie. ban,captcha)")
cmdDecisionsDelete.Flags().StringVarP(delFilter.ValueEquals, "value", "v", "", "the value to match for in the specified scope")
cmdDecisionsDelete.Flags().StringVarP(delFilter.ScenarioEquals, "scenario", "s", "", "the scenario name (ie. crowdsecurity/ssh-bf)")
cmdDecisionsDelete.Flags().StringVar(&delDecisionId, "id", "", "decision id")
cmdDecisionsDelete.Flags().BoolVar(&delDecisionAll, "all", false, "delete all decisions")
cmdDecisionsDelete.Flags().BoolVar(contained, "contained", false, "query decisions contained by range")
cmdDecisions.AddCommand(cmdDecisionsDelete)
var (
importDuration string
importScope string
importReason string
importType string
importFile string
)
var cmdDecisionImport = &cobra.Command{
Use: "import [options]",
Short: "Import decisions from json or csv file",
Long: "expected format :\n" +
"csv : any of duration,origin,reason,scope,type,value, with a header line\n" +
`json : {"duration" : "24h", "origin" : "my-list", "reason" : "my_scenario", "scope" : "ip", "type" : "ban", "value" : "x.y.z.z"}`,
DisableAutoGenTag: true,
Example: `decisions.csv :
duration,scope,value
24h,ip,1.2.3.4
cscsli decisions import -i decisions.csv
decisions.json :
[{"duration" : "4h", "scope" : "ip", "type" : "ban", "value" : "1.2.3.4"}]
`,
Run: func(cmd *cobra.Command, args []string) {
if importFile == "" {
log.Fatalf("Please provide a input file containing decisions with -i flag")
}
csvData, err := os.ReadFile(importFile)
if err != nil {
log.Fatalf("unable to open '%s': %s", importFile, err)
}
type decisionRaw struct {
Duration string `csv:"duration,omitempty" json:"duration,omitempty"`
Origin string `csv:"origin,omitempty" json:"origin,omitempty"`
Scenario string `csv:"reason,omitempty" json:"reason,omitempty"`
Scope string `csv:"scope,omitempty" json:"scope,omitempty"`
Type string `csv:"type,omitempty" json:"type,omitempty"`
Value string `csv:"value" json:"value"`
}
var decisionsListRaw []decisionRaw
switch fileFormat := filepath.Ext(importFile); fileFormat {
case ".json":
if err := json.Unmarshal(csvData, &decisionsListRaw); err != nil {
log.Fatalf("unable to unmarshall json: '%s'", err)
}
case ".csv":
if err := csvutil.Unmarshal(csvData, &decisionsListRaw); err != nil {
log.Fatalf("unable to unmarshall csv: '%s'", err)
}
default:
log.Fatalf("file format not supported for '%s'. supported format are 'json' and 'csv'", importFile)
}
decisionsList := make([]*models.Decision, 0)
for i, decisionLine := range decisionsListRaw {
line := i + 2
if decisionLine.Value == "" {
log.Fatalf("please provide a 'value' in your csv line %d", line)
}
/*deal with defaults and cli-override*/
if decisionLine.Duration == "" {
decisionLine.Duration = defaultDuration
log.Debugf("No 'duration' line %d, using default value: '%s'", line, defaultDuration)
}
if importDuration != "" {
decisionLine.Duration = importDuration
log.Debugf("'duration' line %d, using supplied value: '%s'", line, importDuration)
}
decisionLine.Origin = "cscli-import"
if decisionLine.Scenario == "" {
decisionLine.Scenario = defaultReason
log.Debugf("No 'reason' line %d, using value: '%s'", line, decisionLine.Scenario)
}
if importReason != "" {
decisionLine.Scenario = importReason
log.Debugf("No 'reason' line %d, using supplied value: '%s'", line, importReason)
}
if decisionLine.Type == "" {
decisionLine.Type = defaultType
log.Debugf("No 'type' line %d, using default value: '%s'", line, decisionLine.Type)
}
if importType != "" {
decisionLine.Type = importType
log.Debugf("'type' line %d, using supplied value: '%s'", line, importType)
}
if decisionLine.Scope == "" {
decisionLine.Scope = defaultScope
log.Debugf("No 'scope' line %d, using default value: '%s'", line, decisionLine.Scope)
}
if importScope != "" {
decisionLine.Scope = importScope
log.Debugf("'scope' line %d, using supplied value: '%s'", line, importScope)
}
decision := models.Decision{
Value: types.StrPtr(decisionLine.Value),
Duration: types.StrPtr(decisionLine.Duration),
Origin: types.StrPtr(decisionLine.Origin),
Scenario: types.StrPtr(decisionLine.Scenario),
Type: types.StrPtr(decisionLine.Type),
Scope: types.StrPtr(decisionLine.Scope),
Simulated: new(bool),
}
decisionsList = append(decisionsList, &decision)
}
alerts := models.AddAlertsRequest{}
importAlert := models.Alert{
CreatedAt: time.Now().UTC().Format(time.RFC3339),
Scenario: types.StrPtr(fmt.Sprintf("add: %d IPs", len(decisionsList))),
Message: types.StrPtr(""),
Events: []*models.Event{},
Source: &models.Source{
Scope: types.StrPtr("cscli/manual-import"),
Value: types.StrPtr(""),
},
StartAt: types.StrPtr(time.Now().UTC().Format(time.RFC3339)),
StopAt: types.StrPtr(time.Now().UTC().Format(time.RFC3339)),
Capacity: types.Int32Ptr(0),
Simulated: types.BoolPtr(false),
EventsCount: types.Int32Ptr(int32(len(decisionsList))),
Leakspeed: types.StrPtr(""),
ScenarioHash: types.StrPtr(""),
ScenarioVersion: types.StrPtr(""),
Decisions: decisionsList,
}
alerts = append(alerts, &importAlert)
if len(decisionsList) > 1000 {
log.Infof("You are about to add %d decisions, this may take a while", len(decisionsList))
}
_, _, err = Client.Alerts.Add(context.Background(), alerts)
if err != nil {
log.Fatal(err)
}
log.Infof("%d decisions successfully imported", len(decisionsList))
},
}
cmdDecisionImport.Flags().SortFlags = false
cmdDecisionImport.Flags().StringVarP(&importFile, "input", "i", "", "Input file")
cmdDecisionImport.Flags().StringVarP(&importDuration, "duration", "d", "", "Decision duration (ie. 1h,4h,30m)")
cmdDecisionImport.Flags().StringVar(&importScope, "scope", types.Ip, "Decision scope (ie. ip,range,username)")
cmdDecisionImport.Flags().StringVarP(&importReason, "reason", "R", "", "Decision reason (ie. scenario-name)")
cmdDecisionImport.Flags().StringVarP(&importType, "type", "t", "", "Decision type (ie. ban,captcha,throttle)")
cmdDecisions.AddCommand(cmdDecisionImport)
return cmdDecisions
}
crowdsec-1.4.6/cmd/crowdsec-cli/decisions_table.go 0000664 0000000 0000000 00000002137 14371172257 0022146 0 ustar 00root root 0000000 0000000 package main
import (
"fmt"
"io"
"strconv"
"github.com/crowdsecurity/crowdsec/pkg/models"
)
func decisionsTable(out io.Writer, alerts *models.GetAlertsResponse, printMachine bool) {
t := newTable(out)
t.SetRowLines(false)
header := []string{"ID", "Source", "Scope:Value", "Reason", "Action", "Country", "AS", "Events", "expiration", "Alert ID"}
if printMachine {
header = append(header, "Machine")
}
t.SetHeaders(header...)
for _, alertItem := range *alerts {
for _, decisionItem := range alertItem.Decisions {
if *alertItem.Simulated {
*decisionItem.Type = fmt.Sprintf("(simul)%s", *decisionItem.Type)
}
row := []string{
strconv.Itoa(int(decisionItem.ID)),
*decisionItem.Origin,
*decisionItem.Scope + ":" + *decisionItem.Value,
*decisionItem.Scenario,
*decisionItem.Type,
alertItem.Source.Cn,
alertItem.Source.GetAsNumberName(),
strconv.Itoa(int(*alertItem.EventsCount)),
*decisionItem.Duration,
strconv.Itoa(int(alertItem.ID)),
}
if printMachine {
row = append(row, alertItem.MachineID)
}
t.AddRow(row...)
}
}
t.Render()
}
crowdsec-1.4.6/cmd/crowdsec-cli/explain.go 0000664 0000000 0000000 00000010564 14371172257 0020462 0 ustar 00root root 0000000 0000000 package main
import (
"bufio"
"fmt"
"io"
"os"
"os/exec"
"path/filepath"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"github.com/crowdsecurity/crowdsec/pkg/hubtest"
"github.com/crowdsecurity/crowdsec/pkg/types"
)
func NewExplainCmd() *cobra.Command {
/* ---- HUB COMMAND */
var logFile string
var dsn string
var logLine string
var logType string
var opts hubtest.DumpOpts
var err error
var cmdExplain = &cobra.Command{
Use: "explain",
Short: "Explain log pipeline",
Long: `
Explain log pipeline
`,
Example: `
cscli explain --file ./myfile.log --type nginx
cscli explain --log "Sep 19 18:33:22 scw-d95986 sshd[24347]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=1.2.3.4" --type syslog
cscli explain --dsn "file://myfile.log" --type nginx
tail -n 5 myfile.log | cscli explain --type nginx -f -
`,
Args: cobra.ExactArgs(0),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
fileInfo, _ := os.Stdin.Stat()
if logType == "" || (logLine == "" && logFile == "" && dsn == "") {
printHelp(cmd)
fmt.Println()
fmt.Printf("Please provide --type flag\n")
os.Exit(1)
}
if logFile == "-" && ((fileInfo.Mode() & os.ModeCharDevice) == os.ModeCharDevice) {
log.Fatal("-f - is intended to work with pipes.")
}
var f *os.File
dir := os.TempDir()
tmpFile := ""
// we create a temporary log file if a log line/stdin has been provided
if logLine != "" || logFile == "-" {
tmpFile = filepath.Join(dir, "cscli_test_tmp.log")
f, err = os.Create(tmpFile)
if err != nil {
log.Fatal(err)
}
if logLine != "" {
_, err = f.WriteString(logLine)
if err != nil {
log.Fatal(err)
}
} else if logFile == "-" {
reader := bufio.NewReader(os.Stdin)
errCount := 0
for {
input, err := reader.ReadBytes('\n')
if err != nil && err == io.EOF {
break
}
_, err = f.Write(input)
if err != nil {
errCount++
}
}
if errCount > 0 {
log.Warnf("Failed to write %d lines to tmp file", errCount)
}
}
f.Close()
//this is the file that was going to be read by crowdsec anyway
logFile = tmpFile
}
if logFile != "" {
absolutePath, err := filepath.Abs(logFile)
if err != nil {
log.Fatalf("unable to get absolute path of '%s', exiting", logFile)
}
dsn = fmt.Sprintf("file://%s", absolutePath)
lineCount := types.GetLineCountForFile(absolutePath)
if lineCount > 100 {
log.Warnf("log file contains %d lines. This may take lot of resources.", lineCount)
}
}
if dsn == "" {
log.Fatal("no acquisition (--file or --dsn) provided, can't run cscli test.")
}
cmdArgs := []string{"-c", ConfigFilePath, "-type", logType, "-dsn", dsn, "-dump-data", "./", "-no-api"}
crowdsecCmd := exec.Command("crowdsec", cmdArgs...)
crowdsecCmd.Dir = dir
output, err := crowdsecCmd.CombinedOutput()
if err != nil {
fmt.Println(string(output))
log.Fatalf("fail to run crowdsec for test: %v", err)
}
// rm the temporary log file if only a log line/stdin was provided
if tmpFile != "" {
if err := os.Remove(tmpFile); err != nil {
log.Fatalf("unable to remove tmp log file '%s': %+v", tmpFile, err)
}
}
parserDumpFile := filepath.Join(dir, hubtest.ParserResultFileName)
bucketStateDumpFile := filepath.Join(dir, hubtest.BucketPourResultFileName)
parserDump, err := hubtest.LoadParserDump(parserDumpFile)
if err != nil {
log.Fatalf("unable to load parser dump result: %s", err)
}
bucketStateDump, err := hubtest.LoadBucketPourDump(bucketStateDumpFile)
if err != nil {
log.Fatalf("unable to load bucket dump result: %s", err)
}
hubtest.DumpTree(*parserDump, *bucketStateDump, opts)
},
}
cmdExplain.PersistentFlags().StringVarP(&logFile, "file", "f", "", "Log file to test")
cmdExplain.PersistentFlags().StringVarP(&dsn, "dsn", "d", "", "DSN to test")
cmdExplain.PersistentFlags().StringVarP(&logLine, "log", "l", "", "Log line to test")
cmdExplain.PersistentFlags().StringVarP(&logType, "type", "t", "", "Type of the acquisition to test")
cmdExplain.PersistentFlags().BoolVarP(&opts.Details, "verbose", "v", false, "Display individual changes")
cmdExplain.PersistentFlags().BoolVar(&opts.SkipOk, "failures", false, "Only show failed lines")
return cmdExplain
}
crowdsec-1.4.6/cmd/crowdsec-cli/hub.go 0000664 0000000 0000000 00000011040 14371172257 0017566 0 ustar 00root root 0000000 0000000 package main
import (
"fmt"
"github.com/fatih/color"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"github.com/crowdsecurity/crowdsec/pkg/cwhub"
)
func NewHubCmd() *cobra.Command {
/* ---- HUB COMMAND */
var cmdHub = &cobra.Command{
Use: "hub [action]",
Short: "Manage Hub",
Long: `
Hub management
List/update parsers/scenarios/postoverflows/collections from [Crowdsec Hub](https://hub.crowdsec.net).
The Hub is managed by cscli, to get the latest hub files from [Crowdsec Hub](https://hub.crowdsec.net), you need to update.
`,
Example: `
cscli hub list # List all installed configurations
cscli hub update # Download list of available configurations from the hub
`,
Args: cobra.ExactArgs(0),
DisableAutoGenTag: true,
PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
if csConfig.Cscli == nil {
return fmt.Errorf("you must configure cli before interacting with hub")
}
return nil
},
}
cmdHub.PersistentFlags().StringVarP(&cwhub.HubBranch, "branch", "b", "", "Use given branch from hub")
var cmdHubList = &cobra.Command{
Use: "list [-a]",
Short: "List installed configs",
Args: cobra.ExactArgs(0),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
if err := csConfig.LoadHub(); err != nil {
log.Fatal(err)
}
if err := cwhub.GetHubIdx(csConfig.Hub); err != nil {
log.Info("Run 'sudo cscli hub update' to get the hub index")
log.Fatalf("Failed to get Hub index : %v", err)
}
//use LocalSync to get warnings about tainted / outdated items
_, warn := cwhub.LocalSync(csConfig.Hub)
for _, v := range warn {
log.Info(v)
}
cwhub.DisplaySummary()
ListItems(color.Output, []string{
cwhub.COLLECTIONS, cwhub.PARSERS, cwhub.SCENARIOS, cwhub.PARSERS_OVFLW,
}, args, true, false, all)
},
}
cmdHubList.PersistentFlags().BoolVarP(&all, "all", "a", false, "List disabled items as well")
cmdHub.AddCommand(cmdHubList)
var cmdHubUpdate = &cobra.Command{
Use: "update",
Short: "Fetch available configs from hub",
Long: `
Fetches the [.index.json](https://github.com/crowdsecurity/hub/blob/master/.index.json) file from hub, containing the list of available configs.
`,
Args: cobra.ExactArgs(0),
DisableAutoGenTag: true,
PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
if csConfig.Cscli == nil {
return fmt.Errorf("you must configure cli before interacting with hub")
}
if err := cwhub.SetHubBranch(); err != nil {
return fmt.Errorf("error while setting hub branch: %s", err)
}
return nil
},
Run: func(cmd *cobra.Command, args []string) {
if err := csConfig.LoadHub(); err != nil {
log.Fatal(err)
}
if err := cwhub.UpdateHubIdx(csConfig.Hub); err != nil {
log.Fatalf("Failed to get Hub index : %v", err)
}
//use LocalSync to get warnings about tainted / outdated items
_, warn := cwhub.LocalSync(csConfig.Hub)
for _, v := range warn {
log.Info(v)
}
},
}
cmdHub.AddCommand(cmdHubUpdate)
var cmdHubUpgrade = &cobra.Command{
Use: "upgrade",
Short: "Upgrade all configs installed from hub",
Long: `
Upgrade all configs installed from Crowdsec Hub. Run 'sudo cscli hub update' if you want the latest versions available.
`,
Args: cobra.ExactArgs(0),
DisableAutoGenTag: true,
PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
if csConfig.Cscli == nil {
return fmt.Errorf("you must configure cli before interacting with hub")
}
if err := cwhub.SetHubBranch(); err != nil {
return fmt.Errorf("error while setting hub branch: %s", err)
}
return nil
},
Run: func(cmd *cobra.Command, args []string) {
if err := csConfig.LoadHub(); err != nil {
log.Fatal(err)
}
if err := cwhub.GetHubIdx(csConfig.Hub); err != nil {
log.Info("Run 'sudo cscli hub update' to get the hub index")
log.Fatalf("Failed to get Hub index : %v", err)
}
log.Infof("Upgrading collections")
cwhub.UpgradeConfig(csConfig, cwhub.COLLECTIONS, "", forceAction)
log.Infof("Upgrading parsers")
cwhub.UpgradeConfig(csConfig, cwhub.PARSERS, "", forceAction)
log.Infof("Upgrading scenarios")
cwhub.UpgradeConfig(csConfig, cwhub.SCENARIOS, "", forceAction)
log.Infof("Upgrading postoverflows")
cwhub.UpgradeConfig(csConfig, cwhub.PARSERS_OVFLW, "", forceAction)
},
}
cmdHubUpgrade.PersistentFlags().BoolVar(&forceAction, "force", false, "Force upgrade : Overwrite tainted and outdated files")
cmdHub.AddCommand(cmdHubUpgrade)
return cmdHub
}
crowdsec-1.4.6/cmd/crowdsec-cli/hubtest.go 0000664 0000000 0000000 00000042776 14371172257 0020512 0 ustar 00root root 0000000 0000000 package main
import (
"encoding/json"
"fmt"
"math"
"os"
"path/filepath"
"strings"
"github.com/AlecAivazis/survey/v2"
"github.com/enescakir/emoji"
"github.com/fatih/color"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"gopkg.in/yaml.v2"
"github.com/crowdsecurity/crowdsec/pkg/hubtest"
)
var (
HubTest hubtest.HubTest
)
func NewHubTestCmd() *cobra.Command {
/* ---- HUB COMMAND */
var hubPath string
var logType string
var crowdsecPath string
var cscliPath string
var cmdHubTest = &cobra.Command{
Use: "hubtest",
Short: "Run functional tests on hub configurations",
Long: "Run functional tests on hub configurations (parsers, scenarios, collections...)",
Args: cobra.ExactArgs(0),
DisableAutoGenTag: true,
PersistentPreRun: func(cmd *cobra.Command, args []string) {
var err error
HubTest, err = hubtest.NewHubTest(hubPath, crowdsecPath, cscliPath)
if err != nil {
log.Fatalf("unable to load hubtest: %+v", err)
}
},
}
cmdHubTest.PersistentFlags().StringVar(&hubPath, "hub", ".", "Path to hub folder")
cmdHubTest.PersistentFlags().StringVar(&crowdsecPath, "crowdsec", "crowdsec", "Path to crowdsec")
cmdHubTest.PersistentFlags().StringVar(&cscliPath, "cscli", "cscli", "Path to cscli")
parsers := []string{}
postoverflows := []string{}
scenarios := []string{}
var ignoreParsers bool
var labels map[string]string
var cmdHubTestCreate = &cobra.Command{
Use: "create",
Short: "create [test_name]",
Example: `cscli hubtest create my-awesome-test --type syslog
cscli hubtest create my-nginx-custom-test --type nginx
cscli hubtest create my-scenario-test --parsers crowdsecurity/nginx --scenarios crowdsecurity/http-probing`,
Args: cobra.ExactArgs(1),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
testName := args[0]
testPath := filepath.Join(HubTest.HubTestPath, testName)
if _, err := os.Stat(testPath); os.IsExist(err) {
log.Fatalf("test '%s' already exists in '%s', exiting", testName, testPath)
}
if logType == "" {
log.Fatalf("please provide a type (--type) for the test")
}
if err := os.MkdirAll(testPath, os.ModePerm); err != nil {
log.Fatalf("unable to create folder '%s': %+v", testPath, err)
}
// create empty log file
logFileName := fmt.Sprintf("%s.log", testName)
logFilePath := filepath.Join(testPath, logFileName)
logFile, err := os.Create(logFilePath)
if err != nil {
log.Fatal(err)
}
logFile.Close()
// create empty parser assertion file
parserAssertFilePath := filepath.Join(testPath, hubtest.ParserAssertFileName)
parserAssertFile, err := os.Create(parserAssertFilePath)
if err != nil {
log.Fatal(err)
}
parserAssertFile.Close()
// create empty scenario assertion file
scenarioAssertFilePath := filepath.Join(testPath, hubtest.ScenarioAssertFileName)
scenarioAssertFile, err := os.Create(scenarioAssertFilePath)
if err != nil {
log.Fatal(err)
}
scenarioAssertFile.Close()
parsers = append(parsers, "crowdsecurity/syslog-logs")
parsers = append(parsers, "crowdsecurity/dateparse-enrich")
if len(scenarios) == 0 {
scenarios = append(scenarios, "")
}
if len(postoverflows) == 0 {
postoverflows = append(postoverflows, "")
}
configFileData := &hubtest.HubTestItemConfig{
Parsers: parsers,
Scenarios: scenarios,
PostOVerflows: postoverflows,
LogFile: logFileName,
LogType: logType,
IgnoreParsers: ignoreParsers,
Labels: labels,
}
configFilePath := filepath.Join(testPath, "config.yaml")
fd, err := os.OpenFile(configFilePath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0666)
if err != nil {
log.Fatalf("open: %s", err)
}
data, err := yaml.Marshal(configFileData)
if err != nil {
log.Fatalf("marshal: %s", err)
}
_, err = fd.Write(data)
if err != nil {
log.Fatalf("write: %s", err)
}
if err := fd.Close(); err != nil {
log.Fatalf(" close: %s", err)
}
fmt.Println()
fmt.Printf(" Test name : %s\n", testName)
fmt.Printf(" Test path : %s\n", testPath)
fmt.Printf(" Log file : %s (please fill it with logs)\n", logFilePath)
fmt.Printf(" Parser assertion file : %s (please fill it with assertion)\n", parserAssertFilePath)
fmt.Printf(" Scenario assertion file : %s (please fill it with assertion)\n", scenarioAssertFilePath)
fmt.Printf(" Configuration File : %s (please fill it with parsers, scenarios...)\n", configFilePath)
},
}
cmdHubTestCreate.PersistentFlags().StringVarP(&logType, "type", "t", "", "Log type of the test")
cmdHubTestCreate.Flags().StringSliceVarP(&parsers, "parsers", "p", parsers, "Parsers to add to test")
cmdHubTestCreate.Flags().StringSliceVar(&postoverflows, "postoverflows", postoverflows, "Postoverflows to add to test")
cmdHubTestCreate.Flags().StringSliceVarP(&scenarios, "scenarios", "s", scenarios, "Scenarios to add to test")
cmdHubTestCreate.PersistentFlags().BoolVar(&ignoreParsers, "ignore-parsers", false, "Don't run test on parsers")
cmdHubTest.AddCommand(cmdHubTestCreate)
var noClean bool
var runAll bool
var forceClean bool
var cmdHubTestRun = &cobra.Command{
Use: "run",
Short: "run [test_name]",
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
if !runAll && len(args) == 0 {
printHelp(cmd)
fmt.Println("Please provide test to run or --all flag")
os.Exit(1)
}
if runAll {
if err := HubTest.LoadAllTests(); err != nil {
log.Fatalf("unable to load all tests: %+v", err)
}
} else {
for _, testName := range args {
_, err := HubTest.LoadTestItem(testName)
if err != nil {
log.Fatalf("unable to load test '%s': %s", testName, err)
}
}
}
for _, test := range HubTest.Tests {
if csConfig.Cscli.Output == "human" {
log.Infof("Running test '%s'", test.Name)
}
err := test.Run()
if err != nil {
log.Errorf("running test '%s' failed: %+v", test.Name, err)
}
}
},
PersistentPostRun: func(cmd *cobra.Command, args []string) {
success := true
testResult := make(map[string]bool)
for _, test := range HubTest.Tests {
if test.AutoGen {
if test.ParserAssert.AutoGenAssert {
log.Warningf("Assert file '%s' is empty, generating assertion:", test.ParserAssert.File)
fmt.Println()
fmt.Println(test.ParserAssert.AutoGenAssertData)
}
if test.ScenarioAssert.AutoGenAssert {
log.Warningf("Assert file '%s' is empty, generating assertion:", test.ScenarioAssert.File)
fmt.Println()
fmt.Println(test.ScenarioAssert.AutoGenAssertData)
}
if !noClean {
if err := test.Clean(); err != nil {
log.Fatalf("unable to clean test '%s' env: %s", test.Name, err)
}
}
fmt.Printf("\nPlease fill your assert file(s) for test '%s', exiting\n", test.Name)
os.Exit(1)
}
testResult[test.Name] = test.Success
if test.Success {
if csConfig.Cscli.Output == "human" {
log.Infof("Test '%s' passed successfully (%d assertions)\n", test.Name, test.ParserAssert.NbAssert+test.ScenarioAssert.NbAssert)
}
if !noClean {
if err := test.Clean(); err != nil {
log.Fatalf("unable to clean test '%s' env: %s", test.Name, err)
}
}
} else {
success = false
cleanTestEnv := false
if csConfig.Cscli.Output == "human" {
if len(test.ParserAssert.Fails) > 0 {
fmt.Println()
log.Errorf("Parser test '%s' failed (%d errors)\n", test.Name, len(test.ParserAssert.Fails))
for _, fail := range test.ParserAssert.Fails {
fmt.Printf("(L.%d) %s => %s\n", fail.Line, emoji.RedCircle, fail.Expression)
fmt.Printf(" Actual expression values:\n")
for key, value := range fail.Debug {
fmt.Printf(" %s = '%s'\n", key, strings.TrimSuffix(value, "\n"))
}
fmt.Println()
}
}
if len(test.ScenarioAssert.Fails) > 0 {
fmt.Println()
log.Errorf("Scenario test '%s' failed (%d errors)\n", test.Name, len(test.ScenarioAssert.Fails))
for _, fail := range test.ScenarioAssert.Fails {
fmt.Printf("(L.%d) %s => %s\n", fail.Line, emoji.RedCircle, fail.Expression)
fmt.Printf(" Actual expression values:\n")
for key, value := range fail.Debug {
fmt.Printf(" %s = '%s'\n", key, strings.TrimSuffix(value, "\n"))
}
fmt.Println()
}
}
if !forceClean && !noClean {
prompt := &survey.Confirm{
Message: fmt.Sprintf("\nDo you want to remove runtime folder for test '%s'? (default: Yes)", test.Name),
Default: true,
}
if err := survey.AskOne(prompt, &cleanTestEnv); err != nil {
log.Fatalf("unable to ask to remove runtime folder: %s", err)
}
}
}
if cleanTestEnv || forceClean {
if err := test.Clean(); err != nil {
log.Fatalf("unable to clean test '%s' env: %s", test.Name, err)
}
}
}
}
if csConfig.Cscli.Output == "human" {
hubTestResultTable(color.Output, testResult)
} else if csConfig.Cscli.Output == "json" {
jsonResult := make(map[string][]string, 0)
jsonResult["success"] = make([]string, 0)
jsonResult["fail"] = make([]string, 0)
for testName, success := range testResult {
if success {
jsonResult["success"] = append(jsonResult["success"], testName)
} else {
jsonResult["fail"] = append(jsonResult["fail"], testName)
}
}
jsonStr, err := json.Marshal(jsonResult)
if err != nil {
log.Fatalf("unable to json test result: %s", err)
}
fmt.Println(string(jsonStr))
}
if !success {
os.Exit(1)
}
},
}
cmdHubTestRun.Flags().BoolVar(&noClean, "no-clean", false, "Don't clean runtime environment if test succeed")
cmdHubTestRun.Flags().BoolVar(&forceClean, "clean", false, "Clean runtime environment if test fail")
cmdHubTestRun.Flags().BoolVar(&runAll, "all", false, "Run all tests")
cmdHubTest.AddCommand(cmdHubTestRun)
var cmdHubTestClean = &cobra.Command{
Use: "clean",
Short: "clean [test_name]",
Args: cobra.MinimumNArgs(1),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
for _, testName := range args {
test, err := HubTest.LoadTestItem(testName)
if err != nil {
log.Fatalf("unable to load test '%s': %s", testName, err)
}
if err := test.Clean(); err != nil {
log.Fatalf("unable to clean test '%s' env: %s", test.Name, err)
}
}
},
}
cmdHubTest.AddCommand(cmdHubTestClean)
var cmdHubTestInfo = &cobra.Command{
Use: "info",
Short: "info [test_name]",
Args: cobra.MinimumNArgs(1),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
for _, testName := range args {
test, err := HubTest.LoadTestItem(testName)
if err != nil {
log.Fatalf("unable to load test '%s': %s", testName, err)
}
fmt.Println()
fmt.Printf(" Test name : %s\n", test.Name)
fmt.Printf(" Test path : %s\n", test.Path)
fmt.Printf(" Log file : %s\n", filepath.Join(test.Path, test.Config.LogFile))
fmt.Printf(" Parser assertion file : %s\n", filepath.Join(test.Path, hubtest.ParserAssertFileName))
fmt.Printf(" Scenario assertion file : %s\n", filepath.Join(test.Path, hubtest.ScenarioAssertFileName))
fmt.Printf(" Configuration File : %s\n", filepath.Join(test.Path, "config.yaml"))
}
},
}
cmdHubTest.AddCommand(cmdHubTestInfo)
var cmdHubTestList = &cobra.Command{
Use: "list",
Short: "list",
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
if err := HubTest.LoadAllTests(); err != nil {
log.Fatalf("unable to load all tests: %+v", err)
}
switch csConfig.Cscli.Output {
case "human":
hubTestListTable(color.Output, HubTest.Tests)
case "json":
j, err := json.MarshalIndent(HubTest.Tests, " ", " ")
if err != nil {
log.Fatal(err)
}
fmt.Println(string(j))
default:
log.Fatalf("only human/json output modes are supported")
}
},
}
cmdHubTest.AddCommand(cmdHubTestList)
var showParserCov bool
var showScenarioCov bool
var showOnlyPercent bool
var cmdHubTestCoverage = &cobra.Command{
Use: "coverage",
Short: "coverage",
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
if err := HubTest.LoadAllTests(); err != nil {
log.Fatalf("unable to load all tests: %+v", err)
}
var err error
scenarioCoverage := []hubtest.ScenarioCoverage{}
parserCoverage := []hubtest.ParserCoverage{}
scenarioCoveragePercent := 0
parserCoveragePercent := 0
// if both are false (flag by default), show both
showAll := !showScenarioCov && !showParserCov
if showParserCov || showAll {
parserCoverage, err = HubTest.GetParsersCoverage()
if err != nil {
log.Fatalf("while getting parser coverage : %s", err)
}
parserTested := 0
for _, test := range parserCoverage {
if test.TestsCount > 0 {
parserTested += 1
}
}
parserCoveragePercent = int(math.Round((float64(parserTested) / float64(len(parserCoverage)) * 100)))
}
if showScenarioCov || showAll {
scenarioCoverage, err = HubTest.GetScenariosCoverage()
if err != nil {
log.Fatalf("while getting scenario coverage: %s", err)
}
scenarioTested := 0
for _, test := range scenarioCoverage {
if test.TestsCount > 0 {
scenarioTested += 1
}
}
scenarioCoveragePercent = int(math.Round((float64(scenarioTested) / float64(len(scenarioCoverage)) * 100)))
}
if showOnlyPercent {
if showAll {
fmt.Printf("parsers=%d%%\nscenarios=%d%%", parserCoveragePercent, scenarioCoveragePercent)
} else if showParserCov {
fmt.Printf("parsers=%d%%", parserCoveragePercent)
} else if showScenarioCov {
fmt.Printf("scenarios=%d%%", scenarioCoveragePercent)
}
os.Exit(0)
}
if csConfig.Cscli.Output == "human" {
if showParserCov || showAll {
hubTestParserCoverageTable(color.Output, parserCoverage)
}
if showScenarioCov || showAll {
hubTestScenarioCoverageTable(color.Output, scenarioCoverage)
}
fmt.Println()
if showParserCov || showAll {
fmt.Printf("PARSERS : %d%% of coverage\n", parserCoveragePercent)
}
if showScenarioCov || showAll {
fmt.Printf("SCENARIOS : %d%% of coverage\n", scenarioCoveragePercent)
}
} else if csConfig.Cscli.Output == "json" {
dump, err := json.MarshalIndent(parserCoverage, "", " ")
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s", dump)
dump, err = json.MarshalIndent(scenarioCoverage, "", " ")
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s", dump)
} else {
log.Fatalf("only human/json output modes are supported")
}
},
}
cmdHubTestCoverage.PersistentFlags().BoolVar(&showOnlyPercent, "percent", false, "Show only percentages of coverage")
cmdHubTestCoverage.PersistentFlags().BoolVar(&showParserCov, "parsers", false, "Show only parsers coverage")
cmdHubTestCoverage.PersistentFlags().BoolVar(&showScenarioCov, "scenarios", false, "Show only scenarios coverage")
cmdHubTest.AddCommand(cmdHubTestCoverage)
var evalExpression string
var cmdHubTestEval = &cobra.Command{
Use: "eval",
Short: "eval [test_name]",
Args: cobra.ExactArgs(1),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
for _, testName := range args {
test, err := HubTest.LoadTestItem(testName)
if err != nil {
log.Fatalf("can't load test: %+v", err)
}
err = test.ParserAssert.LoadTest(test.ParserResultFile)
if err != nil {
log.Fatalf("can't load test results from '%s': %+v", test.ParserResultFile, err)
}
output, err := test.ParserAssert.EvalExpression(evalExpression)
if err != nil {
log.Fatal(err)
}
fmt.Print(output)
}
},
}
cmdHubTestEval.PersistentFlags().StringVarP(&evalExpression, "expr", "e", "", "Expression to eval")
cmdHubTest.AddCommand(cmdHubTestEval)
var cmdHubTestExplain = &cobra.Command{
Use: "explain",
Short: "explain [test_name]",
Args: cobra.ExactArgs(1),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
for _, testName := range args {
test, err := HubTest.LoadTestItem(testName)
if err != nil {
log.Fatalf("can't load test: %+v", err)
}
err = test.ParserAssert.LoadTest(test.ParserResultFile)
if err != nil {
err := test.Run()
if err != nil {
log.Fatalf("running test '%s' failed: %+v", test.Name, err)
}
err = test.ParserAssert.LoadTest(test.ParserResultFile)
if err != nil {
log.Fatalf("unable to load parser result after run: %s", err)
}
}
err = test.ScenarioAssert.LoadTest(test.ScenarioResultFile, test.BucketPourResultFile)
if err != nil {
err := test.Run()
if err != nil {
log.Fatalf("running test '%s' failed: %+v", test.Name, err)
}
err = test.ScenarioAssert.LoadTest(test.ScenarioResultFile, test.BucketPourResultFile)
if err != nil {
log.Fatalf("unable to load scenario result after run: %s", err)
}
}
opts := hubtest.DumpOpts{}
hubtest.DumpTree(*test.ParserAssert.TestData, *test.ScenarioAssert.PourData, opts)
}
},
}
cmdHubTest.AddCommand(cmdHubTestExplain)
return cmdHubTest
}
crowdsec-1.4.6/cmd/crowdsec-cli/hubtest_table.go 0000664 0000000 0000000 00000004052 14371172257 0021642 0 ustar 00root root 0000000 0000000 package main
import (
"fmt"
"io"
"github.com/aquasecurity/table"
"github.com/enescakir/emoji"
"github.com/crowdsecurity/crowdsec/pkg/hubtest"
)
func hubTestResultTable(out io.Writer, testResult map[string]bool) {
t := newLightTable(out)
t.SetHeaders("Test", "Result")
t.SetHeaderAlignment(table.AlignLeft)
t.SetAlignment(table.AlignLeft)
for testName, success := range testResult {
status := emoji.CheckMarkButton.String()
if !success {
status = emoji.CrossMark.String()
}
t.AddRow(testName, status)
}
t.Render()
}
func hubTestListTable(out io.Writer, tests []*hubtest.HubTestItem) {
t := newLightTable(out)
t.SetHeaders("Name", "Path")
t.SetHeaderAlignment(table.AlignLeft, table.AlignLeft)
t.SetAlignment(table.AlignLeft, table.AlignLeft)
for _, test := range tests {
t.AddRow(test.Name, test.Path)
}
t.Render()
}
func hubTestParserCoverageTable(out io.Writer, coverage []hubtest.ParserCoverage) {
t := newLightTable(out)
t.SetHeaders("Parser", "Status", "Number of tests")
t.SetHeaderAlignment(table.AlignLeft, table.AlignLeft, table.AlignLeft)
t.SetAlignment(table.AlignLeft, table.AlignLeft, table.AlignLeft)
parserTested := 0
for _, test := range coverage {
status := emoji.RedCircle.String()
if test.TestsCount > 0 {
status = emoji.GreenCircle.String()
parserTested++
}
t.AddRow(test.Parser, status, fmt.Sprintf("%d times (across %d tests)", test.TestsCount, len(test.PresentIn)))
}
t.Render()
}
func hubTestScenarioCoverageTable(out io.Writer, coverage []hubtest.ScenarioCoverage) {
t := newLightTable(out)
t.SetHeaders("Scenario", "Status", "Number of tests")
t.SetHeaderAlignment(table.AlignLeft, table.AlignLeft, table.AlignLeft)
t.SetAlignment(table.AlignLeft, table.AlignLeft, table.AlignLeft)
parserTested := 0
for _, test := range coverage {
status := emoji.RedCircle.String()
if test.TestsCount > 0 {
status = emoji.GreenCircle.String()
parserTested++
}
t.AddRow(test.Scenario, status, fmt.Sprintf("%d times (across %d tests)", test.TestsCount, len(test.PresentIn)))
}
t.Render()
}
crowdsec-1.4.6/cmd/crowdsec-cli/lapi.go 0000664 0000000 0000000 00000012671 14371172257 0017750 0 ustar 00root root 0000000 0000000 package main
import (
"context"
"fmt"
"net/url"
"os"
"strings"
"github.com/go-openapi/strfmt"
"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"gopkg.in/yaml.v2"
"github.com/crowdsecurity/crowdsec/pkg/apiclient"
"github.com/crowdsecurity/crowdsec/pkg/csconfig"
"github.com/crowdsecurity/crowdsec/pkg/cwhub"
"github.com/crowdsecurity/crowdsec/pkg/cwversion"
"github.com/crowdsecurity/crowdsec/pkg/models"
)
var LAPIURLPrefix string = "v1"
var lapiUser string
func NewLapiCmd() *cobra.Command {
var cmdLapi = &cobra.Command{
Use: "lapi [action]",
Short: "Manage interaction with Local API (LAPI)",
Args: cobra.MinimumNArgs(1),
DisableAutoGenTag: true,
PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
if err := csConfig.LoadAPIClient(); err != nil {
return errors.Wrap(err, "loading api client")
}
return nil
},
}
var cmdLapiRegister = &cobra.Command{
Use: "register",
Short: "Register a machine to Local API (LAPI)",
Long: `Register you machine to the Local API (LAPI).
Keep in mind the machine needs to be validated by an administrator on LAPI side to be effective.`,
Args: cobra.MinimumNArgs(0),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
var err error
if lapiUser == "" {
lapiUser, err = generateID("")
if err != nil {
log.Fatalf("unable to generate machine id: %s", err)
}
}
password := strfmt.Password(generatePassword(passwordLength))
if apiURL == "" {
if csConfig.API.Client != nil && csConfig.API.Client.Credentials != nil && csConfig.API.Client.Credentials.URL != "" {
apiURL = csConfig.API.Client.Credentials.URL
} else {
log.Fatalf("No Local API URL. Please provide it in your configuration or with the -u parameter")
}
}
/*URL needs to end with /, but user doesn't care*/
if !strings.HasSuffix(apiURL, "/") {
apiURL += "/"
}
/*URL needs to start with http://, but user doesn't care*/
if !strings.HasPrefix(apiURL, "http://") && !strings.HasPrefix(apiURL, "https://") {
apiURL = "http://" + apiURL
}
apiurl, err := url.Parse(apiURL)
if err != nil {
log.Fatalf("parsing api url: %s", err)
}
_, err = apiclient.RegisterClient(&apiclient.Config{
MachineID: lapiUser,
Password: password,
UserAgent: fmt.Sprintf("crowdsec/%s", cwversion.VersionStr()),
URL: apiurl,
VersionPrefix: LAPIURLPrefix,
}, nil)
if err != nil {
log.Fatalf("api client register: %s", err)
}
log.Printf("Successfully registered to Local API (LAPI)")
var dumpFile string
if outputFile != "" {
dumpFile = outputFile
} else if csConfig.API.Client.CredentialsFilePath != "" {
dumpFile = csConfig.API.Client.CredentialsFilePath
} else {
dumpFile = ""
}
apiCfg := csconfig.ApiCredentialsCfg{
Login: lapiUser,
Password: password.String(),
URL: apiURL,
}
apiConfigDump, err := yaml.Marshal(apiCfg)
if err != nil {
log.Fatalf("unable to marshal api credentials: %s", err)
}
if dumpFile != "" {
err = os.WriteFile(dumpFile, apiConfigDump, 0644)
if err != nil {
log.Fatalf("write api credentials in '%s' failed: %s", dumpFile, err)
}
log.Printf("Local API credentials dumped to '%s'", dumpFile)
} else {
fmt.Printf("%s\n", string(apiConfigDump))
}
log.Warning(ReloadMessage())
},
}
cmdLapiRegister.Flags().StringVarP(&apiURL, "url", "u", "", "URL of the API (ie. http://127.0.0.1)")
cmdLapiRegister.Flags().StringVarP(&outputFile, "file", "f", "", "output file destination")
cmdLapiRegister.Flags().StringVar(&lapiUser, "machine", "", "Name of the machine to register with")
cmdLapi.AddCommand(cmdLapiRegister)
var cmdLapiStatus = &cobra.Command{
Use: "status",
Short: "Check authentication to Local API (LAPI)",
Args: cobra.MinimumNArgs(0),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
var err error
password := strfmt.Password(csConfig.API.Client.Credentials.Password)
apiurl, err := url.Parse(csConfig.API.Client.Credentials.URL)
login := csConfig.API.Client.Credentials.Login
if err != nil {
log.Fatalf("parsing api url ('%s'): %s", apiurl, err)
}
if err := csConfig.LoadHub(); err != nil {
log.Fatal(err)
}
if err := cwhub.GetHubIdx(csConfig.Hub); err != nil {
log.Info("Run 'sudo cscli hub update' to get the hub index")
log.Fatalf("Failed to load hub index : %s", err)
}
scenarios, err := cwhub.GetInstalledScenariosAsString()
if err != nil {
log.Fatalf("failed to get scenarios : %s", err)
}
Client, err = apiclient.NewDefaultClient(apiurl,
LAPIURLPrefix,
fmt.Sprintf("crowdsec/%s", cwversion.VersionStr()),
nil)
if err != nil {
log.Fatalf("init default client: %s", err)
}
t := models.WatcherAuthRequest{
MachineID: &login,
Password: &password,
Scenarios: scenarios,
}
log.Infof("Loaded credentials from %s", csConfig.API.Client.CredentialsFilePath)
log.Infof("Trying to authenticate with username %s on %s", login, apiurl)
_, err = Client.Auth.AuthenticateWatcher(context.Background(), t)
if err != nil {
log.Fatalf("Failed to authenticate to Local API (LAPI) : %s", err)
} else {
log.Infof("You can successfully interact with Local API (LAPI)")
}
},
}
cmdLapi.AddCommand(cmdLapiStatus)
return cmdLapi
}
crowdsec-1.4.6/cmd/crowdsec-cli/machines.go 0000664 0000000 0000000 00000025763 14371172257 0020620 0 ustar 00root root 0000000 0000000 package main
import (
saferand "crypto/rand"
"encoding/csv"
"encoding/json"
"fmt"
"io"
"math/big"
"os"
"strings"
"time"
"github.com/AlecAivazis/survey/v2"
"github.com/enescakir/emoji"
"github.com/fatih/color"
"github.com/go-openapi/strfmt"
"github.com/google/uuid"
"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"gopkg.in/yaml.v2"
"github.com/crowdsecurity/machineid"
"github.com/crowdsecurity/crowdsec/pkg/csconfig"
"github.com/crowdsecurity/crowdsec/pkg/database"
"github.com/crowdsecurity/crowdsec/pkg/database/ent"
"github.com/crowdsecurity/crowdsec/pkg/types"
)
var machineID string
var machinePassword string
var interactive bool
var apiURL string
var outputFile string
var forceAdd bool
var autoAdd bool
var (
passwordLength = 64
upper = "ABCDEFGHIJKLMNOPQRSTUVWXY"
lower = "abcdefghijklmnopqrstuvwxyz"
digits = "0123456789"
)
func generatePassword(length int) string {
charset := upper + lower + digits
charsetLength := len(charset)
buf := make([]byte, length)
for i := 0; i < length; i++ {
rInt, err := saferand.Int(saferand.Reader, big.NewInt(int64(charsetLength)))
if err != nil {
log.Fatalf("failed getting data from prng for password generation : %s", err)
}
buf[i] = charset[rInt.Int64()]
}
return string(buf)
}
// Returns a unique identifier for each crowdsec installation, using an
// identifier of the OS installation where available, otherwise a random
// string.
func generateIDPrefix() (string, error) {
prefix, err := machineid.ID()
if err == nil {
return prefix, nil
}
log.Debugf("failed to get machine-id with usual files: %s", err)
bId, err := uuid.NewRandom()
if err == nil {
return bId.String(), nil
}
return "", errors.Wrap(err, "generating machine id")
}
// Generate a unique identifier, composed by a prefix and a random suffix.
// The prefix can be provided by a parameter to use in test environments.
func generateID(prefix string) (string, error) {
var err error
if prefix == "" {
prefix, err = generateIDPrefix()
}
if err != nil {
return "", err
}
prefix = strings.ReplaceAll(prefix, "-", "")[:32]
suffix := generatePassword(16)
return prefix + suffix, nil
}
func displayLastHeartBeat(m *ent.Machine, fancy bool) string {
var hbDisplay string
if m.LastHeartbeat != nil {
lastHeartBeat := time.Now().UTC().Sub(*m.LastHeartbeat)
hbDisplay = lastHeartBeat.Truncate(time.Second).String()
if fancy && lastHeartBeat > 2*time.Minute {
hbDisplay = fmt.Sprintf("%s %s", emoji.Warning.String(), lastHeartBeat.Truncate(time.Second).String())
}
} else {
hbDisplay = "-"
if fancy {
hbDisplay = emoji.Warning.String() + " -"
}
}
return hbDisplay
}
func getAgents(out io.Writer, dbClient *database.Client) error {
machines, err := dbClient.ListMachines()
if err != nil {
return fmt.Errorf("unable to list machines: %s", err)
}
if csConfig.Cscli.Output == "human" {
getAgentsTable(out, machines)
} else if csConfig.Cscli.Output == "json" {
enc := json.NewEncoder(out)
enc.SetIndent("", " ")
if err := enc.Encode(machines); err != nil {
log.Fatalf("failed to unmarshal")
}
return nil
} else if csConfig.Cscli.Output == "raw" {
csvwriter := csv.NewWriter(out)
err := csvwriter.Write([]string{"machine_id", "ip_address", "updated_at", "validated", "version", "auth_type", "last_heartbeat"})
if err != nil {
log.Fatalf("failed to write header: %s", err)
}
for _, m := range machines {
var validated string
if m.IsValidated {
validated = "true"
} else {
validated = "false"
}
err := csvwriter.Write([]string{m.MachineId, m.IpAddress, m.UpdatedAt.Format(time.RFC3339), validated, m.Version, m.AuthType, displayLastHeartBeat(m, false)})
if err != nil {
log.Fatalf("failed to write raw output : %s", err)
}
}
csvwriter.Flush()
} else {
log.Errorf("unknown output '%s'", csConfig.Cscli.Output)
}
return nil
}
func NewMachinesCmd() *cobra.Command {
/* ---- DECISIONS COMMAND */
var cmdMachines = &cobra.Command{
Use: "machines [action]",
Short: "Manage local API machines [requires local API]",
Long: `To list/add/delete/validate machines.
Note: This command requires database direct access, so is intended to be run on the local API machine.
`,
Example: `cscli machines [action]`,
DisableAutoGenTag: true,
Aliases: []string{"machine"},
PersistentPreRun: func(cmd *cobra.Command, args []string) {
if err := csConfig.LoadAPIServer(); err != nil || csConfig.DisableAPI {
if err != nil {
log.Errorf("local api : %s", err)
}
log.Fatal("Local API is disabled, please run this command on the local API machine")
}
if err := csConfig.LoadDBConfig(); err != nil {
log.Errorf("This command requires direct database access (must be run on the local API machine)")
log.Fatal(err)
}
},
}
var cmdMachinesList = &cobra.Command{
Use: "list",
Short: "List machines",
Long: `List `,
Example: `cscli machines list`,
Args: cobra.MaximumNArgs(1),
DisableAutoGenTag: true,
PreRun: func(cmd *cobra.Command, args []string) {
var err error
dbClient, err = database.NewClient(csConfig.DbConfig)
if err != nil {
log.Fatalf("unable to create new database client: %s", err)
}
},
Run: func(cmd *cobra.Command, args []string) {
err := getAgents(color.Output, dbClient)
if err != nil {
log.Fatalf("unable to list machines: %s", err)
}
},
}
cmdMachines.AddCommand(cmdMachinesList)
var cmdMachinesAdd = &cobra.Command{
Use: "add",
Short: "add machine to the database.",
DisableAutoGenTag: true,
Long: `Register a new machine in the database. cscli should be on the same machine as LAPI.`,
Example: `
cscli machines add --auto
cscli machines add MyTestMachine --auto
cscli machines add MyTestMachine --password MyPassword
`,
PreRun: func(cmd *cobra.Command, args []string) {
var err error
dbClient, err = database.NewClient(csConfig.DbConfig)
if err != nil {
log.Fatalf("unable to create new database client: %s", err)
}
},
Run: func(cmd *cobra.Command, args []string) {
var dumpFile string
var err error
// create machineID if not specified by user
if len(args) == 0 {
if !autoAdd {
printHelp(cmd)
return
}
machineID, err = generateID("")
if err != nil {
log.Fatalf("unable to generate machine id : %s", err)
}
} else {
machineID = args[0]
}
/*check if file already exists*/
if outputFile != "" {
dumpFile = outputFile
} else if csConfig.API.Client != nil && csConfig.API.Client.CredentialsFilePath != "" {
dumpFile = csConfig.API.Client.CredentialsFilePath
}
// create a password if it's not specified by user
if machinePassword == "" && !interactive {
if !autoAdd {
printHelp(cmd)
return
}
machinePassword = generatePassword(passwordLength)
} else if machinePassword == "" && interactive {
qs := &survey.Password{
Message: "Please provide a password for the machine",
}
survey.AskOne(qs, &machinePassword)
}
password := strfmt.Password(machinePassword)
_, err = dbClient.CreateMachine(&machineID, &password, "", true, forceAdd, types.PasswordAuthType)
if err != nil {
log.Fatalf("unable to create machine: %s", err)
}
log.Infof("Machine '%s' successfully added to the local API", machineID)
if apiURL == "" {
if csConfig.API.Client != nil && csConfig.API.Client.Credentials != nil && csConfig.API.Client.Credentials.URL != "" {
apiURL = csConfig.API.Client.Credentials.URL
} else if csConfig.API.Server != nil && csConfig.API.Server.ListenURI != "" {
apiURL = "http://" + csConfig.API.Server.ListenURI
} else {
log.Fatalf("unable to dump an api URL. Please provide it in your configuration or with the -u parameter")
}
}
apiCfg := csconfig.ApiCredentialsCfg{
Login: machineID,
Password: password.String(),
URL: apiURL,
}
apiConfigDump, err := yaml.Marshal(apiCfg)
if err != nil {
log.Fatalf("unable to marshal api credentials: %s", err)
}
if dumpFile != "" && dumpFile != "-" {
err = os.WriteFile(dumpFile, apiConfigDump, 0644)
if err != nil {
log.Fatalf("write api credentials in '%s' failed: %s", dumpFile, err)
}
log.Printf("API credentials dumped to '%s'", dumpFile)
} else {
fmt.Printf("%s\n", string(apiConfigDump))
}
},
}
cmdMachinesAdd.Flags().StringVarP(&machinePassword, "password", "p", "", "machine password to login to the API")
cmdMachinesAdd.Flags().StringVarP(&outputFile, "file", "f", "",
"output file destination (defaults to "+csconfig.DefaultConfigPath("local_api_credentials.yaml"))
cmdMachinesAdd.Flags().StringVarP(&apiURL, "url", "u", "", "URL of the local API")
cmdMachinesAdd.Flags().BoolVarP(&interactive, "interactive", "i", false, "interfactive mode to enter the password")
cmdMachinesAdd.Flags().BoolVarP(&autoAdd, "auto", "a", false, "automatically generate password (and username if not provided)")
cmdMachinesAdd.Flags().BoolVar(&forceAdd, "force", false, "will force add the machine if it already exist")
cmdMachines.AddCommand(cmdMachinesAdd)
var cmdMachinesDelete = &cobra.Command{
Use: "delete --machine MyTestMachine",
Short: "delete machines",
Example: `cscli machines delete "machine_name"`,
Args: cobra.MinimumNArgs(1),
Aliases: []string{"remove"},
DisableAutoGenTag: true,
PreRun: func(cmd *cobra.Command, args []string) {
var err error
dbClient, err = database.NewClient(csConfig.DbConfig)
if err != nil {
log.Fatalf("unable to create new database client: %s", err)
}
},
Run: func(cmd *cobra.Command, args []string) {
machineID = args[0]
for _, machineID := range args {
err := dbClient.DeleteWatcher(machineID)
if err != nil {
log.Errorf("unable to delete machine '%s': %s", machineID, err)
return
}
log.Infof("machine '%s' deleted successfully", machineID)
}
},
}
cmdMachinesDelete.Flags().StringVarP(&machineID, "machine", "m", "", "machine to delete")
cmdMachines.AddCommand(cmdMachinesDelete)
var cmdMachinesValidate = &cobra.Command{
Use: "validate",
Short: "validate a machine to access the local API",
Long: `validate a machine to access the local API.`,
Example: `cscli machines validate "machine_name"`,
Args: cobra.ExactArgs(1),
DisableAutoGenTag: true,
PreRun: func(cmd *cobra.Command, args []string) {
var err error
dbClient, err = database.NewClient(csConfig.DbConfig)
if err != nil {
log.Fatalf("unable to create new database client: %s", err)
}
},
Run: func(cmd *cobra.Command, args []string) {
machineID = args[0]
if err := dbClient.ValidateMachine(machineID); err != nil {
log.Fatalf("unable to validate machine '%s': %s", machineID, err)
}
log.Infof("machine '%s' validated successfully", machineID)
},
}
cmdMachines.AddCommand(cmdMachinesValidate)
return cmdMachines
}
crowdsec-1.4.6/cmd/crowdsec-cli/machines_table.go 0000664 0000000 0000000 00000001667 14371172257 0021764 0 ustar 00root root 0000000 0000000 package main
import (
"io"
"time"
"github.com/aquasecurity/table"
"github.com/enescakir/emoji"
"github.com/crowdsecurity/crowdsec/pkg/database/ent"
)
func getAgentsTable(out io.Writer, machines []*ent.Machine) {
t := newLightTable(out)
t.SetHeaders("Name", "IP Address", "Last Update", "Status", "Version", "Auth Type", "Last Heartbeat")
t.SetHeaderAlignment(table.AlignLeft, table.AlignLeft, table.AlignLeft, table.AlignLeft, table.AlignLeft, table.AlignLeft, table.AlignLeft)
t.SetAlignment(table.AlignLeft, table.AlignLeft, table.AlignLeft, table.AlignLeft, table.AlignLeft, table.AlignLeft, table.AlignLeft)
for _, m := range machines {
var validated string
if m.IsValidated {
validated = emoji.CheckMark.String()
} else {
validated = emoji.Prohibited.String()
}
t.AddRow(m.MachineId, m.IpAddress, m.UpdatedAt.Format(time.RFC3339), validated, m.Version, m.AuthType, displayLastHeartBeat(m, true))
}
t.Render()
}
crowdsec-1.4.6/cmd/crowdsec-cli/main.go 0000664 0000000 0000000 00000015540 14371172257 0017745 0 ustar 00root root 0000000 0000000 package main
import (
"fmt"
"os"
"path"
"path/filepath"
"strings"
"github.com/confluentinc/bincover"
"github.com/fatih/color"
cc "github.com/ivanpirog/coloredcobra"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"github.com/spf13/cobra/doc"
"github.com/crowdsecurity/crowdsec/pkg/csconfig"
"github.com/crowdsecurity/crowdsec/pkg/cwhub"
"github.com/crowdsecurity/crowdsec/pkg/cwversion"
"github.com/crowdsecurity/crowdsec/pkg/database"
)
var bincoverTesting = ""
var trace_lvl, dbg_lvl, nfo_lvl, wrn_lvl, err_lvl bool
var ConfigFilePath string
var csConfig *csconfig.Config
var dbClient *database.Client
var OutputFormat string
var OutputColor string
var downloadOnly bool
var forceAction bool
var purge bool
var all bool
var restoreOldBackup bool
var prometheusURL string
func initConfig() {
var err error
if trace_lvl {
log.SetLevel(log.TraceLevel)
} else if dbg_lvl {
log.SetLevel(log.DebugLevel)
} else if nfo_lvl {
log.SetLevel(log.InfoLevel)
} else if wrn_lvl {
log.SetLevel(log.WarnLevel)
} else if err_lvl {
log.SetLevel(log.ErrorLevel)
}
logFormatter := &log.TextFormatter{TimestampFormat: "02-01-2006 15:04:05", FullTimestamp: true}
log.SetFormatter(logFormatter)
if !inSlice(os.Args[1], NoNeedConfig) {
csConfig, err = csconfig.NewConfig(ConfigFilePath, false, false)
if err != nil {
log.Fatal(err)
}
log.Debugf("Using %s as configuration file", ConfigFilePath)
if err := csConfig.LoadCSCLI(); err != nil {
log.Fatal(err)
}
} else {
csConfig = csconfig.NewDefaultConfig()
}
if csConfig.Cscli == nil {
log.Fatalf("missing 'cscli' configuration in '%s', exiting", ConfigFilePath)
}
if cwhub.HubBranch == "" && csConfig.Cscli.HubBranch != "" {
cwhub.HubBranch = csConfig.Cscli.HubBranch
}
if OutputFormat != "" {
csConfig.Cscli.Output = OutputFormat
if OutputFormat != "json" && OutputFormat != "raw" && OutputFormat != "human" {
log.Fatalf("output format %s unknown", OutputFormat)
}
}
if csConfig.Cscli.Output == "" {
csConfig.Cscli.Output = "human"
}
if csConfig.Cscli.Output == "json" {
log.SetFormatter(&log.JSONFormatter{})
log.SetLevel(log.ErrorLevel)
} else if csConfig.Cscli.Output == "raw" {
log.SetLevel(log.ErrorLevel)
}
if OutputColor != "" {
csConfig.Cscli.Color = OutputColor
if OutputColor != "yes" && OutputColor != "no" && OutputColor != "auto" {
log.Fatalf("output color %s unknown", OutputColor)
}
}
}
var validArgs = []string{
"scenarios", "parsers", "collections", "capi", "lapi", "postoverflows", "machines",
"metrics", "bouncers", "alerts", "decisions", "simulation", "hub", "dashboard",
"config", "completion", "version", "console", "notifications", "support",
}
func prepender(filename string) string {
const header = `---
id: %s
title: %s
---
`
name := filepath.Base(filename)
base := strings.TrimSuffix(name, path.Ext(name))
return fmt.Sprintf(header, base, strings.ReplaceAll(base, "_", " "))
}
func linkHandler(name string) string {
return fmt.Sprintf("/cscli/%s", name)
}
var (
NoNeedConfig = []string{
"help",
"completion",
"version",
"hubtest",
}
)
func main() {
var rootCmd = &cobra.Command{
Use: "cscli",
Short: "cscli allows you to manage crowdsec",
Long: `cscli is the main command to interact with your crowdsec service, scenarios & db.
It is meant to allow you to manage bans, parsers/scenarios/etc, api and generally manage you crowdsec setup.`,
ValidArgs: validArgs,
DisableAutoGenTag: true,
SilenceErrors: true,
SilenceUsage: true,
/*TBD examples*/
}
cc.Init(&cc.Config{
RootCmd: rootCmd,
Headings: cc.Yellow,
Commands: cc.Green + cc.Bold,
CmdShortDescr: cc.Cyan,
Example: cc.Italic,
ExecName: cc.Bold,
Aliases: cc.Bold + cc.Italic,
FlagsDataType: cc.White,
Flags: cc.Green,
FlagsDescr: cc.Cyan,
})
rootCmd.SetOut(color.Output)
var cmdDocGen = &cobra.Command{
Use: "doc",
Short: "Generate the documentation in `./doc/`. Directory must exist.",
Args: cobra.ExactArgs(0),
Hidden: true,
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
if err := doc.GenMarkdownTreeCustom(rootCmd, "./doc/", prepender, linkHandler); err != nil {
log.Fatalf("Failed to generate cobra doc: %s", err)
}
},
}
rootCmd.AddCommand(cmdDocGen)
/*usage*/
var cmdVersion = &cobra.Command{
Use: "version",
Short: "Display version and exit.",
Args: cobra.ExactArgs(0),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
cwversion.Show()
},
}
rootCmd.AddCommand(cmdVersion)
rootCmd.PersistentFlags().StringVarP(&ConfigFilePath, "config", "c", csconfig.DefaultConfigPath("config.yaml"), "path to crowdsec config file")
rootCmd.PersistentFlags().StringVarP(&OutputFormat, "output", "o", "", "Output format: human, json, raw.")
rootCmd.PersistentFlags().StringVarP(&OutputColor, "color", "", "auto", "Output color: yes, no, auto.")
rootCmd.PersistentFlags().BoolVar(&dbg_lvl, "debug", false, "Set logging to debug.")
rootCmd.PersistentFlags().BoolVar(&nfo_lvl, "info", false, "Set logging to info.")
rootCmd.PersistentFlags().BoolVar(&wrn_lvl, "warning", false, "Set logging to warning.")
rootCmd.PersistentFlags().BoolVar(&err_lvl, "error", false, "Set logging to error.")
rootCmd.PersistentFlags().BoolVar(&trace_lvl, "trace", false, "Set logging to trace.")
rootCmd.PersistentFlags().StringVar(&cwhub.HubBranch, "branch", "", "Override hub branch on github")
if err := rootCmd.PersistentFlags().MarkHidden("branch"); err != nil {
log.Fatalf("failed to hide flag: %s", err)
}
if len(os.Args) > 1 {
cobra.OnInitialize(initConfig)
}
/*don't sort flags so we can enforce order*/
rootCmd.Flags().SortFlags = false
rootCmd.PersistentFlags().SortFlags = false
rootCmd.AddCommand(NewConfigCmd())
rootCmd.AddCommand(NewHubCmd())
rootCmd.AddCommand(NewMetricsCmd())
rootCmd.AddCommand(NewDashboardCmd())
rootCmd.AddCommand(NewDecisionsCmd())
rootCmd.AddCommand(NewAlertsCmd())
// rootCmd.AddCommand(NewInspectCmd())
rootCmd.AddCommand(NewSimulationCmds())
rootCmd.AddCommand(NewBouncersCmd())
rootCmd.AddCommand(NewMachinesCmd())
rootCmd.AddCommand(NewParsersCmd())
rootCmd.AddCommand(NewScenariosCmd())
rootCmd.AddCommand(NewCollectionsCmd())
rootCmd.AddCommand(NewPostOverflowsCmd())
rootCmd.AddCommand(NewCapiCmd())
rootCmd.AddCommand(NewLapiCmd())
rootCmd.AddCommand(NewCompletionCmd())
rootCmd.AddCommand(NewConsoleCmd())
rootCmd.AddCommand(NewExplainCmd())
rootCmd.AddCommand(NewHubTestCmd())
rootCmd.AddCommand(NewNotificationsCmd())
rootCmd.AddCommand(NewSupportCmd())
if err := rootCmd.Execute(); err != nil {
if bincoverTesting != "" {
log.Debug("coverage report is enabled")
}
exitCode := 1
log.NewEntry(log.StandardLogger()).Log(log.FatalLevel, err)
if bincoverTesting == "" {
os.Exit(exitCode)
}
bincover.ExitCode = exitCode
}
}
crowdsec-1.4.6/cmd/crowdsec-cli/main_test.go 0000664 0000000 0000000 00000000245 14371172257 0021000 0 ustar 00root root 0000000 0000000 //go:build testrunmain
package main
import (
"testing"
"github.com/confluentinc/bincover"
)
func TestBincoverRunMain(t *testing.T) {
bincover.RunTest(main)
}
crowdsec-1.4.6/cmd/crowdsec-cli/messages.go 0000664 0000000 0000000 00000001000 14371172257 0020612 0 ustar 00root root 0000000 0000000 package main
import (
"fmt"
"runtime"
)
// ReloadMessage returns a description of the task required to reload
// the crowdsec configuration, according to the operating system.
func ReloadMessage() string {
var msg string
switch runtime.GOOS {
case "windows":
msg = "Please restart the crowdsec service"
case "freebsd":
msg = `Run 'sudo service crowdsec reload'`
default:
msg = `Run 'sudo systemctl reload crowdsec'`
}
return fmt.Sprintf("%s for the new configuration to be effective.", msg)
}
crowdsec-1.4.6/cmd/crowdsec-cli/metrics.go 0000664 0000000 0000000 00000022152 14371172257 0020464 0 ustar 00root root 0000000 0000000 package main
import (
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
"github.com/fatih/color"
dto "github.com/prometheus/client_model/go"
"github.com/prometheus/prom2json"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"gopkg.in/yaml.v2"
"github.com/crowdsecurity/crowdsec/pkg/types"
)
// FormatPrometheusMetrics is a complete rip from prom2json
func FormatPrometheusMetrics(out io.Writer, url string, formatType string) error {
mfChan := make(chan *dto.MetricFamily, 1024)
// Start with the DefaultTransport for sane defaults.
transport := http.DefaultTransport.(*http.Transport).Clone()
// Conservatively disable HTTP keep-alives as this program will only
// ever need a single HTTP request.
transport.DisableKeepAlives = true
// Timeout early if the server doesn't even return the headers.
transport.ResponseHeaderTimeout = time.Minute
go func() {
defer types.CatchPanic("crowdsec/ShowPrometheus")
err := prom2json.FetchMetricFamilies(url, mfChan, transport)
if err != nil {
log.Fatalf("failed to fetch prometheus metrics : %v", err)
}
}()
result := []*prom2json.Family{}
for mf := range mfChan {
result = append(result, prom2json.NewFamily(mf))
}
log.Debugf("Finished reading prometheus output, %d entries", len(result))
/*walk*/
lapi_decisions_stats := map[string]struct {
NonEmpty int
Empty int
}{}
acquis_stats := map[string]map[string]int{}
parsers_stats := map[string]map[string]int{}
buckets_stats := map[string]map[string]int{}
lapi_stats := map[string]map[string]int{}
lapi_machine_stats := map[string]map[string]map[string]int{}
lapi_bouncer_stats := map[string]map[string]map[string]int{}
decisions_stats := map[string]map[string]map[string]int{}
alerts_stats := map[string]int{}
for idx, fam := range result {
if !strings.HasPrefix(fam.Name, "cs_") {
continue
}
log.Tracef("round %d", idx)
for _, m := range fam.Metrics {
metric, ok := m.(prom2json.Metric)
if !ok {
log.Debugf("failed to convert metric to prom2json.Metric")
continue
}
name, ok := metric.Labels["name"]
if !ok {
log.Debugf("no name in Metric %v", metric.Labels)
}
source, ok := metric.Labels["source"]
if !ok {
log.Debugf("no source in Metric %v for %s", metric.Labels, fam.Name)
} else {
if srctype, ok := metric.Labels["type"]; ok {
source = srctype + ":" + source
}
}
value := m.(prom2json.Metric).Value
machine := metric.Labels["machine"]
bouncer := metric.Labels["bouncer"]
route := metric.Labels["route"]
method := metric.Labels["method"]
reason := metric.Labels["reason"]
origin := metric.Labels["origin"]
action := metric.Labels["action"]
fval, err := strconv.ParseFloat(value, 32)
if err != nil {
log.Errorf("Unexpected int value %s : %s", value, err)
}
ival := int(fval)
switch fam.Name {
/*buckets*/
case "cs_bucket_created_total":
if _, ok := buckets_stats[name]; !ok {
buckets_stats[name] = make(map[string]int)
}
buckets_stats[name]["instantiation"] += ival
case "cs_buckets":
if _, ok := buckets_stats[name]; !ok {
buckets_stats[name] = make(map[string]int)
}
buckets_stats[name]["curr_count"] += ival
case "cs_bucket_overflowed_total":
if _, ok := buckets_stats[name]; !ok {
buckets_stats[name] = make(map[string]int)
}
buckets_stats[name]["overflow"] += ival
case "cs_bucket_poured_total":
if _, ok := buckets_stats[name]; !ok {
buckets_stats[name] = make(map[string]int)
}
if _, ok := acquis_stats[source]; !ok {
acquis_stats[source] = make(map[string]int)
}
buckets_stats[name]["pour"] += ival
acquis_stats[source]["pour"] += ival
case "cs_bucket_underflowed_total":
if _, ok := buckets_stats[name]; !ok {
buckets_stats[name] = make(map[string]int)
}
buckets_stats[name]["underflow"] += ival
/*acquis*/
case "cs_parser_hits_total":
if _, ok := acquis_stats[source]; !ok {
acquis_stats[source] = make(map[string]int)
}
acquis_stats[source]["reads"] += ival
case "cs_parser_hits_ok_total":
if _, ok := acquis_stats[source]; !ok {
acquis_stats[source] = make(map[string]int)
}
acquis_stats[source]["parsed"] += ival
case "cs_parser_hits_ko_total":
if _, ok := acquis_stats[source]; !ok {
acquis_stats[source] = make(map[string]int)
}
acquis_stats[source]["unparsed"] += ival
case "cs_node_hits_total":
if _, ok := parsers_stats[name]; !ok {
parsers_stats[name] = make(map[string]int)
}
parsers_stats[name]["hits"] += ival
case "cs_node_hits_ok_total":
if _, ok := parsers_stats[name]; !ok {
parsers_stats[name] = make(map[string]int)
}
parsers_stats[name]["parsed"] += ival
case "cs_node_hits_ko_total":
if _, ok := parsers_stats[name]; !ok {
parsers_stats[name] = make(map[string]int)
}
parsers_stats[name]["unparsed"] += ival
case "cs_lapi_route_requests_total":
if _, ok := lapi_stats[route]; !ok {
lapi_stats[route] = make(map[string]int)
}
lapi_stats[route][method] += ival
case "cs_lapi_machine_requests_total":
if _, ok := lapi_machine_stats[machine]; !ok {
lapi_machine_stats[machine] = make(map[string]map[string]int)
}
if _, ok := lapi_machine_stats[machine][route]; !ok {
lapi_machine_stats[machine][route] = make(map[string]int)
}
lapi_machine_stats[machine][route][method] += ival
case "cs_lapi_bouncer_requests_total":
if _, ok := lapi_bouncer_stats[bouncer]; !ok {
lapi_bouncer_stats[bouncer] = make(map[string]map[string]int)
}
if _, ok := lapi_bouncer_stats[bouncer][route]; !ok {
lapi_bouncer_stats[bouncer][route] = make(map[string]int)
}
lapi_bouncer_stats[bouncer][route][method] += ival
case "cs_lapi_decisions_ko_total", "cs_lapi_decisions_ok_total":
if _, ok := lapi_decisions_stats[bouncer]; !ok {
lapi_decisions_stats[bouncer] = struct {
NonEmpty int
Empty int
}{}
}
x := lapi_decisions_stats[bouncer]
if fam.Name == "cs_lapi_decisions_ko_total" {
x.Empty += ival
} else if fam.Name == "cs_lapi_decisions_ok_total" {
x.NonEmpty += ival
}
lapi_decisions_stats[bouncer] = x
case "cs_active_decisions":
if _, ok := decisions_stats[reason]; !ok {
decisions_stats[reason] = make(map[string]map[string]int)
}
if _, ok := decisions_stats[reason][origin]; !ok {
decisions_stats[reason][origin] = make(map[string]int)
}
decisions_stats[reason][origin][action] += ival
case "cs_alerts":
/*if _, ok := alerts_stats[scenario]; !ok {
alerts_stats[scenario] = make(map[string]int)
}*/
alerts_stats[reason] += ival
default:
continue
}
}
}
if formatType == "human" {
acquisStatsTable(out, acquis_stats)
bucketStatsTable(out, buckets_stats)
parserStatsTable(out, parsers_stats)
lapiStatsTable(out, lapi_stats)
lapiMachineStatsTable(out, lapi_machine_stats)
lapiBouncerStatsTable(out, lapi_bouncer_stats)
lapiDecisionStatsTable(out, lapi_decisions_stats)
decisionStatsTable(out, decisions_stats)
alertStatsTable(out, alerts_stats)
} else if formatType == "json" {
for _, val := range []interface{}{acquis_stats, parsers_stats, buckets_stats, lapi_stats, lapi_bouncer_stats, lapi_machine_stats, lapi_decisions_stats, decisions_stats, alerts_stats} {
x, err := json.MarshalIndent(val, "", " ")
if err != nil {
return fmt.Errorf("failed to unmarshal metrics : %v", err)
}
out.Write(x)
}
return nil
} else if formatType == "raw" {
for _, val := range []interface{}{acquis_stats, parsers_stats, buckets_stats, lapi_stats, lapi_bouncer_stats, lapi_machine_stats, lapi_decisions_stats, decisions_stats, alerts_stats} {
x, err := yaml.Marshal(val)
if err != nil {
return fmt.Errorf("failed to unmarshal metrics : %v", err)
}
out.Write(x)
}
return nil
}
return nil
}
var noUnit bool
func NewMetricsCmd() *cobra.Command {
/* ---- UPDATE COMMAND */
var cmdMetrics = &cobra.Command{
Use: "metrics",
Short: "Display crowdsec prometheus metrics.",
Long: `Fetch metrics from the prometheus server and display them in a human-friendly way`,
Args: cobra.ExactArgs(0),
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
if err := csConfig.LoadPrometheus(); err != nil {
log.Fatal(err)
}
if !csConfig.Prometheus.Enabled {
log.Warning("Prometheus is not enabled, can't show metrics")
os.Exit(1)
}
if prometheusURL == "" {
prometheusURL = csConfig.Cscli.PrometheusUrl
}
if prometheusURL == "" {
log.Errorf("No prometheus url, please specify in %s or via -u", *csConfig.FilePath)
os.Exit(1)
}
err := FormatPrometheusMetrics(color.Output, prometheusURL+"/metrics", csConfig.Cscli.Output)
if err != nil {
log.Fatalf("could not fetch prometheus metrics: %s", err)
}
},
}
cmdMetrics.PersistentFlags().StringVarP(&prometheusURL, "url", "u", "", "Prometheus url (http://:/metrics)")
cmdMetrics.PersistentFlags().BoolVar(&noUnit, "no-unit", false, "Show the real number instead of formatted with units")
return cmdMetrics
}
crowdsec-1.4.6/cmd/crowdsec-cli/metrics_table.go 0000664 0000000 0000000 00000014667 14371172257 0021647 0 ustar 00root root 0000000 0000000 package main
import (
"fmt"
"io"
"sort"
"github.com/aquasecurity/table"
log "github.com/sirupsen/logrus"
)
func lapiMetricsToTable(t *table.Table, stats map[string]map[string]map[string]int) int {
// stats: machine -> route -> method -> count
// sort keys to keep consistent order when printing
machineKeys := []string{}
for k := range stats {
machineKeys = append(machineKeys, k)
}
sort.Strings(machineKeys)
numRows := 0
for _, machine := range machineKeys {
// oneRow: route -> method -> count
machineRow := stats[machine]
for routeName, route := range machineRow {
for methodName, count := range route {
row := []string{
machine,
routeName,
methodName,
}
if count != 0 {
row = append(row, fmt.Sprintf("%d", count))
} else {
row = append(row, "-")
}
t.AddRow(row...)
numRows++
}
}
}
return numRows
}
func metricsToTable(t *table.Table, stats map[string]map[string]int, keys []string) (int, error) {
if t == nil {
return 0, fmt.Errorf("nil table")
}
// sort keys to keep consistent order when printing
sortedKeys := []string{}
for k := range stats {
sortedKeys = append(sortedKeys, k)
}
sort.Strings(sortedKeys)
numRows := 0
for _, alabel := range sortedKeys {
astats, ok := stats[alabel]
if !ok {
continue
}
row := []string{
alabel,
}
for _, sl := range keys {
if v, ok := astats[sl]; ok && v != 0 {
numberToShow := fmt.Sprintf("%d", v)
if !noUnit {
numberToShow = formatNumber(v)
}
row = append(row, numberToShow)
} else {
row = append(row, "-")
}
}
t.AddRow(row...)
numRows++
}
return numRows, nil
}
func bucketStatsTable(out io.Writer, stats map[string]map[string]int) {
t := newTable(out)
t.SetRowLines(false)
t.SetHeaders("Bucket", "Current Count", "Overflows", "Instantiated", "Poured", "Expired")
t.SetAlignment(table.AlignLeft, table.AlignLeft, table.AlignLeft, table.AlignLeft, table.AlignLeft, table.AlignLeft)
keys := []string{"curr_count", "overflow", "instantiation", "pour", "underflow"}
if numRows, err := metricsToTable(t, stats, keys); err != nil {
log.Warningf("while collecting acquis stats: %s", err)
} else if numRows > 0 {
renderTableTitle(out, "\nBucket Metrics:")
t.Render()
}
}
func acquisStatsTable(out io.Writer, stats map[string]map[string]int) {
t := newTable(out)
t.SetRowLines(false)
t.SetHeaders("Source", "Lines read", "Lines parsed", "Lines unparsed", "Lines poured to bucket")
t.SetAlignment(table.AlignLeft, table.AlignLeft, table.AlignLeft, table.AlignLeft, table.AlignLeft)
keys := []string{"reads", "parsed", "unparsed", "pour"}
if numRows, err := metricsToTable(t, stats, keys); err != nil {
log.Warningf("while collecting acquis stats: %s", err)
} else if numRows > 0 {
renderTableTitle(out, "\nAcquisition Metrics:")
t.Render()
}
}
func parserStatsTable(out io.Writer, stats map[string]map[string]int) {
t := newTable(out)
t.SetRowLines(false)
t.SetHeaders("Parsers", "Hits", "Parsed", "Unparsed")
t.SetAlignment(table.AlignLeft, table.AlignLeft, table.AlignLeft, table.AlignLeft)
keys := []string{"hits", "parsed", "unparsed"}
if numRows, err := metricsToTable(t, stats, keys); err != nil {
log.Warningf("while collecting acquis stats: %s", err)
} else if numRows > 0 {
renderTableTitle(out, "\nParser Metrics:")
t.Render()
}
}
func lapiStatsTable(out io.Writer, stats map[string]map[string]int) {
t := newTable(out)
t.SetRowLines(false)
t.SetHeaders("Route", "Method", "Hits")
t.SetAlignment(table.AlignLeft, table.AlignLeft, table.AlignLeft)
// unfortunately, we can't reuse metricsToTable as the structure is too different :/
sortedKeys := []string{}
for k := range stats {
sortedKeys = append(sortedKeys, k)
}
sort.Strings(sortedKeys)
numRows := 0
for _, alabel := range sortedKeys {
astats := stats[alabel]
subKeys := []string{}
for skey := range astats {
subKeys = append(subKeys, skey)
}
sort.Strings(subKeys)
for _, sl := range subKeys {
row := []string{
alabel,
sl,
fmt.Sprintf("%d", astats[sl]),
}
t.AddRow(row...)
numRows++
}
}
if numRows > 0 {
renderTableTitle(out, "\nLocal Api Metrics:")
t.Render()
}
}
func lapiMachineStatsTable(out io.Writer, stats map[string]map[string]map[string]int) {
t := newTable(out)
t.SetRowLines(false)
t.SetHeaders("Machine", "Route", "Method", "Hits")
t.SetAlignment(table.AlignLeft, table.AlignLeft, table.AlignLeft, table.AlignLeft)
numRows := lapiMetricsToTable(t, stats)
if numRows > 0 {
renderTableTitle(out, "\nLocal Api Machines Metrics:")
t.Render()
}
}
func lapiBouncerStatsTable(out io.Writer, stats map[string]map[string]map[string]int) {
t := newTable(out)
t.SetRowLines(false)
t.SetHeaders("Bouncer", "Route", "Method", "Hits")
t.SetAlignment(table.AlignLeft, table.AlignLeft, table.AlignLeft, table.AlignLeft)
numRows := lapiMetricsToTable(t, stats)
if numRows > 0 {
renderTableTitle(out, "\nLocal Api Bouncers Metrics:")
t.Render()
}
}
func lapiDecisionStatsTable(out io.Writer, stats map[string]struct {
NonEmpty int
Empty int
},
) {
t := newTable(out)
t.SetRowLines(false)
t.SetHeaders("Bouncer", "Empty answers", "Non-empty answers")
t.SetAlignment(table.AlignLeft, table.AlignLeft, table.AlignLeft)
numRows := 0
for bouncer, hits := range stats {
t.AddRow(
bouncer,
fmt.Sprintf("%d", hits.Empty),
fmt.Sprintf("%d", hits.NonEmpty),
)
numRows++
}
if numRows > 0 {
renderTableTitle(out, "\nLocal Api Bouncers Decisions:")
t.Render()
}
}
func decisionStatsTable(out io.Writer, stats map[string]map[string]map[string]int) {
t := newTable(out)
t.SetRowLines(false)
t.SetHeaders("Reason", "Origin", "Action", "Count")
t.SetAlignment(table.AlignLeft, table.AlignLeft, table.AlignLeft, table.AlignLeft)
numRows := 0
for reason, origins := range stats {
for origin, actions := range origins {
for action, hits := range actions {
t.AddRow(
reason,
origin,
action,
fmt.Sprintf("%d", hits),
)
numRows++
}
}
}
if numRows > 0 {
renderTableTitle(out, "\nLocal Api Decisions:")
t.Render()
}
}
func alertStatsTable(out io.Writer, stats map[string]int) {
t := newTable(out)
t.SetRowLines(false)
t.SetHeaders("Reason", "Count")
t.SetAlignment(table.AlignLeft, table.AlignLeft)
numRows := 0
for scenario, hits := range stats {
t.AddRow(
scenario,
fmt.Sprintf("%d", hits),
)
numRows++
}
if numRows > 0 {
renderTableTitle(out, "\nLocal Api Alerts:")
t.Render()
}
}
crowdsec-1.4.6/cmd/crowdsec-cli/notifications.go 0000664 0000000 0000000 00000025221 14371172257 0021667 0 ustar 00root root 0000000 0000000 package main
import (
"context"
"encoding/csv"
"encoding/json"
"fmt"
"io/fs"
"net/url"
"os"
"path/filepath"
"strconv"
"strings"
"time"
"github.com/fatih/color"
"github.com/go-openapi/strfmt"
"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"gopkg.in/tomb.v2"
"github.com/crowdsecurity/crowdsec/pkg/apiclient"
"github.com/crowdsecurity/crowdsec/pkg/csconfig"
"github.com/crowdsecurity/crowdsec/pkg/csplugin"
"github.com/crowdsecurity/crowdsec/pkg/csprofiles"
"github.com/crowdsecurity/crowdsec/pkg/cwversion"
)
type NotificationsCfg struct {
Config csplugin.PluginConfig `json:"plugin_config"`
Profiles []*csconfig.ProfileCfg `json:"associated_profiles"`
ids []uint
}
func NewNotificationsCmd() *cobra.Command {
var cmdNotifications = &cobra.Command{
Use: "notifications [action]",
Short: "Helper for notification plugin configuration",
Long: "To list/inspect/test notification template",
Args: cobra.MinimumNArgs(1),
Aliases: []string{"notifications", "notification"},
DisableAutoGenTag: true,
PersistentPreRun: func(cmd *cobra.Command, args []string) {
var (
err error
)
if err = csConfig.API.Server.LoadProfiles(); err != nil {
log.Fatal(err)
}
if csConfig.ConfigPaths.NotificationDir == "" {
log.Fatalf("config_paths.notification_dir is not set in crowdsec config")
}
},
}
var cmdNotificationsList = &cobra.Command{
Use: "list",
Short: "List active notifications plugins",
Long: `List active notifications plugins`,
Example: `cscli notifications list`,
Args: cobra.ExactArgs(0),
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, arg []string) error {
ncfgs, err := getNotificationsConfiguration()
if err != nil {
return errors.Wrap(err, "Can't build profiles configuration")
}
if csConfig.Cscli.Output == "human" {
notificationListTable(color.Output, ncfgs)
} else if csConfig.Cscli.Output == "json" {
x, err := json.MarshalIndent(ncfgs, "", " ")
if err != nil {
return errors.New("failed to marshal notification configuration")
}
fmt.Printf("%s", string(x))
} else if csConfig.Cscli.Output == "raw" {
csvwriter := csv.NewWriter(os.Stdout)
err := csvwriter.Write([]string{"Name", "Type", "Profile name"})
if err != nil {
return errors.Wrap(err, "failed to write raw header")
}
for _, b := range ncfgs {
profilesList := []string{}
for _, p := range b.Profiles {
profilesList = append(profilesList, p.Name)
}
err := csvwriter.Write([]string{b.Config.Name, b.Config.Type, strings.Join(profilesList, ", ")})
if err != nil {
return errors.Wrap(err, "failed to write raw content")
}
}
csvwriter.Flush()
}
return nil
},
}
cmdNotifications.AddCommand(cmdNotificationsList)
var cmdNotificationsInspect = &cobra.Command{
Use: "inspect",
Short: "Inspect active notifications plugin configuration",
Long: `Inspect active notifications plugin and show configuration`,
Example: `cscli notifications inspect `,
Args: cobra.ExactArgs(1),
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, arg []string) error {
var (
cfg NotificationsCfg
ok bool
)
pluginName := arg[0]
if pluginName == "" {
errors.New("Please provide a plugin name to inspect")
}
ncfgs, err := getNotificationsConfiguration()
if err != nil {
return errors.Wrap(err, "Can't build profiles configuration")
}
if cfg, ok = ncfgs[pluginName]; !ok {
return errors.New("The provided plugin name doesn't exist or isn't active")
}
if csConfig.Cscli.Output == "human" || csConfig.Cscli.Output == "raw" {
fmt.Printf(" - %15s: %15s\n", "Type", cfg.Config.Type)
fmt.Printf(" - %15s: %15s\n", "Name", cfg.Config.Name)
fmt.Printf(" - %15s: %15s\n", "Timeout", cfg.Config.TimeOut)
fmt.Printf(" - %15s: %15s\n", "Format", cfg.Config.Format)
for k, v := range cfg.Config.Config {
fmt.Printf(" - %15s: %15v\n", k, v)
}
} else if csConfig.Cscli.Output == "json" {
x, err := json.MarshalIndent(cfg, "", " ")
if err != nil {
return errors.New("failed to marshal notification configuration")
}
fmt.Printf("%s", string(x))
}
return nil
},
}
cmdNotifications.AddCommand(cmdNotificationsInspect)
var remediation bool
var alertOverride string
var cmdNotificationsReinject = &cobra.Command{
Use: "reinject",
Short: "reinject alert into notifications system",
Long: `Reinject alert into notifications system`,
Example: `
cscli notifications reinject
cscli notifications reinject --remediation
cscli notifications reinject -a '{"remediation": true,"scenario":"notification/test"}'
`,
Args: cobra.ExactArgs(1),
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, args []string) error {
var (
pluginBroker csplugin.PluginBroker
pluginTomb tomb.Tomb
)
if len(args) != 1 {
printHelp(cmd)
return errors.New("Wrong number of argument: there should be one argument")
}
//first: get the alert
id, err := strconv.Atoi(args[0])
if err != nil {
return errors.New(fmt.Sprintf("bad alert id %s", args[0]))
}
if err := csConfig.LoadAPIClient(); err != nil {
return errors.Wrapf(err, "loading api client")
}
if csConfig.API.Client == nil {
return errors.New("There is no configuration on 'api_client:'")
}
if csConfig.API.Client.Credentials == nil {
return errors.New(fmt.Sprintf("Please provide credentials for the API in '%s'", csConfig.API.Client.CredentialsFilePath))
}
apiURL, err := url.Parse(csConfig.API.Client.Credentials.URL)
if err != nil {
return errors.Wrapf(err, "error parsing the URL of the API")
}
client, err := apiclient.NewClient(&apiclient.Config{
MachineID: csConfig.API.Client.Credentials.Login,
Password: strfmt.Password(csConfig.API.Client.Credentials.Password),
UserAgent: fmt.Sprintf("crowdsec/%s", cwversion.VersionStr()),
URL: apiURL,
VersionPrefix: "v1",
})
if err != nil {
return errors.Wrapf(err, "error creating the client for the API")
}
alert, _, err := client.Alerts.GetByID(context.Background(), id)
if err != nil {
return errors.Wrapf(err, fmt.Sprintf("can't find alert with id %s", args[0]))
}
if alertOverride != "" {
if err = json.Unmarshal([]byte(alertOverride), alert); err != nil {
return errors.Wrapf(err, "Can't unmarshal the data given in the alert flag")
}
}
if !remediation {
alert.Remediation = true
}
// second we start plugins
err = pluginBroker.Init(csConfig.PluginConfig, csConfig.API.Server.Profiles, csConfig.ConfigPaths)
if err != nil {
return errors.Wrapf(err, "Can't initialize plugins")
}
pluginTomb.Go(func() error {
pluginBroker.Run(&pluginTomb)
return nil
})
//third: get the profile(s), and process the whole stuff
profiles, err := csprofiles.NewProfile(csConfig.API.Server.Profiles)
if err != nil {
return errors.Wrap(err, "Cannot extract profiles from configuration")
}
for id, profile := range profiles {
_, matched, err := profile.EvaluateProfile(alert)
if err != nil {
return errors.Wrapf(err, "can't evaluate profile %s", profile.Cfg.Name)
}
if !matched {
log.Infof("The profile %s didn't match", profile.Cfg.Name)
continue
}
log.Infof("The profile %s matched, sending to its configured notification plugins", profile.Cfg.Name)
loop:
for {
select {
case pluginBroker.PluginChannel <- csplugin.ProfileAlert{
ProfileID: uint(id),
Alert: alert,
}:
break loop
default:
time.Sleep(50 * time.Millisecond)
log.Info("sleeping\n")
}
}
if profile.Cfg.OnSuccess == "break" {
log.Infof("The profile %s contains a 'on_success: break' so bailing out", profile.Cfg.Name)
break
}
}
// time.Sleep(2 * time.Second) // There's no mechanism to ensure notification has been sent
pluginTomb.Kill(errors.New("terminating"))
pluginTomb.Wait()
return nil
},
}
cmdNotificationsReinject.Flags().BoolVarP(&remediation, "remediation", "r", false, "Set Alert.Remediation to false in the reinjected alert (see your profile filter configuration)")
cmdNotificationsReinject.Flags().StringVarP(&alertOverride, "alert", "a", "", "JSON string used to override alert fields in the reinjected alert (see crowdsec/pkg/models/alert.go in the source tree for the full definition of the object)")
cmdNotifications.AddCommand(cmdNotificationsReinject)
return cmdNotifications
}
func getNotificationsConfiguration() (map[string]NotificationsCfg, error) {
pcfgs := map[string]csplugin.PluginConfig{}
wf := func(path string, info fs.FileInfo, err error) error {
if info == nil {
return errors.Wrapf(err, "error while traversing directory %s", path)
}
name := filepath.Join(csConfig.ConfigPaths.NotificationDir, info.Name()) //Avoid calling info.Name() twice
if (strings.HasSuffix(name, "yaml") || strings.HasSuffix(name, "yml")) && !(info.IsDir()) {
ts, err := csplugin.ParsePluginConfigFile(name)
if err != nil {
return errors.Wrapf(err, "Loading notifification plugin configuration with %s", name)
}
for _, t := range ts {
pcfgs[t.Name] = t
}
}
return nil
}
if err := filepath.Walk(csConfig.ConfigPaths.NotificationDir, wf); err != nil {
return nil, errors.Wrap(err, "Loading notifification plugin configuration")
}
// A bit of a tricky stuf now: reconcile profiles and notification plugins
ncfgs := map[string]NotificationsCfg{}
profiles, err := csprofiles.NewProfile(csConfig.API.Server.Profiles)
if err != nil {
return nil, errors.Wrap(err, "Cannot extract profiles from configuration")
}
for profileID, profile := range profiles {
loop:
for _, notif := range profile.Cfg.Notifications {
for name, pc := range pcfgs {
if notif == name {
if _, ok := ncfgs[pc.Name]; !ok {
ncfgs[pc.Name] = NotificationsCfg{
Config: pc,
Profiles: []*csconfig.ProfileCfg{profile.Cfg},
ids: []uint{uint(profileID)},
}
continue loop
}
tmp := ncfgs[pc.Name]
for _, pr := range tmp.Profiles {
var profiles []*csconfig.ProfileCfg
if pr.Name == profile.Cfg.Name {
continue
}
profiles = append(tmp.Profiles, profile.Cfg)
ids := append(tmp.ids, uint(profileID))
ncfgs[pc.Name] = NotificationsCfg{
Config: tmp.Config,
Profiles: profiles,
ids: ids,
}
}
}
}
}
}
return ncfgs, nil
}
crowdsec-1.4.6/cmd/crowdsec-cli/notifications_table.go 0000664 0000000 0000000 00000001131 14371172257 0023030 0 ustar 00root root 0000000 0000000 package main
import (
"io"
"strings"
"github.com/aquasecurity/table"
)
func notificationListTable(out io.Writer, ncfgs map[string]NotificationsCfg) {
t := newLightTable(out)
t.SetHeaders("Name", "Type", "Profile name")
t.SetHeaderAlignment(table.AlignLeft, table.AlignLeft, table.AlignLeft)
t.SetAlignment(table.AlignLeft, table.AlignLeft, table.AlignLeft)
for _, b := range ncfgs {
profilesList := []string{}
for _, p := range b.Profiles {
profilesList = append(profilesList, p.Name)
}
t.AddRow(b.Config.Name, b.Config.Type, strings.Join(profilesList, ", "))
}
t.Render()
}
crowdsec-1.4.6/cmd/crowdsec-cli/parsers.go 0000664 0000000 0000000 00000014620 14371172257 0020476 0 ustar 00root root 0000000 0000000 package main
import (
"fmt"
"github.com/fatih/color"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"github.com/crowdsecurity/crowdsec/pkg/cwhub"
)
func NewParsersCmd() *cobra.Command {
var cmdParsers = &cobra.Command{
Use: "parsers [action] [config]",
Short: "Install/Remove/Upgrade/Inspect parser(s) from hub",
Example: `cscli parsers install crowdsecurity/sshd-logs
cscli parsers inspect crowdsecurity/sshd-logs
cscli parsers upgrade crowdsecurity/sshd-logs
cscli parsers list
cscli parsers remove crowdsecurity/sshd-logs
`,
Args: cobra.MinimumNArgs(1),
Aliases: []string{"parser"},
DisableAutoGenTag: true,
PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
if err := csConfig.LoadHub(); err != nil {
log.Fatal(err)
}
if csConfig.Hub == nil {
return fmt.Errorf("you must configure cli before interacting with hub")
}
if err := cwhub.SetHubBranch(); err != nil {
return fmt.Errorf("error while setting hub branch: %s", err)
}
if err := cwhub.GetHubIdx(csConfig.Hub); err != nil {
log.Info("Run 'sudo cscli hub update' to get the hub index")
log.Fatalf("Failed to get Hub index : %v", err)
}
return nil
},
PersistentPostRun: func(cmd *cobra.Command, args []string) {
if cmd.Name() == "inspect" || cmd.Name() == "list" {
return
}
log.Infof(ReloadMessage())
},
}
var ignoreError bool
var cmdParsersInstall = &cobra.Command{
Use: "install [config]",
Short: "Install given parser(s)",
Long: `Fetch and install given parser(s) from hub`,
Example: `cscli parsers install crowdsec/xxx crowdsec/xyz`,
Args: cobra.MinimumNArgs(1),
DisableAutoGenTag: true,
ValidArgsFunction: func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
return compAllItems(cwhub.PARSERS, args, toComplete)
},
Run: func(cmd *cobra.Command, args []string) {
for _, name := range args {
t := cwhub.GetItem(cwhub.PARSERS, name)
if t == nil {
nearestItem, score := GetDistance(cwhub.PARSERS, name)
Suggest(cwhub.PARSERS, name, nearestItem.Name, score, ignoreError)
continue
}
if err := cwhub.InstallItem(csConfig, name, cwhub.PARSERS, forceAction, downloadOnly); err != nil {
if ignoreError {
log.Errorf("Error while installing '%s': %s", name, err)
} else {
log.Fatalf("Error while installing '%s': %s", name, err)
}
}
}
},
}
cmdParsersInstall.PersistentFlags().BoolVarP(&downloadOnly, "download-only", "d", false, "Only download packages, don't enable")
cmdParsersInstall.PersistentFlags().BoolVar(&forceAction, "force", false, "Force install : Overwrite tainted and outdated files")
cmdParsersInstall.PersistentFlags().BoolVar(&ignoreError, "ignore", false, "Ignore errors when installing multiple parsers")
cmdParsers.AddCommand(cmdParsersInstall)
var cmdParsersRemove = &cobra.Command{
Use: "remove [config]",
Short: "Remove given parser(s)",
Long: `Remove given parse(s) from hub`,
Aliases: []string{"delete"},
Example: `cscli parsers remove crowdsec/xxx crowdsec/xyz`,
DisableAutoGenTag: true,
ValidArgsFunction: func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
return compInstalledItems(cwhub.PARSERS, args, toComplete)
},
Run: func(cmd *cobra.Command, args []string) {
if all {
cwhub.RemoveMany(csConfig, cwhub.PARSERS, "", all, purge, forceAction)
return
}
if len(args) == 0 {
log.Fatalf("Specify at least one parser to remove or '--all' flag.")
}
for _, name := range args {
cwhub.RemoveMany(csConfig, cwhub.PARSERS, name, all, purge, forceAction)
}
},
}
cmdParsersRemove.PersistentFlags().BoolVar(&purge, "purge", false, "Delete source file too")
cmdParsersRemove.PersistentFlags().BoolVar(&forceAction, "force", false, "Force remove : Remove tainted and outdated files")
cmdParsersRemove.PersistentFlags().BoolVar(&all, "all", false, "Delete all the parsers")
cmdParsers.AddCommand(cmdParsersRemove)
var cmdParsersUpgrade = &cobra.Command{
Use: "upgrade [config]",
Short: "Upgrade given parser(s)",
Long: `Fetch and upgrade given parser(s) from hub`,
Example: `cscli parsers upgrade crowdsec/xxx crowdsec/xyz`,
DisableAutoGenTag: true,
ValidArgsFunction: func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
return compInstalledItems(cwhub.PARSERS, args, toComplete)
},
Run: func(cmd *cobra.Command, args []string) {
if all {
cwhub.UpgradeConfig(csConfig, cwhub.PARSERS, "", forceAction)
} else {
if len(args) == 0 {
log.Fatalf("no target parser to upgrade")
}
for _, name := range args {
cwhub.UpgradeConfig(csConfig, cwhub.PARSERS, name, forceAction)
}
}
},
}
cmdParsersUpgrade.PersistentFlags().BoolVar(&all, "all", false, "Upgrade all the parsers")
cmdParsersUpgrade.PersistentFlags().BoolVar(&forceAction, "force", false, "Force upgrade : Overwrite tainted and outdated files")
cmdParsers.AddCommand(cmdParsersUpgrade)
var cmdParsersInspect = &cobra.Command{
Use: "inspect [name]",
Short: "Inspect given parser",
Long: `Inspect given parser`,
Example: `cscli parsers inspect crowdsec/xxx`,
DisableAutoGenTag: true,
Args: cobra.MinimumNArgs(1),
ValidArgsFunction: func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
return compInstalledItems(cwhub.PARSERS, args, toComplete)
},
Run: func(cmd *cobra.Command, args []string) {
InspectItem(args[0], cwhub.PARSERS)
},
}
cmdParsersInspect.PersistentFlags().StringVarP(&prometheusURL, "url", "u", "", "Prometheus url")
cmdParsers.AddCommand(cmdParsersInspect)
var cmdParsersList = &cobra.Command{
Use: "list [name]",
Short: "List all parsers or given one",
Long: `List all parsers or given one`,
Example: `cscli parsers list
cscli parser list crowdsecurity/xxx`,
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
ListItems(color.Output, []string{cwhub.PARSERS}, args, false, true, all)
},
}
cmdParsersList.PersistentFlags().BoolVarP(&all, "all", "a", false, "List disabled items as well")
cmdParsers.AddCommand(cmdParsersList)
return cmdParsers
}
crowdsec-1.4.6/cmd/crowdsec-cli/postoverflows.go 0000664 0000000 0000000 00000015354 14371172257 0021760 0 ustar 00root root 0000000 0000000 package main
import (
"fmt"
"github.com/fatih/color"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"github.com/crowdsecurity/crowdsec/pkg/cwhub"
)
func NewPostOverflowsCmd() *cobra.Command {
var cmdPostOverflows = &cobra.Command{
Use: "postoverflows [action] [config]",
Short: "Install/Remove/Upgrade/Inspect postoverflow(s) from hub",
Example: `cscli postoverflows install crowdsecurity/cdn-whitelist
cscli postoverflows inspect crowdsecurity/cdn-whitelist
cscli postoverflows upgrade crowdsecurity/cdn-whitelist
cscli postoverflows list
cscli postoverflows remove crowdsecurity/cdn-whitelist`,
Args: cobra.MinimumNArgs(1),
Aliases: []string{"postoverflow"},
DisableAutoGenTag: true,
PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
if err := csConfig.LoadHub(); err != nil {
log.Fatal(err)
}
if csConfig.Hub == nil {
return fmt.Errorf("you must configure cli before interacting with hub")
}
if err := cwhub.SetHubBranch(); err != nil {
return fmt.Errorf("error while setting hub branch: %s", err)
}
if err := cwhub.GetHubIdx(csConfig.Hub); err != nil {
log.Info("Run 'sudo cscli hub update' to get the hub index")
log.Fatalf("Failed to get Hub index : %v", err)
}
return nil
},
PersistentPostRun: func(cmd *cobra.Command, args []string) {
if cmd.Name() == "inspect" || cmd.Name() == "list" {
return
}
log.Infof(ReloadMessage())
},
}
var ignoreError bool
var cmdPostOverflowsInstall = &cobra.Command{
Use: "install [config]",
Short: "Install given postoverflow(s)",
Long: `Fetch and install given postoverflow(s) from hub`,
Example: `cscli postoverflows install crowdsec/xxx crowdsec/xyz`,
Args: cobra.MinimumNArgs(1),
DisableAutoGenTag: true,
ValidArgsFunction: func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
return compAllItems(cwhub.PARSERS_OVFLW, args, toComplete)
},
Run: func(cmd *cobra.Command, args []string) {
for _, name := range args {
t := cwhub.GetItem(cwhub.PARSERS_OVFLW, name)
if t == nil {
nearestItem, score := GetDistance(cwhub.PARSERS_OVFLW, name)
Suggest(cwhub.PARSERS_OVFLW, name, nearestItem.Name, score, ignoreError)
continue
}
if err := cwhub.InstallItem(csConfig, name, cwhub.PARSERS_OVFLW, forceAction, downloadOnly); err != nil {
if ignoreError {
log.Errorf("Error while installing '%s': %s", name, err)
} else {
log.Fatalf("Error while installing '%s': %s", name, err)
}
}
}
},
}
cmdPostOverflowsInstall.PersistentFlags().BoolVarP(&downloadOnly, "download-only", "d", false, "Only download packages, don't enable")
cmdPostOverflowsInstall.PersistentFlags().BoolVar(&forceAction, "force", false, "Force install : Overwrite tainted and outdated files")
cmdPostOverflowsInstall.PersistentFlags().BoolVar(&ignoreError, "ignore", false, "Ignore errors when installing multiple postoverflows")
cmdPostOverflows.AddCommand(cmdPostOverflowsInstall)
var cmdPostOverflowsRemove = &cobra.Command{
Use: "remove [config]",
Short: "Remove given postoverflow(s)",
Long: `remove given postoverflow(s)`,
Example: `cscli postoverflows remove crowdsec/xxx crowdsec/xyz`,
DisableAutoGenTag: true,
Aliases: []string{"delete"},
ValidArgsFunction: func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
return compInstalledItems(cwhub.PARSERS_OVFLW, args, toComplete)
},
Run: func(cmd *cobra.Command, args []string) {
if all {
cwhub.RemoveMany(csConfig, cwhub.PARSERS_OVFLW, "", all, purge, forceAction)
return
}
if len(args) == 0 {
log.Fatalf("Specify at least one postoverflow to remove or '--all' flag.")
}
for _, name := range args {
cwhub.RemoveMany(csConfig, cwhub.PARSERS_OVFLW, name, all, purge, forceAction)
}
},
}
cmdPostOverflowsRemove.PersistentFlags().BoolVar(&purge, "purge", false, "Delete source file too")
cmdPostOverflowsRemove.PersistentFlags().BoolVar(&forceAction, "force", false, "Force remove : Remove tainted and outdated files")
cmdPostOverflowsRemove.PersistentFlags().BoolVar(&all, "all", false, "Delete all the postoverflows")
cmdPostOverflows.AddCommand(cmdPostOverflowsRemove)
var cmdPostOverflowsUpgrade = &cobra.Command{
Use: "upgrade [config]",
Short: "Upgrade given postoverflow(s)",
Long: `Fetch and Upgrade given postoverflow(s) from hub`,
Example: `cscli postoverflows upgrade crowdsec/xxx crowdsec/xyz`,
DisableAutoGenTag: true,
ValidArgsFunction: func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
return compInstalledItems(cwhub.PARSERS_OVFLW, args, toComplete)
},
Run: func(cmd *cobra.Command, args []string) {
if all {
cwhub.UpgradeConfig(csConfig, cwhub.PARSERS_OVFLW, "", forceAction)
} else {
if len(args) == 0 {
log.Fatalf("no target postoverflow to upgrade")
}
for _, name := range args {
cwhub.UpgradeConfig(csConfig, cwhub.PARSERS_OVFLW, name, forceAction)
}
}
},
}
cmdPostOverflowsUpgrade.PersistentFlags().BoolVarP(&all, "all", "a", false, "Upgrade all the postoverflows")
cmdPostOverflowsUpgrade.PersistentFlags().BoolVar(&forceAction, "force", false, "Force upgrade : Overwrite tainted and outdated files")
cmdPostOverflows.AddCommand(cmdPostOverflowsUpgrade)
var cmdPostOverflowsInspect = &cobra.Command{
Use: "inspect [config]",
Short: "Inspect given postoverflow",
Long: `Inspect given postoverflow`,
Example: `cscli postoverflows inspect crowdsec/xxx crowdsec/xyz`,
DisableAutoGenTag: true,
ValidArgsFunction: func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
return compInstalledItems(cwhub.PARSERS_OVFLW, args, toComplete)
},
Args: cobra.MinimumNArgs(1),
Run: func(cmd *cobra.Command, args []string) {
InspectItem(args[0], cwhub.PARSERS_OVFLW)
},
}
cmdPostOverflows.AddCommand(cmdPostOverflowsInspect)
var cmdPostOverflowsList = &cobra.Command{
Use: "list [config]",
Short: "List all postoverflows or given one",
Long: `List all postoverflows or given one`,
Example: `cscli postoverflows list
cscli postoverflows list crowdsecurity/xxx`,
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
ListItems(color.Output, []string{cwhub.PARSERS_OVFLW}, args, false, true, all)
},
}
cmdPostOverflowsList.PersistentFlags().BoolVarP(&all, "all", "a", false, "List disabled items as well")
cmdPostOverflows.AddCommand(cmdPostOverflowsList)
return cmdPostOverflows
}
crowdsec-1.4.6/cmd/crowdsec-cli/scenarios.go 0000664 0000000 0000000 00000014562 14371172257 0021012 0 ustar 00root root 0000000 0000000 package main
import (
"fmt"
"github.com/fatih/color"
"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"github.com/crowdsecurity/crowdsec/pkg/cwhub"
)
func NewScenariosCmd() *cobra.Command {
var cmdScenarios = &cobra.Command{
Use: "scenarios [action] [config]",
Short: "Install/Remove/Upgrade/Inspect scenario(s) from hub",
Example: `cscli scenarios list [-a]
cscli scenarios install crowdsecurity/ssh-bf
cscli scenarios inspect crowdsecurity/ssh-bf
cscli scenarios upgrade crowdsecurity/ssh-bf
cscli scenarios remove crowdsecurity/ssh-bf
`,
Args: cobra.MinimumNArgs(1),
Aliases: []string{"scenario"},
DisableAutoGenTag: true,
PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
if err := csConfig.LoadHub(); err != nil {
log.Fatal(err)
}
if csConfig.Hub == nil {
return fmt.Errorf("you must configure cli before interacting with hub")
}
if err := cwhub.SetHubBranch(); err != nil {
return errors.Wrap(err, "while setting hub branch")
}
if err := cwhub.GetHubIdx(csConfig.Hub); err != nil {
log.Info("Run 'sudo cscli hub update' to get the hub index")
log.Fatalf("Failed to get Hub index : %v", err)
}
return nil
},
PersistentPostRun: func(cmd *cobra.Command, args []string) {
if cmd.Name() == "inspect" || cmd.Name() == "list" {
return
}
log.Infof(ReloadMessage())
},
}
var ignoreError bool
var cmdScenariosInstall = &cobra.Command{
Use: "install [config]",
Short: "Install given scenario(s)",
Long: `Fetch and install given scenario(s) from hub`,
Example: `cscli scenarios install crowdsec/xxx crowdsec/xyz`,
Args: cobra.MinimumNArgs(1),
ValidArgsFunction: func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
return compAllItems(cwhub.SCENARIOS, args, toComplete)
},
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
for _, name := range args {
t := cwhub.GetItem(cwhub.SCENARIOS, name)
if t == nil {
nearestItem, score := GetDistance(cwhub.SCENARIOS, name)
Suggest(cwhub.SCENARIOS, name, nearestItem.Name, score, ignoreError)
continue
}
if err := cwhub.InstallItem(csConfig, name, cwhub.SCENARIOS, forceAction, downloadOnly); err != nil {
if ignoreError {
log.Errorf("Error while installing '%s': %s", name, err)
} else {
log.Fatalf("Error while installing '%s': %s", name, err)
}
}
}
},
}
cmdScenariosInstall.PersistentFlags().BoolVarP(&downloadOnly, "download-only", "d", false, "Only download packages, don't enable")
cmdScenariosInstall.PersistentFlags().BoolVar(&forceAction, "force", false, "Force install : Overwrite tainted and outdated files")
cmdScenariosInstall.PersistentFlags().BoolVar(&ignoreError, "ignore", false, "Ignore errors when installing multiple scenarios")
cmdScenarios.AddCommand(cmdScenariosInstall)
var cmdScenariosRemove = &cobra.Command{
Use: "remove [config]",
Short: "Remove given scenario(s)",
Long: `remove given scenario(s)`,
Example: `cscli scenarios remove crowdsec/xxx crowdsec/xyz`,
Aliases: []string{"delete"},
ValidArgsFunction: func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
return compInstalledItems(cwhub.SCENARIOS, args, toComplete)
},
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
if all {
cwhub.RemoveMany(csConfig, cwhub.SCENARIOS, "", all, purge, forceAction)
return
}
if len(args) == 0 {
log.Fatalf("Specify at least one scenario to remove or '--all' flag.")
}
for _, name := range args {
cwhub.RemoveMany(csConfig, cwhub.SCENARIOS, name, all, purge, forceAction)
}
},
}
cmdScenariosRemove.PersistentFlags().BoolVar(&purge, "purge", false, "Delete source file too")
cmdScenariosRemove.PersistentFlags().BoolVar(&forceAction, "force", false, "Force remove : Remove tainted and outdated files")
cmdScenariosRemove.PersistentFlags().BoolVar(&all, "all", false, "Delete all the scenarios")
cmdScenarios.AddCommand(cmdScenariosRemove)
var cmdScenariosUpgrade = &cobra.Command{
Use: "upgrade [config]",
Short: "Upgrade given scenario(s)",
Long: `Fetch and Upgrade given scenario(s) from hub`,
Example: `cscli scenarios upgrade crowdsec/xxx crowdsec/xyz`,
ValidArgsFunction: func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
return compInstalledItems(cwhub.SCENARIOS, args, toComplete)
},
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
if all {
cwhub.UpgradeConfig(csConfig, cwhub.SCENARIOS, "", forceAction)
} else {
if len(args) == 0 {
log.Fatalf("no target scenario to upgrade")
}
for _, name := range args {
cwhub.UpgradeConfig(csConfig, cwhub.SCENARIOS, name, forceAction)
}
}
},
}
cmdScenariosUpgrade.PersistentFlags().BoolVarP(&all, "all", "a", false, "Upgrade all the scenarios")
cmdScenariosUpgrade.PersistentFlags().BoolVar(&forceAction, "force", false, "Force upgrade : Overwrite tainted and outdated files")
cmdScenarios.AddCommand(cmdScenariosUpgrade)
var cmdScenariosInspect = &cobra.Command{
Use: "inspect [config]",
Short: "Inspect given scenario",
Long: `Inspect given scenario`,
Example: `cscli scenarios inspect crowdsec/xxx`,
Args: cobra.MinimumNArgs(1),
ValidArgsFunction: func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
return compInstalledItems(cwhub.SCENARIOS, args, toComplete)
},
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
InspectItem(args[0], cwhub.SCENARIOS)
},
}
cmdScenariosInspect.PersistentFlags().StringVarP(&prometheusURL, "url", "u", "", "Prometheus url")
cmdScenarios.AddCommand(cmdScenariosInspect)
var cmdScenariosList = &cobra.Command{
Use: "list [config]",
Short: "List all scenario(s) or given one",
Long: `List all scenario(s) or given one`,
Example: `cscli scenarios list
cscli scenarios list crowdsecurity/xxx`,
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
ListItems(color.Output, []string{cwhub.SCENARIOS}, args, false, true, all)
},
}
cmdScenariosList.PersistentFlags().BoolVarP(&all, "all", "a", false, "List disabled items as well")
cmdScenarios.AddCommand(cmdScenariosList)
return cmdScenarios
}
crowdsec-1.4.6/cmd/crowdsec-cli/simulation.go 0000664 0000000 0000000 00000020247 14371172257 0021205 0 ustar 00root root 0000000 0000000 package main
import (
"fmt"
"os"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"gopkg.in/yaml.v2"
"github.com/crowdsecurity/crowdsec/pkg/cwhub"
)
func addToExclusion(name string) error {
csConfig.Cscli.SimulationConfig.Exclusions = append(csConfig.Cscli.SimulationConfig.Exclusions, name)
return nil
}
func removeFromExclusion(name string) error {
index := indexOf(name, csConfig.Cscli.SimulationConfig.Exclusions)
// Remove element from the slice
csConfig.Cscli.SimulationConfig.Exclusions[index] = csConfig.Cscli.SimulationConfig.Exclusions[len(csConfig.Cscli.SimulationConfig.Exclusions)-1]
csConfig.Cscli.SimulationConfig.Exclusions[len(csConfig.Cscli.SimulationConfig.Exclusions)-1] = ""
csConfig.Cscli.SimulationConfig.Exclusions = csConfig.Cscli.SimulationConfig.Exclusions[:len(csConfig.Cscli.SimulationConfig.Exclusions)-1]
return nil
}
func enableGlobalSimulation() error {
csConfig.Cscli.SimulationConfig.Simulation = new(bool)
*csConfig.Cscli.SimulationConfig.Simulation = true
csConfig.Cscli.SimulationConfig.Exclusions = []string{}
if err := dumpSimulationFile(); err != nil {
log.Fatalf("unable to dump simulation file: %s", err)
}
log.Printf("global simulation: enabled")
return nil
}
func dumpSimulationFile() error {
newConfigSim, err := yaml.Marshal(csConfig.Cscli.SimulationConfig)
if err != nil {
return fmt.Errorf("unable to marshal simulation configuration: %s", err)
}
err = os.WriteFile(csConfig.ConfigPaths.SimulationFilePath, newConfigSim, 0644)
if err != nil {
return fmt.Errorf("write simulation config in '%s' failed: %s", csConfig.ConfigPaths.SimulationFilePath, err)
}
log.Debugf("updated simulation file %s", csConfig.ConfigPaths.SimulationFilePath)
return nil
}
func disableGlobalSimulation() error {
csConfig.Cscli.SimulationConfig.Simulation = new(bool)
*csConfig.Cscli.SimulationConfig.Simulation = false
csConfig.Cscli.SimulationConfig.Exclusions = []string{}
newConfigSim, err := yaml.Marshal(csConfig.Cscli.SimulationConfig)
if err != nil {
return fmt.Errorf("unable to marshal new simulation configuration: %s", err)
}
err = os.WriteFile(csConfig.ConfigPaths.SimulationFilePath, newConfigSim, 0644)
if err != nil {
return fmt.Errorf("unable to write new simulation config in '%s' : %s", csConfig.ConfigPaths.SimulationFilePath, err)
}
log.Printf("global simulation: disabled")
return nil
}
func simulationStatus() error {
if csConfig.Cscli.SimulationConfig == nil {
log.Printf("global simulation: disabled (configuration file is missing)")
return nil
}
if *csConfig.Cscli.SimulationConfig.Simulation {
log.Println("global simulation: enabled")
if len(csConfig.Cscli.SimulationConfig.Exclusions) > 0 {
log.Println("Scenarios not in simulation mode :")
for _, scenario := range csConfig.Cscli.SimulationConfig.Exclusions {
log.Printf(" - %s", scenario)
}
}
} else {
log.Println("global simulation: disabled")
if len(csConfig.Cscli.SimulationConfig.Exclusions) > 0 {
log.Println("Scenarios in simulation mode :")
for _, scenario := range csConfig.Cscli.SimulationConfig.Exclusions {
log.Printf(" - %s", scenario)
}
}
}
return nil
}
func NewSimulationCmds() *cobra.Command {
var cmdSimulation = &cobra.Command{
Use: "simulation [command]",
Short: "Manage simulation status of scenarios",
Example: `cscli simulation status
cscli simulation enable crowdsecurity/ssh-bf
cscli simulation disable crowdsecurity/ssh-bf`,
DisableAutoGenTag: true,
PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
if err := csConfig.LoadSimulation(); err != nil {
log.Fatal(err)
}
if csConfig.Cscli == nil {
return fmt.Errorf("you must configure cli before using simulation")
}
if csConfig.Cscli.SimulationConfig == nil {
return fmt.Errorf("no simulation configured")
}
return nil
},
PersistentPostRun: func(cmd *cobra.Command, args []string) {
if cmd.Name() != "status" {
log.Infof(ReloadMessage())
}
},
}
cmdSimulation.Flags().SortFlags = false
cmdSimulation.PersistentFlags().SortFlags = false
var forceGlobalSimulation bool
var cmdSimulationEnable = &cobra.Command{
Use: "enable [scenario] [-global]",
Short: "Enable the simulation, globally or on specified scenarios",
Example: `cscli simulation enable`,
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
if err := csConfig.LoadHub(); err != nil {
log.Fatal(err)
}
if err := cwhub.GetHubIdx(csConfig.Hub); err != nil {
log.Info("Run 'sudo cscli hub update' to get the hub index")
log.Fatalf("Failed to get Hub index : %v", err)
}
if len(args) > 0 {
for _, scenario := range args {
var item = cwhub.GetItem(cwhub.SCENARIOS, scenario)
if item == nil {
log.Errorf("'%s' doesn't exist or is not a scenario", scenario)
continue
}
if !item.Installed {
log.Warningf("'%s' isn't enabled", scenario)
}
isExcluded := inSlice(scenario, csConfig.Cscli.SimulationConfig.Exclusions)
if *csConfig.Cscli.SimulationConfig.Simulation && !isExcluded {
log.Warning("global simulation is already enabled")
continue
}
if !*csConfig.Cscli.SimulationConfig.Simulation && isExcluded {
log.Warningf("simulation for '%s' already enabled", scenario)
continue
}
if *csConfig.Cscli.SimulationConfig.Simulation && isExcluded {
if err := removeFromExclusion(scenario); err != nil {
log.Fatal(err)
}
log.Printf("simulation enabled for '%s'", scenario)
continue
}
if err := addToExclusion(scenario); err != nil {
log.Fatal(err)
}
log.Printf("simulation mode for '%s' enabled", scenario)
}
if err := dumpSimulationFile(); err != nil {
log.Fatalf("simulation enable: %s", err)
}
} else if forceGlobalSimulation {
if err := enableGlobalSimulation(); err != nil {
log.Fatalf("unable to enable global simulation mode : %s", err)
}
} else {
printHelp(cmd)
}
},
}
cmdSimulationEnable.Flags().BoolVarP(&forceGlobalSimulation, "global", "g", false, "Enable global simulation (reverse mode)")
cmdSimulation.AddCommand(cmdSimulationEnable)
var cmdSimulationDisable = &cobra.Command{
Use: "disable [scenario]",
Short: "Disable the simulation mode. Disable only specified scenarios",
Example: `cscli simulation disable`,
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
if len(args) > 0 {
for _, scenario := range args {
isExcluded := inSlice(scenario, csConfig.Cscli.SimulationConfig.Exclusions)
if !*csConfig.Cscli.SimulationConfig.Simulation && !isExcluded {
log.Warningf("%s isn't in simulation mode", scenario)
continue
}
if !*csConfig.Cscli.SimulationConfig.Simulation && isExcluded {
if err := removeFromExclusion(scenario); err != nil {
log.Fatal(err)
}
log.Printf("simulation mode for '%s' disabled", scenario)
continue
}
if isExcluded {
log.Warningf("simulation mode is enabled but is already disable for '%s'", scenario)
continue
}
if err := addToExclusion(scenario); err != nil {
log.Fatal(err)
}
log.Printf("simulation mode for '%s' disabled", scenario)
}
if err := dumpSimulationFile(); err != nil {
log.Fatalf("simulation disable: %s", err)
}
} else if forceGlobalSimulation {
if err := disableGlobalSimulation(); err != nil {
log.Fatalf("unable to disable global simulation mode : %s", err)
}
} else {
printHelp(cmd)
}
},
}
cmdSimulationDisable.Flags().BoolVarP(&forceGlobalSimulation, "global", "g", false, "Disable global simulation (reverse mode)")
cmdSimulation.AddCommand(cmdSimulationDisable)
var cmdSimulationStatus = &cobra.Command{
Use: "status",
Short: "Show simulation mode status",
Example: `cscli simulation status`,
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
if err := simulationStatus(); err != nil {
log.Fatal(err)
}
},
PersistentPostRun: func(cmd *cobra.Command, args []string) {
},
}
cmdSimulation.AddCommand(cmdSimulationStatus)
return cmdSimulation
}
crowdsec-1.4.6/cmd/crowdsec-cli/support.go 0000664 0000000 0000000 00000027655 14371172257 0020547 0 ustar 00root root 0000000 0000000 package main
import (
"archive/zip"
"bytes"
"context"
"fmt"
"io"
"net/http"
"net/url"
"os"
"path/filepath"
"regexp"
"strings"
"github.com/blackfireio/osinfo"
"github.com/go-openapi/strfmt"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"github.com/crowdsecurity/crowdsec/pkg/apiclient"
"github.com/crowdsecurity/crowdsec/pkg/cwhub"
"github.com/crowdsecurity/crowdsec/pkg/cwversion"
"github.com/crowdsecurity/crowdsec/pkg/database"
"github.com/crowdsecurity/crowdsec/pkg/models"
"github.com/crowdsecurity/crowdsec/pkg/types"
)
const (
SUPPORT_METRICS_HUMAN_PATH = "metrics/metrics.human"
SUPPORT_METRICS_PROMETHEUS_PATH = "metrics/metrics.prometheus"
SUPPORT_VERSION_PATH = "version.txt"
SUPPORT_OS_INFO_PATH = "osinfo.txt"
SUPPORT_PARSERS_PATH = "hub/parsers.txt"
SUPPORT_SCENARIOS_PATH = "hub/scenarios.txt"
SUPPORT_COLLECTIONS_PATH = "hub/collections.txt"
SUPPORT_POSTOVERFLOWS_PATH = "hub/postoverflows.txt"
SUPPORT_BOUNCERS_PATH = "lapi/bouncers.txt"
SUPPORT_AGENTS_PATH = "lapi/agents.txt"
SUPPORT_CROWDSEC_CONFIG_PATH = "config/crowdsec.yaml"
SUPPORT_LAPI_STATUS_PATH = "lapi_status.txt"
SUPPORT_CAPI_STATUS_PATH = "capi_status.txt"
SUPPORT_ACQUISITION_CONFIG_BASE_PATH = "config/acquis/"
SUPPORT_CROWDSEC_PROFILE_PATH = "config/profiles.yaml"
)
func collectMetrics() ([]byte, []byte, error) {
log.Info("Collecting prometheus metrics")
err := csConfig.LoadPrometheus()
if err != nil {
return nil, nil, err
}
if csConfig.Cscli.PrometheusUrl == "" {
log.Warn("No Prometheus URL configured, metrics will not be collected")
return nil, nil, fmt.Errorf("prometheus_uri is not set")
}
humanMetrics := bytes.NewBuffer(nil)
err = FormatPrometheusMetrics(humanMetrics, csConfig.Cscli.PrometheusUrl+"/metrics", "human")
if err != nil {
return nil, nil, fmt.Errorf("could not fetch promtheus metrics: %s", err)
}
req, err := http.NewRequest(http.MethodGet, csConfig.Cscli.PrometheusUrl+"/metrics", nil)
if err != nil {
return nil, nil, fmt.Errorf("could not create requests to prometheus endpoint: %s", err)
}
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
return nil, nil, fmt.Errorf("could not get metrics from prometheus endpoint: %s", err)
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
return nil, nil, fmt.Errorf("could not read metrics from prometheus endpoint: %s", err)
}
return humanMetrics.Bytes(), body, nil
}
func collectVersion() []byte {
log.Info("Collecting version")
return []byte(cwversion.ShowStr())
}
func collectOSInfo() ([]byte, error) {
log.Info("Collecting OS info")
info, err := osinfo.GetOSInfo()
if err != nil {
return nil, err
}
w := bytes.NewBuffer(nil)
w.WriteString(fmt.Sprintf("Architecture: %s\n", info.Architecture))
w.WriteString(fmt.Sprintf("Family: %s\n", info.Family))
w.WriteString(fmt.Sprintf("ID: %s\n", info.ID))
w.WriteString(fmt.Sprintf("Name: %s\n", info.Name))
w.WriteString(fmt.Sprintf("Codename: %s\n", info.Codename))
w.WriteString(fmt.Sprintf("Version: %s\n", info.Version))
w.WriteString(fmt.Sprintf("Build: %s\n", info.Build))
return w.Bytes(), nil
}
func initHub() error {
if err := csConfig.LoadHub(); err != nil {
return fmt.Errorf("cannot load hub: %s", err)
}
if csConfig.Hub == nil {
return fmt.Errorf("hub not configured")
}
if err := cwhub.SetHubBranch(); err != nil {
return fmt.Errorf("cannot set hub branch: %s", err)
}
if err := cwhub.GetHubIdx(csConfig.Hub); err != nil {
return fmt.Errorf("no hub index found: %s", err)
}
return nil
}
func collectHubItems(itemType string) []byte {
out := bytes.NewBuffer(nil)
log.Infof("Collecting %s list", itemType)
ListItems(out, []string{itemType}, []string{}, false, true, all)
return out.Bytes()
}
func collectBouncers(dbClient *database.Client) ([]byte, error) {
out := bytes.NewBuffer(nil)
err := getBouncers(out, dbClient)
if err != nil {
return nil, err
}
return out.Bytes(), nil
}
func collectAgents(dbClient *database.Client) ([]byte, error) {
out := bytes.NewBuffer(nil)
err := getAgents(out, dbClient)
if err != nil {
return nil, err
}
return out.Bytes(), nil
}
func collectAPIStatus(login string, password string, endpoint string, prefix string) []byte {
if csConfig.API.Client == nil || csConfig.API.Client.Credentials == nil {
return []byte("No agent credentials found, are we LAPI ?")
}
pwd := strfmt.Password(password)
apiurl, err := url.Parse(endpoint)
if err != nil {
return []byte(fmt.Sprintf("cannot parse API URL: %s", err))
}
scenarios, err := cwhub.GetInstalledScenariosAsString()
if err != nil {
return []byte(fmt.Sprintf("could not collect scenarios: %s", err))
}
Client, err = apiclient.NewDefaultClient(apiurl,
prefix,
fmt.Sprintf("crowdsec/%s", cwversion.VersionStr()),
nil)
if err != nil {
return []byte(fmt.Sprintf("could not init client: %s", err))
}
t := models.WatcherAuthRequest{
MachineID: &login,
Password: &pwd,
Scenarios: scenarios,
}
_, err = Client.Auth.AuthenticateWatcher(context.Background(), t)
if err != nil {
return []byte(fmt.Sprintf("Could not authenticate to API: %s", err))
} else {
return []byte("Successfully authenticated to LAPI")
}
}
func collectCrowdsecConfig() []byte {
log.Info("Collecting crowdsec config")
config, err := os.ReadFile(*csConfig.FilePath)
if err != nil {
return []byte(fmt.Sprintf("could not read config file: %s", err))
}
r := regexp.MustCompile(`(\s+password:|\s+user:|\s+host:)\s+.*`)
return r.ReplaceAll(config, []byte("$1 ****REDACTED****"))
}
func collectCrowdsecProfile() []byte {
log.Info("Collecting crowdsec profile")
config, err := os.ReadFile(csConfig.API.Server.ProfilesPath)
if err != nil {
return []byte(fmt.Sprintf("could not read profile file: %s", err))
}
return config
}
func collectAcquisitionConfig() map[string][]byte {
log.Info("Collecting acquisition config")
ret := make(map[string][]byte)
for _, filename := range csConfig.Crowdsec.AcquisitionFiles {
fileContent, err := os.ReadFile(filename)
if err != nil {
ret[filename] = []byte(fmt.Sprintf("could not read file: %s", err))
} else {
ret[filename] = fileContent
}
}
return ret
}
func NewSupportCmd() *cobra.Command {
var cmdSupport = &cobra.Command{
Use: "support [action]",
Short: "Provide commands to help during support",
Args: cobra.MinimumNArgs(1),
DisableAutoGenTag: true,
PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
return nil
},
}
var outFile string
cmdDump := &cobra.Command{
Use: "dump",
Short: "Dump all your configuration to a zip file for easier support",
Long: `Dump the following informations:
- Crowdsec version
- OS version
- Installed collections list
- Installed parsers list
- Installed scenarios list
- Installed postoverflows list
- Bouncers list
- Machines list
- CAPI status
- LAPI status
- Crowdsec config (sensitive information like username and password are redacted)
- Crowdsec metrics`,
Example: `cscli support dump
cscli support dump -f /tmp/crowdsec-support.zip
`,
Args: cobra.NoArgs,
DisableAutoGenTag: true,
Run: func(cmd *cobra.Command, args []string) {
var err error
var skipHub, skipDB, skipCAPI, skipLAPI, skipAgent bool
infos := map[string][]byte{
SUPPORT_VERSION_PATH: collectVersion(),
}
if outFile == "" {
outFile = "/tmp/crowdsec-support.zip"
}
dbClient, err = database.NewClient(csConfig.DbConfig)
if err != nil {
log.Warnf("Could not connect to database: %s", err)
skipDB = true
infos[SUPPORT_BOUNCERS_PATH] = []byte(err.Error())
infos[SUPPORT_AGENTS_PATH] = []byte(err.Error())
}
if err := csConfig.LoadAPIServer(); err != nil {
log.Warnf("could not load LAPI, skipping CAPI check")
skipLAPI = true
infos[SUPPORT_CAPI_STATUS_PATH] = []byte(err.Error())
}
if err := csConfig.LoadCrowdsec(); err != nil {
log.Warnf("could not load agent config, skipping crowdsec config check")
skipAgent = true
}
err = initHub()
if err != nil {
log.Warn("Could not init hub, running on LAPI ? Hub related information will not be collected")
skipHub = true
infos[SUPPORT_PARSERS_PATH] = []byte(err.Error())
infos[SUPPORT_SCENARIOS_PATH] = []byte(err.Error())
infos[SUPPORT_POSTOVERFLOWS_PATH] = []byte(err.Error())
infos[SUPPORT_COLLECTIONS_PATH] = []byte(err.Error())
}
if csConfig.API.Client == nil || csConfig.API.Client.Credentials == nil {
log.Warn("no agent credentials found, skipping LAPI connectivity check")
if _, ok := infos[SUPPORT_LAPI_STATUS_PATH]; ok {
infos[SUPPORT_LAPI_STATUS_PATH] = append(infos[SUPPORT_LAPI_STATUS_PATH], []byte("\nNo LAPI credentials found")...)
}
skipLAPI = true
}
if csConfig.API.Server == nil || csConfig.API.Server.OnlineClient.Credentials == nil {
log.Warn("no CAPI credentials found, skipping CAPI connectivity check")
skipCAPI = true
}
infos[SUPPORT_METRICS_HUMAN_PATH], infos[SUPPORT_METRICS_PROMETHEUS_PATH], err = collectMetrics()
if err != nil {
log.Warnf("could not collect prometheus metrics information: %s", err)
infos[SUPPORT_METRICS_HUMAN_PATH] = []byte(err.Error())
infos[SUPPORT_METRICS_PROMETHEUS_PATH] = []byte(err.Error())
}
infos[SUPPORT_OS_INFO_PATH], err = collectOSInfo()
if err != nil {
log.Warnf("could not collect OS information: %s", err)
infos[SUPPORT_OS_INFO_PATH] = []byte(err.Error())
}
infos[SUPPORT_CROWDSEC_CONFIG_PATH] = collectCrowdsecConfig()
if !skipHub {
infos[SUPPORT_PARSERS_PATH] = collectHubItems(cwhub.PARSERS)
infos[SUPPORT_SCENARIOS_PATH] = collectHubItems(cwhub.SCENARIOS)
infos[SUPPORT_POSTOVERFLOWS_PATH] = collectHubItems(cwhub.PARSERS_OVFLW)
infos[SUPPORT_COLLECTIONS_PATH] = collectHubItems(cwhub.COLLECTIONS)
}
if !skipDB {
infos[SUPPORT_BOUNCERS_PATH], err = collectBouncers(dbClient)
if err != nil {
log.Warnf("could not collect bouncers information: %s", err)
infos[SUPPORT_BOUNCERS_PATH] = []byte(err.Error())
}
infos[SUPPORT_AGENTS_PATH], err = collectAgents(dbClient)
if err != nil {
log.Warnf("could not collect agents information: %s", err)
infos[SUPPORT_AGENTS_PATH] = []byte(err.Error())
}
}
if !skipCAPI {
log.Info("Collecting CAPI status")
infos[SUPPORT_CAPI_STATUS_PATH] = collectAPIStatus(csConfig.API.Server.OnlineClient.Credentials.Login,
csConfig.API.Server.OnlineClient.Credentials.Password,
csConfig.API.Server.OnlineClient.Credentials.URL,
CAPIURLPrefix)
}
if !skipLAPI {
log.Info("Collection LAPI status")
infos[SUPPORT_LAPI_STATUS_PATH] = collectAPIStatus(csConfig.API.Client.Credentials.Login,
csConfig.API.Client.Credentials.Password,
csConfig.API.Client.Credentials.URL,
LAPIURLPrefix)
infos[SUPPORT_CROWDSEC_PROFILE_PATH] = collectCrowdsecProfile()
}
if !skipAgent {
acquis := collectAcquisitionConfig()
for filename, content := range acquis {
fname := strings.ReplaceAll(filename, string(filepath.Separator), "___")
infos[SUPPORT_ACQUISITION_CONFIG_BASE_PATH+fname] = content
}
}
w := bytes.NewBuffer(nil)
zipWriter := zip.NewWriter(w)
for filename, data := range infos {
fw, err := zipWriter.Create(filename)
if err != nil {
log.Errorf("Could not add zip entry for %s: %s", filename, err)
continue
}
fw.Write([]byte(types.StripAnsiString(string(data))))
}
err = zipWriter.Close()
if err != nil {
log.Fatalf("could not finalize zip file: %s", err)
}
err = os.WriteFile(outFile, w.Bytes(), 0600)
if err != nil {
log.Fatalf("could not write zip file to %s: %s", outFile, err)
}
log.Infof("Written zip file to %s", outFile)
},
}
cmdDump.Flags().StringVarP(&outFile, "outFile", "f", "", "File to dump the information to")
cmdSupport.AddCommand(cmdDump)
return cmdSupport
}
crowdsec-1.4.6/cmd/crowdsec-cli/tables.go 0000664 0000000 0000000 00000003256 14371172257 0020274 0 ustar 00root root 0000000 0000000 package main
import (
"fmt"
"io"
"os"
"github.com/aquasecurity/table"
isatty "github.com/mattn/go-isatty"
)
func shouldWeColorize() bool {
if csConfig.Cscli.Color == "yes" {
return true
}
if csConfig.Cscli.Color == "no" {
return false
}
return isatty.IsTerminal(os.Stdout.Fd()) || isatty.IsCygwinTerminal(os.Stdout.Fd())
}
func newTable(out io.Writer) *table.Table {
if out == nil {
panic("newTable: out is nil")
}
t := table.New(out)
if shouldWeColorize() {
t.SetLineStyle(table.StyleBrightBlack)
t.SetHeaderStyle(table.StyleItalic)
}
if shouldWeColorize() {
t.SetDividers(table.UnicodeRoundedDividers)
} else {
t.SetDividers(table.ASCIIDividers)
}
return t
}
func newLightTable(out io.Writer) *table.Table {
if out == nil {
panic("newTable: out is nil")
}
t := newTable(out)
t.SetRowLines(false)
t.SetBorderLeft(false)
t.SetBorderRight(false)
// This leaves three spaces between columns:
// left padding, invisible border, right padding
// There is no way to make two spaces without
// a SetColumnLines() method, but it's close enough.
t.SetPadding(1)
if shouldWeColorize() {
t.SetDividers(table.Dividers{
ALL: "â",
NES: "â",
NSW: "â",
NEW: "â",
ESW: "â",
NE: "â",
NW: "â",
SW: "â",
ES: "â",
EW: "â",
NS: " ",
})
} else {
t.SetDividers(table.Dividers{
ALL: "-",
NES: "-",
NSW: "-",
NEW: "-",
ESW: "-",
NE: "-",
NW: "-",
SW: "-",
ES: "-",
EW: "-",
NS: " ",
})
}
return t
}
func renderTableTitle(out io.Writer, title string) {
if out == nil {
panic("renderTableTitle: out is nil")
}
if title == "" {
return
}
fmt.Fprintln(out, title)
}
crowdsec-1.4.6/cmd/crowdsec-cli/utils.go 0000664 0000000 0000000 00000047730 14371172257 0020167 0 ustar 00root root 0000000 0000000 package main
import (
"encoding/csv"
"encoding/json"
"fmt"
"io"
"math"
"net"
"net/http"
"os"
"strconv"
"strings"
"time"
"github.com/fatih/color"
dto "github.com/prometheus/client_model/go"
"github.com/prometheus/prom2json"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"github.com/texttheater/golang-levenshtein/levenshtein"
"gopkg.in/yaml.v2"
"github.com/crowdsecurity/crowdsec/pkg/cwhub"
"github.com/crowdsecurity/crowdsec/pkg/types"
)
const MaxDistance = 7
func printHelp(cmd *cobra.Command) {
err := cmd.Help()
if err != nil {
log.Fatalf("unable to print help(): %s", err)
}
}
func inSlice(s string, slice []string) bool {
for _, str := range slice {
if s == str {
return true
}
}
return false
}
func indexOf(s string, slice []string) int {
for i, elem := range slice {
if s == elem {
return i
}
}
return -1
}
func LoadHub() error {
if err := csConfig.LoadHub(); err != nil {
log.Fatal(err)
}
if csConfig.Hub == nil {
return fmt.Errorf("unable to load hub")
}
if err := cwhub.SetHubBranch(); err != nil {
log.Warningf("unable to set hub branch (%s), default to master", err)
}
if err := cwhub.GetHubIdx(csConfig.Hub); err != nil {
return fmt.Errorf("Failed to get Hub index : '%w'. Run 'sudo cscli hub update' to get the hub index", err)
}
return nil
}
func Suggest(itemType string, baseItem string, suggestItem string, score int, ignoreErr bool) {
errMsg := ""
if score < MaxDistance {
errMsg = fmt.Sprintf("unable to find %s '%s', did you mean %s ?", itemType, baseItem, suggestItem)
} else {
errMsg = fmt.Sprintf("unable to find %s '%s'", itemType, baseItem)
}
if ignoreErr {
log.Error(errMsg)
} else {
log.Fatalf(errMsg)
}
}
func GetDistance(itemType string, itemName string) (*cwhub.Item, int) {
allItems := make([]string, 0)
nearestScore := 100
nearestItem := &cwhub.Item{}
hubItems := cwhub.GetHubStatusForItemType(itemType, "", true)
for _, item := range hubItems {
allItems = append(allItems, item.Name)
}
for _, s := range allItems {
d := levenshtein.DistanceForStrings([]rune(itemName), []rune(s), levenshtein.DefaultOptions)
if d < nearestScore {
nearestScore = d
nearestItem = cwhub.GetItem(itemType, s)
}
}
return nearestItem, nearestScore
}
func compAllItems(itemType string, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
if err := LoadHub(); err != nil {
return nil, cobra.ShellCompDirectiveDefault
}
comp := make([]string, 0)
hubItems := cwhub.GetHubStatusForItemType(itemType, "", true)
for _, item := range hubItems {
if !inSlice(item.Name, args) && strings.Contains(item.Name, toComplete) {
comp = append(comp, item.Name)
}
}
cobra.CompDebugln(fmt.Sprintf("%s: %+v", itemType, comp), true)
return comp, cobra.ShellCompDirectiveNoFileComp
}
func compInstalledItems(itemType string, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
if err := LoadHub(); err != nil {
return nil, cobra.ShellCompDirectiveDefault
}
var items []string
var err error
switch itemType {
case cwhub.PARSERS:
items, err = cwhub.GetInstalledParsersAsString()
case cwhub.SCENARIOS:
items, err = cwhub.GetInstalledScenariosAsString()
case cwhub.PARSERS_OVFLW:
items, err = cwhub.GetInstalledPostOverflowsAsString()
case cwhub.COLLECTIONS:
items, err = cwhub.GetInstalledCollectionsAsString()
default:
return nil, cobra.ShellCompDirectiveDefault
}
if err != nil {
cobra.CompDebugln(fmt.Sprintf("list installed %s err: %s", itemType, err), true)
return nil, cobra.ShellCompDirectiveDefault
}
comp := make([]string, 0)
if toComplete != "" {
for _, item := range items {
if strings.Contains(item, toComplete) {
comp = append(comp, item)
}
}
} else {
comp = items
}
cobra.CompDebugln(fmt.Sprintf("%s: %+v", itemType, comp), true)
return comp, cobra.ShellCompDirectiveNoFileComp
}
func ListItems(out io.Writer, itemTypes []string, args []string, showType bool, showHeader bool, all bool) {
var hubStatusByItemType = make(map[string][]cwhub.ItemHubStatus)
for _, itemType := range itemTypes {
itemName := ""
if len(args) == 1 {
itemName = args[0]
}
hubStatusByItemType[itemType] = cwhub.GetHubStatusForItemType(itemType, itemName, all)
}
if csConfig.Cscli.Output == "human" {
for _, itemType := range itemTypes {
var statuses []cwhub.ItemHubStatus
var ok bool
if statuses, ok = hubStatusByItemType[itemType]; !ok {
log.Errorf("unknown item type: %s", itemType)
continue
}
listHubItemTable(out, "\n"+strings.ToUpper(itemType), statuses)
}
} else if csConfig.Cscli.Output == "json" {
x, err := json.MarshalIndent(hubStatusByItemType, "", " ")
if err != nil {
log.Fatalf("failed to unmarshal")
}
out.Write(x)
} else if csConfig.Cscli.Output == "raw" {
csvwriter := csv.NewWriter(out)
if showHeader {
header := []string{"name", "status", "version", "description"}
if showType {
header = append(header, "type")
}
err := csvwriter.Write(header)
if err != nil {
log.Fatalf("failed to write header: %s", err)
}
}
for _, itemType := range itemTypes {
var statuses []cwhub.ItemHubStatus
var ok bool
if statuses, ok = hubStatusByItemType[itemType]; !ok {
log.Errorf("unknown item type: %s", itemType)
continue
}
for _, status := range statuses {
if status.LocalVersion == "" {
status.LocalVersion = "n/a"
}
row := []string{
status.Name,
status.Status,
status.LocalVersion,
status.Description,
}
if showType {
row = append(row, itemType)
}
err := csvwriter.Write(row)
if err != nil {
log.Fatalf("failed to write raw output : %s", err)
}
}
}
csvwriter.Flush()
}
}
func InspectItem(name string, objecitemType string) {
hubItem := cwhub.GetItem(objecitemType, name)
if hubItem == nil {
log.Fatalf("unable to retrieve item.")
}
var b []byte
var err error
switch csConfig.Cscli.Output {
case "human", "raw":
b, err = yaml.Marshal(*hubItem)
if err != nil {
log.Fatalf("unable to marshal item : %s", err)
}
case "json":
b, err = json.MarshalIndent(*hubItem, "", " ")
if err != nil {
log.Fatalf("unable to marshal item : %s", err)
}
}
fmt.Printf("%s", string(b))
if csConfig.Cscli.Output == "json" || csConfig.Cscli.Output == "raw" {
return
}
if prometheusURL == "" {
//This is technically wrong to do this, as the prometheus section contains a listen address, not an URL to query prometheus
//But for ease of use, we will use the listen address as the prometheus URL because it will be 127.0.0.1 in the default case
listenAddr := csConfig.Prometheus.ListenAddr
if listenAddr == "" {
listenAddr = "127.0.0.1"
}
listenPort := csConfig.Prometheus.ListenPort
if listenPort == 0 {
listenPort = 6060
}
prometheusURL = fmt.Sprintf("http://%s:%d/metrics", listenAddr, listenPort)
log.Debugf("No prometheus URL provided using: %s", prometheusURL)
}
fmt.Printf("\nCurrent metrics : \n")
ShowMetrics(hubItem)
}
func manageCliDecisionAlerts(ip *string, ipRange *string, scope *string, value *string) error {
/*if a range is provided, change the scope*/
if *ipRange != "" {
_, _, err := net.ParseCIDR(*ipRange)
if err != nil {
return fmt.Errorf("%s isn't a valid range", *ipRange)
}
}
if *ip != "" {
ipRepr := net.ParseIP(*ip)
if ipRepr == nil {
return fmt.Errorf("%s isn't a valid ip", *ip)
}
}
//avoid confusion on scope (ip vs Ip and range vs Range)
switch strings.ToLower(*scope) {
case "ip":
*scope = types.Ip
case "range":
*scope = types.Range
case "country":
*scope = types.Country
case "as":
*scope = types.AS
}
return nil
}
func ShowMetrics(hubItem *cwhub.Item) {
switch hubItem.Type {
case cwhub.PARSERS:
metrics := GetParserMetric(prometheusURL, hubItem.Name)
parserMetricsTable(color.Output, hubItem.Name, metrics)
case cwhub.SCENARIOS:
metrics := GetScenarioMetric(prometheusURL, hubItem.Name)
scenarioMetricsTable(color.Output, hubItem.Name, metrics)
case cwhub.COLLECTIONS:
for _, item := range hubItem.Parsers {
metrics := GetParserMetric(prometheusURL, item)
parserMetricsTable(color.Output, item, metrics)
}
for _, item := range hubItem.Scenarios {
metrics := GetScenarioMetric(prometheusURL, item)
scenarioMetricsTable(color.Output, item, metrics)
}
for _, item := range hubItem.Collections {
hubItem = cwhub.GetItem(cwhub.COLLECTIONS, item)
if hubItem == nil {
log.Fatalf("unable to retrieve item '%s' from collection '%s'", item, hubItem.Name)
}
ShowMetrics(hubItem)
}
default:
log.Errorf("item of type '%s' is unknown", hubItem.Type)
}
}
// GetParserMetric is a complete rip from prom2json
func GetParserMetric(url string, itemName string) map[string]map[string]int {
stats := make(map[string]map[string]int)
result := GetPrometheusMetric(url)
for idx, fam := range result {
if !strings.HasPrefix(fam.Name, "cs_") {
continue
}
log.Tracef("round %d", idx)
for _, m := range fam.Metrics {
metric, ok := m.(prom2json.Metric)
if !ok {
log.Debugf("failed to convert metric to prom2json.Metric")
continue
}
name, ok := metric.Labels["name"]
if !ok {
log.Debugf("no name in Metric %v", metric.Labels)
}
if name != itemName {
continue
}
source, ok := metric.Labels["source"]
if !ok {
log.Debugf("no source in Metric %v", metric.Labels)
} else {
if srctype, ok := metric.Labels["type"]; ok {
source = srctype + ":" + source
}
}
value := m.(prom2json.Metric).Value
fval, err := strconv.ParseFloat(value, 32)
if err != nil {
log.Errorf("Unexpected int value %s : %s", value, err)
continue
}
ival := int(fval)
switch fam.Name {
case "cs_reader_hits_total":
if _, ok := stats[source]; !ok {
stats[source] = make(map[string]int)
stats[source]["parsed"] = 0
stats[source]["reads"] = 0
stats[source]["unparsed"] = 0
stats[source]["hits"] = 0
}
stats[source]["reads"] += ival
case "cs_parser_hits_ok_total":
if _, ok := stats[source]; !ok {
stats[source] = make(map[string]int)
}
stats[source]["parsed"] += ival
case "cs_parser_hits_ko_total":
if _, ok := stats[source]; !ok {
stats[source] = make(map[string]int)
}
stats[source]["unparsed"] += ival
case "cs_node_hits_total":
if _, ok := stats[source]; !ok {
stats[source] = make(map[string]int)
}
stats[source]["hits"] += ival
case "cs_node_hits_ok_total":
if _, ok := stats[source]; !ok {
stats[source] = make(map[string]int)
}
stats[source]["parsed"] += ival
case "cs_node_hits_ko_total":
if _, ok := stats[source]; !ok {
stats[source] = make(map[string]int)
}
stats[source]["unparsed"] += ival
default:
continue
}
}
}
return stats
}
func GetScenarioMetric(url string, itemName string) map[string]int {
stats := make(map[string]int)
stats["instantiation"] = 0
stats["curr_count"] = 0
stats["overflow"] = 0
stats["pour"] = 0
stats["underflow"] = 0
result := GetPrometheusMetric(url)
for idx, fam := range result {
if !strings.HasPrefix(fam.Name, "cs_") {
continue
}
log.Tracef("round %d", idx)
for _, m := range fam.Metrics {
metric, ok := m.(prom2json.Metric)
if !ok {
log.Debugf("failed to convert metric to prom2json.Metric")
continue
}
name, ok := metric.Labels["name"]
if !ok {
log.Debugf("no name in Metric %v", metric.Labels)
}
if name != itemName {
continue
}
value := m.(prom2json.Metric).Value
fval, err := strconv.ParseFloat(value, 32)
if err != nil {
log.Errorf("Unexpected int value %s : %s", value, err)
continue
}
ival := int(fval)
switch fam.Name {
case "cs_bucket_created_total":
stats["instantiation"] += ival
case "cs_buckets":
stats["curr_count"] += ival
case "cs_bucket_overflowed_total":
stats["overflow"] += ival
case "cs_bucket_poured_total":
stats["pour"] += ival
case "cs_bucket_underflowed_total":
stats["underflow"] += ival
default:
continue
}
}
}
return stats
}
// it's a rip of the cli version, but in silent-mode
func silenceInstallItem(name string, obtype string) (string, error) {
var item = cwhub.GetItem(obtype, name)
if item == nil {
return "", fmt.Errorf("error retrieving item")
}
it := *item
if downloadOnly && it.Downloaded && it.UpToDate {
return fmt.Sprintf("%s is already downloaded and up-to-date", it.Name), nil
}
it, err := cwhub.DownloadLatest(csConfig.Hub, it, forceAction, false)
if err != nil {
return "", fmt.Errorf("error while downloading %s : %v", it.Name, err)
}
if err := cwhub.AddItem(obtype, it); err != nil {
return "", err
}
if downloadOnly {
return fmt.Sprintf("Downloaded %s to %s", it.Name, csConfig.Cscli.HubDir+"/"+it.RemotePath), nil
}
it, err = cwhub.EnableItem(csConfig.Hub, it)
if err != nil {
return "", fmt.Errorf("error while enabling %s : %v", it.Name, err)
}
if err := cwhub.AddItem(obtype, it); err != nil {
return "", err
}
return fmt.Sprintf("Enabled %s", it.Name), nil
}
func GetPrometheusMetric(url string) []*prom2json.Family {
mfChan := make(chan *dto.MetricFamily, 1024)
// Start with the DefaultTransport for sane defaults.
transport := http.DefaultTransport.(*http.Transport).Clone()
// Conservatively disable HTTP keep-alives as this program will only
// ever need a single HTTP request.
transport.DisableKeepAlives = true
// Timeout early if the server doesn't even return the headers.
transport.ResponseHeaderTimeout = time.Minute
go func() {
defer types.CatchPanic("crowdsec/GetPrometheusMetric")
err := prom2json.FetchMetricFamilies(url, mfChan, transport)
if err != nil {
log.Fatalf("failed to fetch prometheus metrics : %v", err)
}
}()
result := []*prom2json.Family{}
for mf := range mfChan {
result = append(result, prom2json.NewFamily(mf))
}
log.Debugf("Finished reading prometheus output, %d entries", len(result))
return result
}
func RestoreHub(dirPath string) error {
var err error
if err := csConfig.LoadHub(); err != nil {
return err
}
if err := cwhub.SetHubBranch(); err != nil {
return fmt.Errorf("error while setting hub branch: %s", err)
}
for _, itype := range cwhub.ItemTypes {
itemDirectory := fmt.Sprintf("%s/%s/", dirPath, itype)
if _, err = os.Stat(itemDirectory); err != nil {
log.Infof("no %s in backup", itype)
continue
}
/*restore the upstream items*/
upstreamListFN := fmt.Sprintf("%s/upstream-%s.json", itemDirectory, itype)
file, err := os.ReadFile(upstreamListFN)
if err != nil {
return fmt.Errorf("error while opening %s : %s", upstreamListFN, err)
}
var upstreamList []string
err = json.Unmarshal(file, &upstreamList)
if err != nil {
return fmt.Errorf("error unmarshaling %s : %s", upstreamListFN, err)
}
for _, toinstall := range upstreamList {
label, err := silenceInstallItem(toinstall, itype)
if err != nil {
log.Errorf("Error while installing %s : %s", toinstall, err)
} else if label != "" {
log.Infof("Installed %s : %s", toinstall, label)
} else {
log.Printf("Installed %s : ok", toinstall)
}
}
/*restore the local and tainted items*/
files, err := os.ReadDir(itemDirectory)
if err != nil {
return fmt.Errorf("failed enumerating files of %s : %s", itemDirectory, err)
}
for _, file := range files {
//this was the upstream data
if file.Name() == fmt.Sprintf("upstream-%s.json", itype) {
continue
}
if itype == cwhub.PARSERS || itype == cwhub.PARSERS_OVFLW {
//we expect a stage here
if !file.IsDir() {
continue
}
stage := file.Name()
stagedir := fmt.Sprintf("%s/%s/%s/", csConfig.ConfigPaths.ConfigDir, itype, stage)
log.Debugf("Found stage %s in %s, target directory : %s", stage, itype, stagedir)
if err = os.MkdirAll(stagedir, os.ModePerm); err != nil {
return fmt.Errorf("error while creating stage directory %s : %s", stagedir, err)
}
/*find items*/
ifiles, err := os.ReadDir(itemDirectory + "/" + stage + "/")
if err != nil {
return fmt.Errorf("failed enumerating files of %s : %s", itemDirectory+"/"+stage, err)
}
//finally copy item
for _, tfile := range ifiles {
log.Infof("Going to restore local/tainted [%s]", tfile.Name())
sourceFile := fmt.Sprintf("%s/%s/%s", itemDirectory, stage, tfile.Name())
destinationFile := fmt.Sprintf("%s%s", stagedir, tfile.Name())
if err = types.CopyFile(sourceFile, destinationFile); err != nil {
return fmt.Errorf("failed copy %s %s to %s : %s", itype, sourceFile, destinationFile, err)
}
log.Infof("restored %s to %s", sourceFile, destinationFile)
}
} else {
log.Infof("Going to restore local/tainted [%s]", file.Name())
sourceFile := fmt.Sprintf("%s/%s", itemDirectory, file.Name())
destinationFile := fmt.Sprintf("%s/%s/%s", csConfig.ConfigPaths.ConfigDir, itype, file.Name())
if err = types.CopyFile(sourceFile, destinationFile); err != nil {
return fmt.Errorf("failed copy %s %s to %s : %s", itype, sourceFile, destinationFile, err)
}
log.Infof("restored %s to %s", sourceFile, destinationFile)
}
}
}
return nil
}
func BackupHub(dirPath string) error {
var err error
var itemDirectory string
var upstreamParsers []string
for _, itemType := range cwhub.ItemTypes {
clog := log.WithFields(log.Fields{
"type": itemType,
})
itemMap := cwhub.GetItemMap(itemType)
if itemMap == nil {
clog.Infof("No %s to backup.", itemType)
continue
}
itemDirectory = fmt.Sprintf("%s/%s/", dirPath, itemType)
if err := os.MkdirAll(itemDirectory, os.ModePerm); err != nil {
return fmt.Errorf("error while creating %s : %s", itemDirectory, err)
}
upstreamParsers = []string{}
for k, v := range itemMap {
clog = clog.WithFields(log.Fields{
"file": v.Name,
})
if !v.Installed { //only backup installed ones
clog.Debugf("[%s] : not installed", k)
continue
}
//for the local/tainted ones, we backup the full file
if v.Tainted || v.Local || !v.UpToDate {
//we need to backup stages for parsers
if itemType == cwhub.PARSERS || itemType == cwhub.PARSERS_OVFLW {
fstagedir := fmt.Sprintf("%s%s", itemDirectory, v.Stage)
if err := os.MkdirAll(fstagedir, os.ModePerm); err != nil {
return fmt.Errorf("error while creating stage dir %s : %s", fstagedir, err)
}
}
clog.Debugf("[%s] : backuping file (tainted:%t local:%t up-to-date:%t)", k, v.Tainted, v.Local, v.UpToDate)
tfile := fmt.Sprintf("%s%s/%s", itemDirectory, v.Stage, v.FileName)
if err = types.CopyFile(v.LocalPath, tfile); err != nil {
return fmt.Errorf("failed copy %s %s to %s : %s", itemType, v.LocalPath, tfile, err)
}
clog.Infof("local/tainted saved %s to %s", v.LocalPath, tfile)
continue
}
clog.Debugf("[%s] : from hub, just backup name (up-to-date:%t)", k, v.UpToDate)
clog.Infof("saving, version:%s, up-to-date:%t", v.Version, v.UpToDate)
upstreamParsers = append(upstreamParsers, v.Name)
}
//write the upstream items
upstreamParsersFname := fmt.Sprintf("%s/upstream-%s.json", itemDirectory, itemType)
upstreamParsersContent, err := json.MarshalIndent(upstreamParsers, "", " ")
if err != nil {
return fmt.Errorf("failed marshaling upstream parsers : %s", err)
}
err = os.WriteFile(upstreamParsersFname, upstreamParsersContent, 0644)
if err != nil {
return fmt.Errorf("unable to write to %s %s : %s", itemType, upstreamParsersFname, err)
}
clog.Infof("Wrote %d entries for %s to %s", len(upstreamParsers), itemType, upstreamParsersFname)
}
return nil
}
type unit struct {
value int64
symbol string
}
var ranges = []unit{
{
value: 1e18,
symbol: "E",
},
{
value: 1e15,
symbol: "P",
},
{
value: 1e12,
symbol: "T",
},
{
value: 1e6,
symbol: "M",
},
{
value: 1e3,
symbol: "k",
},
{
value: 1,
symbol: "",
},
}
func formatNumber(num int) string {
goodUnit := unit{}
for _, u := range ranges {
if int64(num) >= u.value {
goodUnit = u
break
}
}
if goodUnit.value == 1 {
return fmt.Sprintf("%d%s", num, goodUnit.symbol)
}
res := math.Round(float64(num)/float64(goodUnit.value)*100) / 100
return fmt.Sprintf("%.2f%s", res, goodUnit.symbol)
}
crowdsec-1.4.6/cmd/crowdsec-cli/utils_table.go 0000664 0000000 0000000 00000003333 14371172257 0021325 0 ustar 00root root 0000000 0000000 package main
import (
"fmt"
"io"
"github.com/aquasecurity/table"
"github.com/enescakir/emoji"
"github.com/crowdsecurity/crowdsec/pkg/cwhub"
)
func listHubItemTable(out io.Writer, title string, statuses []cwhub.ItemHubStatus) {
t := newLightTable(out)
t.SetHeaders("Name", fmt.Sprintf("%v Status", emoji.Package), "Version", "Local Path")
t.SetHeaderAlignment(table.AlignLeft, table.AlignLeft, table.AlignLeft, table.AlignLeft)
t.SetAlignment(table.AlignLeft, table.AlignLeft, table.AlignLeft, table.AlignLeft)
for _, status := range statuses {
t.AddRow(status.Name, status.UTF8_Status, status.LocalVersion, status.LocalPath)
}
renderTableTitle(out, title)
t.Render()
}
func scenarioMetricsTable(out io.Writer, itemName string, metrics map[string]int) {
if metrics["instantiation"] == 0 {
return
}
t := newTable(out)
t.SetHeaders("Current Count", "Overflows", "Instantiated", "Poured", "Expired")
t.AddRow(
fmt.Sprintf("%d", metrics["curr_count"]),
fmt.Sprintf("%d", metrics["overflow"]),
fmt.Sprintf("%d", metrics["instantiation"]),
fmt.Sprintf("%d", metrics["pour"]),
fmt.Sprintf("%d", metrics["underflow"]),
)
renderTableTitle(out, fmt.Sprintf("\n - (Scenario) %s:", itemName))
t.Render()
}
func parserMetricsTable(out io.Writer, itemName string, metrics map[string]map[string]int) {
skip := true
t := newTable(out)
t.SetHeaders("Parsers", "Hits", "Parsed", "Unparsed")
for source, stats := range metrics {
if stats["hits"] > 0 {
t.AddRow(
source,
fmt.Sprintf("%d", stats["hits"]),
fmt.Sprintf("%d", stats["parsed"]),
fmt.Sprintf("%d", stats["unparsed"]),
)
skip = false
}
}
if !skip {
renderTableTitle(out, fmt.Sprintf("\n - (Parser) %s:", itemName))
t.Render()
}
}
crowdsec-1.4.6/cmd/crowdsec/ 0000775 0000000 0000000 00000000000 14371172257 0015720 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/cmd/crowdsec/Makefile 0000664 0000000 0000000 00000005443 14371172257 0017366 0 ustar 00root root 0000000 0000000 ifeq ($(OS), Windows_NT)
SHELL := pwsh.exe
.SHELLFLAGS := -NoProfile -Command
EXT = .exe
endif
# Go parameters
GOCMD = go
GOBUILD = $(GOCMD) build
GOCLEAN = $(GOCMD) clean
GOTEST = $(GOCMD) test
GOGET = $(GOCMD) get
CROWDSEC_BIN = crowdsec$(EXT)
# names longer than 15 chars break 'pgrep'
CROWDSEC_BIN_COVER = $(CROWDSEC_BIN).cover
PREFIX ?= "/"
CFG_PREFIX = $(PREFIX)"/etc/crowdsec/config/"
BIN_PREFIX = $(PREFIX)"/usr/local/bin/"
DATA_PREFIX = $(PREFIX)"/var/run/crowdsec/"
PID_DIR = $(PREFIX)"/var/run/"
SYSTEMD_PATH_FILE = "/etc/systemd/system/crowdsec.service"
.PHONY: all
all: clean test build
build: clean
$(GOBUILD) $(LD_OPTS) $(BUILD_VENDOR_FLAGS) -o $(CROWDSEC_BIN)
build-bincover: clean
$(GOTEST) . -tags testrunmain -coverpkg=$(go list github.com/crowdsecurity/crowdsec/... | grep -v -e 'pkg/database' -e 'plugins/notifications' -e 'pkg/protobufs' -e 'pkg/cwversions' -e 'pkg/cstest' -e 'pkg/models') -covermode=atomic $(LD_OPTS) -c -o $(CROWDSEC_BIN_COVER)
test:
$(GOTEST) $(LD_OPTS) -v ./...
clean:
@$(RM) $(CROWDSEC_BIN) $(CROWDSEC_BIN).test $(CROWDSEC_BIN_COVER) $(WIN_IGNORE_ERR)
.PHONY: install
install: install-conf install-bin
.PHONY: install-conf
install-conf:
mkdir -p $(DATA_PREFIX) || exit
(cd ../.. / && find ./data -type f -exec install -Dm 755 "{}" "$(DATA_PREFIX){}" \; && cd ./cmd/crowdsec) || exit
(cd ../../config && find ./patterns -type f -exec install -Dm 755 "{}" "$(CFG_PREFIX){}" \; && cd ../cmd/crowdsec) || exit
mkdir -p "$(CFG_PREFIX)" || exit
mkdir -p "$(CFG_PREFIX)/parsers" || exit
mkdir -p "$(CFG_PREFIX)/scenarios" || exit
mkdir -p "$(CFG_PREFIX)/postoverflows" || exit
mkdir -p "$(CFG_PREFIX)/collections" || exit
mkdir -p "$(CFG_PREFIX)/patterns" || exit
install -v -m 755 -D ../../config/prod.yaml "$(CFG_PREFIX)" || exit
install -v -m 755 -D ../../config/dev.yaml "$(CFG_PREFIX)" || exit
install -v -m 755 -D ../../config/acquis.yaml "$(CFG_PREFIX)" || exit
install -v -m 755 -D ../../config/profiles.yaml "$(CFG_PREFIX)" || exit
install -v -m 755 -D ../../config/api.yaml "$(CFG_PREFIX)" || exit
mkdir -p $(PID_DIR) || exit
PID=$(PID_DIR) DATA=$(DATA_PREFIX)"/data/" CFG=$(CFG_PREFIX) envsubst < ../../config/prod.yaml > $(CFG_PREFIX)"/default.yaml"
.PHONY: install-bin
install-bin:
install -v -m 755 -D "$(CROWDSEC_BIN)" "$(BIN_PREFIX)/$(CROWDSEC_BIN)" || exit
.PHONY: systemd"$(BIN_PREFI"$(BIN_PREFIX)/$(CROWDSEC_BIN)""$(BIN_PREFIX)/$(CROWDSEC_BIN)"X)/$(CROWDSEC_BIN)"
systemd: install
CFG=$(CFG_PREFIX) PID=$(PID_DIR) BIN=$(BIN_PREFIX)"/"$(CROWDSEC_BIN) envsubst < ../../config/crowdsec.service > "$(SYSTEMD_PATH_FILE)"
systemctl daemon-reload
.PHONY: uninstall
uninstall:
$(RM) $(CFG_PREFIX) $(WIN_IGNORE_ERR)
$(RM) $(DATA_PREFIX) $(WIN_IGNORE_ERR)
$(RM) "$(BIN_PREFIX)/$(CROWDSEC_BIN)" $(WIN_IGNORE_ERR)
$(RM) "$(SYSTEMD_PATH_FILE)" $(WIN_IGNORE_ERR)
crowdsec-1.4.6/cmd/crowdsec/api.go 0000664 0000000 0000000 00000004501 14371172257 0017020 0 ustar 00root root 0000000 0000000 package main
import (
"runtime"
"time"
"github.com/crowdsecurity/crowdsec/pkg/apiserver"
"github.com/crowdsecurity/crowdsec/pkg/csconfig"
"github.com/crowdsecurity/crowdsec/pkg/types"
"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
)
func initAPIServer(cConfig *csconfig.Config) (*apiserver.APIServer, error) {
apiServer, err := apiserver.NewServer(cConfig.API.Server)
if err != nil {
return nil, errors.Wrap(err, "unable to run local API")
}
if hasPlugins(cConfig.API.Server.Profiles) {
log.Info("initiating plugin broker")
//On windows, the plugins are always run as medium-integrity processes, so we don't care about plugin_config
if cConfig.PluginConfig == nil && runtime.GOOS != "windows" {
return nil, errors.New("plugins are enabled, but the plugin_config section is missing in the configuration")
}
if cConfig.ConfigPaths.NotificationDir == "" {
return nil, errors.New("plugins are enabled, but config_paths.notification_dir is not defined")
}
if cConfig.ConfigPaths.PluginDir == "" {
return nil, errors.New("plugins are enabled, but config_paths.plugin_dir is not defined")
}
err = pluginBroker.Init(cConfig.PluginConfig, cConfig.API.Server.Profiles, cConfig.ConfigPaths)
if err != nil {
return nil, errors.Wrap(err, "unable to run local API")
}
log.Info("initiated plugin broker")
apiServer.AttachPluginBroker(&pluginBroker)
}
err = apiServer.InitController()
if err != nil {
return nil, errors.Wrap(err, "unable to run local API")
}
return apiServer, nil
}
func serveAPIServer(apiServer *apiserver.APIServer, apiReady chan bool) {
apiTomb.Go(func() error {
defer types.CatchPanic("crowdsec/serveAPIServer")
go func() {
defer types.CatchPanic("crowdsec/runAPIServer")
log.Debugf("serving API after %s ms", time.Since(crowdsecT0))
if err := apiServer.Run(apiReady); err != nil {
log.Fatal(err)
}
}()
pluginTomb.Go(func() error {
pluginBroker.Run(&pluginTomb)
return nil
})
<-apiTomb.Dying() // lock until go routine is dying
pluginTomb.Kill(nil)
log.Infof("serve: shutting down api server")
if err := apiServer.Shutdown(); err != nil {
return err
}
return nil
})
}
func hasPlugins(profiles []*csconfig.ProfileCfg) bool {
for _, profile := range profiles {
if len(profile.Notifications) != 0 {
return true
}
}
return false
}
crowdsec-1.4.6/cmd/crowdsec/crowdsec.go 0000664 0000000 0000000 00000017056 14371172257 0020071 0 ustar 00root root 0000000 0000000 package main
import (
"fmt"
"os"
"sync"
"time"
"path/filepath"
"github.com/crowdsecurity/crowdsec/pkg/acquisition"
"github.com/crowdsecurity/crowdsec/pkg/csconfig"
"github.com/crowdsecurity/crowdsec/pkg/cwhub"
leaky "github.com/crowdsecurity/crowdsec/pkg/leakybucket"
"github.com/crowdsecurity/crowdsec/pkg/parser"
"github.com/crowdsecurity/crowdsec/pkg/types"
"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
"gopkg.in/yaml.v2"
)
func initCrowdsec(cConfig *csconfig.Config) (*parser.Parsers, error) {
var err error
// Populate cwhub package tools
if err := cwhub.GetHubIdx(cConfig.Hub); err != nil {
return &parser.Parsers{}, fmt.Errorf("Failed to load hub index : %s", err)
}
// Start loading configs
csParsers := newParsers()
if csParsers, err = parser.LoadParsers(cConfig, csParsers); err != nil {
return &parser.Parsers{}, fmt.Errorf("Failed to load parsers: %s", err)
}
if err := LoadBuckets(cConfig); err != nil {
return &parser.Parsers{}, fmt.Errorf("Failed to load scenarios: %s", err)
}
if err := LoadAcquisition(cConfig); err != nil {
return &parser.Parsers{}, fmt.Errorf("Error while loading acquisition config : %s", err)
}
return csParsers, nil
}
func runCrowdsec(cConfig *csconfig.Config, parsers *parser.Parsers) error {
inputLineChan := make(chan types.Event)
inputEventChan := make(chan types.Event)
//start go-routines for parsing, buckets pour and outputs.
parserWg := &sync.WaitGroup{}
parsersTomb.Go(func() error {
parserWg.Add(1)
for i := 0; i < cConfig.Crowdsec.ParserRoutinesCount; i++ {
parsersTomb.Go(func() error {
defer types.CatchPanic("crowdsec/runParse")
if err := runParse(inputLineChan, inputEventChan, *parsers.Ctx, parsers.Nodes); err != nil { //this error will never happen as parser.Parse is not able to return errors
log.Fatalf("starting parse error : %s", err)
return err
}
return nil
})
}
parserWg.Done()
return nil
})
parserWg.Wait()
bucketWg := &sync.WaitGroup{}
bucketsTomb.Go(func() error {
bucketWg.Add(1)
/*restore previous state as well if present*/
if cConfig.Crowdsec.BucketStateFile != "" {
log.Warningf("Restoring buckets state from %s", cConfig.Crowdsec.BucketStateFile)
if err := leaky.LoadBucketsState(cConfig.Crowdsec.BucketStateFile, buckets, holders); err != nil {
return fmt.Errorf("unable to restore buckets : %s", err)
}
}
for i := 0; i < cConfig.Crowdsec.BucketsRoutinesCount; i++ {
bucketsTomb.Go(func() error {
defer types.CatchPanic("crowdsec/runPour")
if err := runPour(inputEventChan, holders, buckets, cConfig); err != nil {
log.Fatalf("starting pour error : %s", err)
return err
}
return nil
})
}
bucketWg.Done()
return nil
})
bucketWg.Wait()
outputWg := &sync.WaitGroup{}
outputsTomb.Go(func() error {
outputWg.Add(1)
for i := 0; i < cConfig.Crowdsec.OutputRoutinesCount; i++ {
outputsTomb.Go(func() error {
defer types.CatchPanic("crowdsec/runOutput")
if err := runOutput(inputEventChan, outputEventChan, buckets, *parsers.Povfwctx, parsers.Povfwnodes, *cConfig.API.Client.Credentials); err != nil {
log.Fatalf("starting outputs error : %s", err)
return err
}
return nil
})
}
outputWg.Done()
return nil
})
outputWg.Wait()
if cConfig.Prometheus != nil && cConfig.Prometheus.Enabled {
aggregated := false
if cConfig.Prometheus.Level == "aggregated" {
aggregated = true
}
if err := acquisition.GetMetrics(dataSources, aggregated); err != nil {
return errors.Wrap(err, "while fetching prometheus metrics for datasources.")
}
}
log.Info("Starting processing data")
if err := acquisition.StartAcquisition(dataSources, inputLineChan, &acquisTomb); err != nil {
log.Fatalf("starting acquisition error : %s", err)
return err
}
return nil
}
func serveCrowdsec(parsers *parser.Parsers, cConfig *csconfig.Config, agentReady chan bool) {
crowdsecTomb.Go(func() error {
defer types.CatchPanic("crowdsec/serveCrowdsec")
go func() {
defer types.CatchPanic("crowdsec/runCrowdsec")
// this logs every time, even at config reload
log.Debugf("running agent after %s ms", time.Since(crowdsecT0))
agentReady <- true
if err := runCrowdsec(cConfig, parsers); err != nil {
log.Fatalf("unable to start crowdsec routines: %s", err)
}
}()
/*we should stop in two cases :
- crowdsecTomb has been Killed() : it might be shutdown or reload, so stop
- acquisTomb is dead, it means that we were in "cat" mode and files are done reading, quit
*/
waitOnTomb()
log.Debugf("Shutting down crowdsec routines")
if err := ShutdownCrowdsecRoutines(); err != nil {
log.Fatalf("unable to shutdown crowdsec routines: %s", err)
}
log.Debugf("everything is dead, return crowdsecTomb")
if dumpStates {
dumpParserState()
dumpOverflowState()
dumpBucketsPour()
os.Exit(0)
}
return nil
})
}
func dumpBucketsPour() {
fd, err := os.OpenFile(filepath.Join(parser.DumpFolder, "bucketpour-dump.yaml"), os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0666)
if err != nil {
log.Fatalf("open: %s", err)
}
out, err := yaml.Marshal(leaky.BucketPourCache)
if err != nil {
log.Fatalf("marshal: %s", err)
}
b, err := fd.Write(out)
if err != nil {
log.Fatalf("write: %s", err)
}
log.Tracef("wrote %d bytes", b)
if err := fd.Close(); err != nil {
log.Fatalf(" close: %s", err)
}
}
func dumpParserState() {
fd, err := os.OpenFile(filepath.Join(parser.DumpFolder, "parser-dump.yaml"), os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0666)
if err != nil {
log.Fatalf("open: %s", err)
}
out, err := yaml.Marshal(parser.StageParseCache)
if err != nil {
log.Fatalf("marshal: %s", err)
}
b, err := fd.Write(out)
if err != nil {
log.Fatalf("write: %s", err)
}
log.Tracef("wrote %d bytes", b)
if err := fd.Close(); err != nil {
log.Fatalf(" close: %s", err)
}
}
func dumpOverflowState() {
fd, err := os.OpenFile(filepath.Join(parser.DumpFolder, "bucket-dump.yaml"), os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0666)
if err != nil {
log.Fatalf("open: %s", err)
}
out, err := yaml.Marshal(bucketOverflows)
if err != nil {
log.Fatalf("marshal: %s", err)
}
b, err := fd.Write(out)
if err != nil {
log.Fatalf("write: %s", err)
}
log.Tracef("wrote %d bytes", b)
if err := fd.Close(); err != nil {
log.Fatalf(" close: %s", err)
}
}
func waitOnTomb() {
for {
select {
case <-acquisTomb.Dead():
/*if it's acquisition dying it means that we were in "cat" mode.
while shutting down, we need to give time for all buckets to process in flight data*/
log.Warning("Acquisition is finished, shutting down")
/*
While it might make sense to want to shut-down parser/buckets/etc. as soon as acquisition is finished,
we might have some pending buckets: buckets that overflowed, but whose LeakRoutine are still alive because they
are waiting to be able to "commit" (push to api). This can happen specifically in a context where a lot of logs
are going to trigger overflow (ie. trigger buckets with ~100% of the logs triggering an overflow).
To avoid this (which would mean that we would "lose" some overflows), let's monitor the number of live buckets.
However, because of the blackhole mechanism, we can't really wait for the number of LeakRoutine to go to zero
(we might have to wait $blackhole_duration).
So: we are waiting for the number of buckets to stop decreasing before returning. "how long" we should wait
is a bit of the trick question, as some operations (ie. reverse dns or such in post-overflow) can take some time :)
*/
return
case <-crowdsecTomb.Dying():
log.Infof("Crowdsec engine shutting down")
return
}
}
}
crowdsec-1.4.6/cmd/crowdsec/event_log_hook_windows.go 0000664 0000000 0000000 00000001437 14371172257 0023030 0 ustar 00root root 0000000 0000000 package main
import (
log "github.com/sirupsen/logrus"
"golang.org/x/sys/windows/svc/eventlog"
)
type EventLogHook struct {
LogLevels []log.Level
evtlog *eventlog.Log
}
func (e *EventLogHook) Fire(entry *log.Entry) error {
line, err := entry.String()
if err != nil {
return err
}
switch entry.Level {
case log.PanicLevel:
return e.evtlog.Error(300, line)
case log.FatalLevel:
return e.evtlog.Error(301, line)
case log.ErrorLevel:
return e.evtlog.Error(302, line)
case log.WarnLevel:
return e.evtlog.Warning(303, line)
case log.InfoLevel:
return e.evtlog.Info(304, line)
case log.DebugLevel:
return e.evtlog.Info(305, line)
case log.TraceLevel:
return e.evtlog.Info(306, line)
}
return nil
}
func (e *EventLogHook) Levels() []log.Level {
return e.LogLevels
}
crowdsec-1.4.6/cmd/crowdsec/main.go 0000664 0000000 0000000 00000022162 14371172257 0017176 0 ustar 00root root 0000000 0000000 package main
import (
"flag"
"fmt"
_ "net/http/pprof"
"os"
"runtime"
"sort"
"strings"
"time"
"github.com/confluentinc/bincover"
"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
"gopkg.in/tomb.v2"
"github.com/crowdsecurity/crowdsec/pkg/acquisition"
"github.com/crowdsecurity/crowdsec/pkg/csconfig"
"github.com/crowdsecurity/crowdsec/pkg/csplugin"
"github.com/crowdsecurity/crowdsec/pkg/cwhub"
"github.com/crowdsecurity/crowdsec/pkg/cwversion"
"github.com/crowdsecurity/crowdsec/pkg/leakybucket"
"github.com/crowdsecurity/crowdsec/pkg/parser"
"github.com/crowdsecurity/crowdsec/pkg/types"
)
var (
/*tombs for the parser, buckets and outputs.*/
acquisTomb tomb.Tomb
parsersTomb tomb.Tomb
bucketsTomb tomb.Tomb
outputsTomb tomb.Tomb
apiTomb tomb.Tomb
crowdsecTomb tomb.Tomb
pluginTomb tomb.Tomb
flags *Flags
/*the state of acquisition*/
dataSources []acquisition.DataSource
/*the state of the buckets*/
holders []leakybucket.BucketFactory
buckets *leakybucket.Buckets
outputEventChan chan types.Event //the buckets init returns its own chan that is used for multiplexing
/*settings*/
lastProcessedItem time.Time /*keep track of last item timestamp in time-machine. it is used to GC buckets when we dump them.*/
pluginBroker csplugin.PluginBroker
)
var bincoverTesting = ""
type Flags struct {
ConfigFile string
TraceLevel bool
DebugLevel bool
InfoLevel bool
WarnLevel bool
PrintVersion bool
SingleFileType string
Labels map[string]string
OneShotDSN string
TestMode bool
DisableAgent bool
DisableAPI bool
WinSvc string
DisableCAPI bool
}
type labelsMap map[string]string
// Return new parsers
// nodes and povfwnodes are already initialized in parser.LoadStages
func newParsers() *parser.Parsers {
parsers := &parser.Parsers{
Ctx: &parser.UnixParserCtx{},
Povfwctx: &parser.UnixParserCtx{},
StageFiles: make([]parser.Stagefile, 0),
PovfwStageFiles: make([]parser.Stagefile, 0),
}
for _, itemType := range []string{cwhub.PARSERS, cwhub.PARSERS_OVFLW} {
for _, hubParserItem := range cwhub.GetItemMap(itemType) {
if hubParserItem.Installed {
stagefile := parser.Stagefile{
Filename: hubParserItem.LocalPath,
Stage: hubParserItem.Stage,
}
if itemType == cwhub.PARSERS {
parsers.StageFiles = append(parsers.StageFiles, stagefile)
}
if itemType == cwhub.PARSERS_OVFLW {
parsers.PovfwStageFiles = append(parsers.PovfwStageFiles, stagefile)
}
}
}
}
if parsers.StageFiles != nil {
sort.Slice(parsers.StageFiles, func(i, j int) bool {
return parsers.StageFiles[i].Filename < parsers.StageFiles[j].Filename
})
}
if parsers.PovfwStageFiles != nil {
sort.Slice(parsers.PovfwStageFiles, func(i, j int) bool {
return parsers.PovfwStageFiles[i].Filename < parsers.PovfwStageFiles[j].Filename
})
}
return parsers
}
func LoadBuckets(cConfig *csconfig.Config) error {
var (
err error
files []string
)
for _, hubScenarioItem := range cwhub.GetItemMap(cwhub.SCENARIOS) {
if hubScenarioItem.Installed {
files = append(files, hubScenarioItem.LocalPath)
}
}
buckets = leakybucket.NewBuckets()
log.Infof("Loading %d scenario files", len(files))
holders, outputEventChan, err = leakybucket.LoadBuckets(cConfig.Crowdsec, files, &bucketsTomb, buckets)
if err != nil {
return fmt.Errorf("scenario loading failed: %v", err)
}
if cConfig.Prometheus != nil && cConfig.Prometheus.Enabled {
for holderIndex := range holders {
holders[holderIndex].Profiling = true
}
}
return nil
}
func LoadAcquisition(cConfig *csconfig.Config) error {
var err error
if flags.SingleFileType != "" || flags.OneShotDSN != "" {
if flags.OneShotDSN == "" || flags.SingleFileType == "" {
return fmt.Errorf("-type requires a -dsn argument")
}
flags.Labels = labels
flags.Labels["type"] = flags.SingleFileType
dataSources, err = acquisition.LoadAcquisitionFromDSN(flags.OneShotDSN, flags.Labels)
if err != nil {
return errors.Wrapf(err, "failed to configure datasource for %s", flags.OneShotDSN)
}
} else {
dataSources, err = acquisition.LoadAcquisitionFromFile(cConfig.Crowdsec)
if err != nil {
return err
}
}
return nil
}
var dumpFolder string
var dumpStates bool
var labels = make(labelsMap)
func (l *labelsMap) String() string {
return "labels"
}
func (l labelsMap) Set(label string) error {
split := strings.Split(label, ":")
if len(split) != 2 {
return errors.Wrapf(errors.New("Bad Format"), "for Label '%s'", label)
}
l[split[0]] = split[1]
return nil
}
func (f *Flags) Parse() {
flag.StringVar(&f.ConfigFile, "c", csconfig.DefaultConfigPath("config.yaml"), "configuration file")
flag.BoolVar(&f.TraceLevel, "trace", false, "VERY verbose")
flag.BoolVar(&f.DebugLevel, "debug", false, "print debug-level on stderr")
flag.BoolVar(&f.InfoLevel, "info", false, "print info-level on stderr")
flag.BoolVar(&f.WarnLevel, "warning", false, "print warning-level on stderr")
flag.BoolVar(&f.PrintVersion, "version", false, "display version")
flag.StringVar(&f.OneShotDSN, "dsn", "", "Process a single data source in time-machine")
flag.StringVar(&f.SingleFileType, "type", "", "Labels.type for file in time-machine")
flag.Var(&labels, "label", "Additional Labels for file in time-machine")
flag.BoolVar(&f.TestMode, "t", false, "only test configs")
flag.BoolVar(&f.DisableAgent, "no-cs", false, "disable crowdsec agent")
flag.BoolVar(&f.DisableAPI, "no-api", false, "disable local API")
flag.BoolVar(&f.DisableCAPI, "no-capi", false, "disable communication with Central API")
flag.StringVar(&f.WinSvc, "winsvc", "", "Windows service Action : Install, Remove etc..")
flag.StringVar(&dumpFolder, "dump-data", "", "dump parsers/buckets raw outputs")
flag.Parse()
}
// LoadConfig returns a configuration parsed from configuration file
func LoadConfig(cConfig *csconfig.Config) error {
if dumpFolder != "" {
parser.ParseDump = true
parser.DumpFolder = dumpFolder
leakybucket.BucketPourTrack = true
dumpStates = true
}
if !flags.DisableAgent {
if err := cConfig.LoadCrowdsec(); err != nil {
return err
}
}
if !flags.DisableAPI {
if err := cConfig.LoadAPIServer(); err != nil {
return err
}
}
if !cConfig.DisableAgent && (cConfig.API == nil || cConfig.API.Client == nil || cConfig.API.Client.Credentials == nil) {
return errors.New("missing local API credentials for crowdsec agent, abort")
}
if cConfig.DisableAPI && cConfig.DisableAgent {
return errors.New("You must run at least the API Server or crowdsec")
}
if flags.WarnLevel {
logLevel := log.WarnLevel
cConfig.Common.LogLevel = &logLevel
}
if flags.InfoLevel || cConfig.Common.LogLevel == nil {
logLevel := log.InfoLevel
cConfig.Common.LogLevel = &logLevel
}
if flags.DebugLevel {
logLevel := log.DebugLevel
cConfig.Common.LogLevel = &logLevel
}
if flags.TraceLevel {
logLevel := log.TraceLevel
cConfig.Common.LogLevel = &logLevel
}
if flags.TestMode && !cConfig.DisableAgent {
cConfig.Crowdsec.LintOnly = true
}
if flags.SingleFileType != "" && flags.OneShotDSN != "" {
if cConfig.API != nil && cConfig.API.Server != nil {
cConfig.API.Server.OnlineClient = nil
}
/*if the api is disabled as well, just read file and exit, don't daemonize*/
if flags.DisableAPI {
cConfig.Common.Daemonize = false
}
cConfig.Common.LogMedia = "stdout"
log.Infof("single file mode : log_media=%s daemonize=%t", cConfig.Common.LogMedia, cConfig.Common.Daemonize)
}
if cConfig.Common.PidDir != "" {
log.Warn("Deprecation warning: the pid_dir config can be safely removed and is not required")
}
if cConfig.Common.Daemonize && runtime.GOOS == "windows" {
log.Debug("Daemonization is not supported on Windows, disabling")
cConfig.Common.Daemonize = false
}
return nil
}
// exitWithCode must be called right before the program termination,
// to allow measuring functional test coverage in case of abnormal exit.
//
// without bincover: log error and exit with code
// with bincover: log error and tell bincover the exit code, then return
func exitWithCode(exitCode int, err error) {
if err != nil {
// this method of logging a fatal error does not
// trigger a program exit (as stated by the authors, it
// is not going to change in logrus to keep backward
// compatibility), and allows us to report coverage.
log.NewEntry(log.StandardLogger()).Log(log.FatalLevel, err)
}
if bincoverTesting == "" {
os.Exit(exitCode)
}
bincover.ExitCode = exitCode
}
// crowdsecT0 can be used to measure start time of services,
// or uptime of the application
var crowdsecT0 time.Time
func main() {
crowdsecT0 = time.Now()
defer types.CatchPanic("crowdsec/main")
log.Debugf("os.Args: %v", os.Args)
// Handle command line arguments
flags = &Flags{}
flags.Parse()
if len(flag.Args()) > 0 {
fmt.Fprintf(os.Stderr, "argument provided but not defined: %s\n", flag.Args()[0])
flag.Usage()
// the flag package exits with 2 in case of unknown flag
exitWithCode(2, nil)
return
}
if flags.PrintVersion {
cwversion.Show()
exitWithCode(0, nil)
return
}
exitCode := 0
err := StartRunSvc()
if err != nil {
exitCode = 1
}
exitWithCode(exitCode, err)
}
crowdsec-1.4.6/cmd/crowdsec/main_test.go 0000664 0000000 0000000 00000000245 14371172257 0020233 0 ustar 00root root 0000000 0000000 //go:build testrunmain
package main
import (
"github.com/confluentinc/bincover"
"testing"
)
func TestBincoverRunMain(t *testing.T) {
bincover.RunTest(main)
}
crowdsec-1.4.6/cmd/crowdsec/metrics.go 0000664 0000000 0000000 00000013215 14371172257 0017717 0 ustar 00root root 0000000 0000000 package main
import (
"fmt"
"time"
v1 "github.com/crowdsecurity/crowdsec/pkg/apiserver/controllers/v1"
"github.com/crowdsecurity/crowdsec/pkg/csconfig"
"github.com/crowdsecurity/crowdsec/pkg/cwversion"
"github.com/crowdsecurity/crowdsec/pkg/database"
leaky "github.com/crowdsecurity/crowdsec/pkg/leakybucket"
"github.com/crowdsecurity/crowdsec/pkg/parser"
"github.com/crowdsecurity/crowdsec/pkg/types"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
"net/http"
log "github.com/sirupsen/logrus"
)
/*prometheus*/
var globalParserHits = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "cs_parser_hits_total",
Help: "Total events entered the parser.",
},
[]string{"source", "type"},
)
var globalParserHitsOk = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "cs_parser_hits_ok_total",
Help: "Total events were successfully parsed.",
},
[]string{"source", "type"},
)
var globalParserHitsKo = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "cs_parser_hits_ko_total",
Help: "Total events were unsuccessfully parsed.",
},
[]string{"source", "type"},
)
var globalBucketPourKo = prometheus.NewCounter(
prometheus.CounterOpts{
Name: "cs_bucket_pour_ko_total",
Help: "Total events were not poured in a bucket.",
},
)
var globalBucketPourOk = prometheus.NewCounter(
prometheus.CounterOpts{
Name: "cs_bucket_pour_ok_total",
Help: "Total events were poured in at least one bucket.",
},
)
var globalCsInfo = prometheus.NewGauge(
prometheus.GaugeOpts{
Name: "cs_info",
Help: "Information about Crowdsec.",
ConstLabels: prometheus.Labels{"version": cwversion.VersionStr()},
},
)
var globalActiveDecisions = prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Name: "cs_active_decisions",
Help: "Number of active decisions.",
},
[]string{"reason", "origin", "action"},
)
var globalAlerts = prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Name: "cs_alerts",
Help: "Number of alerts (excluding CAPI).",
},
[]string{"reason"},
)
var globalParsingHistogram = prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Help: "Time spent parsing a line",
Name: "cs_parsing_time_seconds",
Buckets: []float64{0.0005, 0.001, 0.0015, 0.002, 0.0025, 0.003, 0.004, 0.005, 0.0075, 0.01},
},
[]string{"type", "source"},
)
var globalPourHistogram = prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Name: "cs_bucket_pour_seconds",
Help: "Time spent pouring an event to buckets.",
Buckets: []float64{0.001, 0.002, 0.005, 0.01, 0.015, 0.02, 0.03, 0.04, 0.05},
},
[]string{"type", "source"},
)
func computeDynamicMetrics(next http.Handler, dbClient *database.Client) http.HandlerFunc {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if dbClient == nil {
next.ServeHTTP(w, r)
return
}
decisionsFilters := make(map[string][]string, 0)
decisions, err := dbClient.QueryDecisionCountByScenario(decisionsFilters)
if err != nil {
log.Errorf("Error querying decisions for metrics: %v", err)
next.ServeHTTP(w, r)
return
}
globalActiveDecisions.Reset()
for _, d := range decisions {
globalActiveDecisions.With(prometheus.Labels{"reason": d.Scenario, "origin": d.Origin, "action": d.Type}).Set(float64(d.Count))
}
globalAlerts.Reset()
alertsFilter := map[string][]string{
"include_capi": {"false"},
}
alerts, err := dbClient.AlertsCountPerScenario(alertsFilter)
if err != nil {
log.Errorf("Error querying alerts for metrics: %v", err)
next.ServeHTTP(w, r)
return
}
for k, v := range alerts {
globalAlerts.With(prometheus.Labels{"reason": k}).Set(float64(v))
}
next.ServeHTTP(w, r)
})
}
func registerPrometheus(config *csconfig.PrometheusCfg) {
if !config.Enabled {
return
}
if config.ListenAddr == "" {
log.Warning("prometheus is enabled, but the listen address is empty, using '127.0.0.1'")
config.ListenAddr = "127.0.0.1"
}
if config.ListenPort == 0 {
log.Warning("prometheus is enabled, but the listen port is empty, using '6060'")
config.ListenPort = 6060
}
// Registering prometheus
// If in aggregated mode, do not register events associated with a source, to keep the cardinality low
if config.Level == "aggregated" {
log.Infof("Loading aggregated prometheus collectors")
prometheus.MustRegister(globalParserHits, globalParserHitsOk, globalParserHitsKo,
globalCsInfo, globalParsingHistogram, globalPourHistogram,
leaky.BucketsUnderflow, leaky.BucketsCanceled, leaky.BucketsInstantiation, leaky.BucketsOverflow,
v1.LapiRouteHits,
leaky.BucketsCurrentCount)
} else {
log.Infof("Loading prometheus collectors")
prometheus.MustRegister(globalParserHits, globalParserHitsOk, globalParserHitsKo,
parser.NodesHits, parser.NodesHitsOk, parser.NodesHitsKo,
globalCsInfo, globalParsingHistogram, globalPourHistogram,
v1.LapiRouteHits, v1.LapiMachineHits, v1.LapiBouncerHits, v1.LapiNilDecisions, v1.LapiNonNilDecisions, v1.LapiResponseTime,
leaky.BucketsPour, leaky.BucketsUnderflow, leaky.BucketsCanceled, leaky.BucketsInstantiation, leaky.BucketsOverflow, leaky.BucketsCurrentCount,
globalActiveDecisions, globalAlerts)
}
}
func servePrometheus(config *csconfig.PrometheusCfg, dbClient *database.Client, apiReady chan bool, agentReady chan bool) {
if !config.Enabled {
return
}
defer types.CatchPanic("crowdsec/servePrometheus")
http.Handle("/metrics", computeDynamicMetrics(promhttp.Handler(), dbClient))
<-apiReady
<-agentReady
log.Debugf("serving metrics after %s ms", time.Since(crowdsecT0))
if err := http.ListenAndServe(fmt.Sprintf("%s:%d", config.ListenAddr, config.ListenPort), nil); err != nil {
log.Warningf("prometheus: %s", err)
}
}
crowdsec-1.4.6/cmd/crowdsec/output.go 0000664 0000000 0000000 00000011646 14371172257 0017617 0 ustar 00root root 0000000 0000000 package main
import (
"context"
"fmt"
"net/url"
"sync"
"time"
"github.com/crowdsecurity/crowdsec/pkg/apiclient"
"github.com/crowdsecurity/crowdsec/pkg/csconfig"
"github.com/crowdsecurity/crowdsec/pkg/cwhub"
"github.com/crowdsecurity/crowdsec/pkg/cwversion"
leaky "github.com/crowdsecurity/crowdsec/pkg/leakybucket"
"github.com/crowdsecurity/crowdsec/pkg/models"
"github.com/crowdsecurity/crowdsec/pkg/parser"
"github.com/crowdsecurity/crowdsec/pkg/types"
"github.com/go-openapi/strfmt"
"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
)
func dedupAlerts(alerts []types.RuntimeAlert) ([]*models.Alert, error) {
var dedupCache []*models.Alert
for idx, alert := range alerts {
log.Tracef("alert %d/%d", idx, len(alerts))
/*if we have more than one source, we need to dedup */
if len(alert.Sources) == 0 || len(alert.Sources) == 1 {
dedupCache = append(dedupCache, alert.Alert)
continue
}
for k, src := range alert.Sources {
refsrc := *alert.Alert //copy
log.Tracef("source[%s]", k)
refsrc.Source = &src
dedupCache = append(dedupCache, &refsrc)
}
}
if len(dedupCache) != len(alerts) {
log.Tracef("went from %d to %d alerts", len(alerts), len(dedupCache))
}
return dedupCache, nil
}
func PushAlerts(alerts []types.RuntimeAlert, client *apiclient.ApiClient) error {
ctx := context.Background()
alertsToPush, err := dedupAlerts(alerts)
if err != nil {
return errors.Wrap(err, "failed to transform alerts for api")
}
_, _, err = client.Alerts.Add(ctx, alertsToPush)
if err != nil {
return errors.Wrap(err, "failed sending alert to LAPI")
}
return nil
}
var bucketOverflows []types.Event
func runOutput(input chan types.Event, overflow chan types.Event, buckets *leaky.Buckets,
postOverflowCTX parser.UnixParserCtx, postOverflowNodes []parser.Node, apiConfig csconfig.ApiCredentialsCfg) error {
var err error
ticker := time.NewTicker(1 * time.Second)
var cache []types.RuntimeAlert
var cacheMutex sync.Mutex
scenarios, err := cwhub.GetInstalledScenariosAsString()
if err != nil {
return errors.Wrapf(err, "loading list of installed hub scenarios: %s", err)
}
apiURL, err := url.Parse(apiConfig.URL)
if err != nil {
return errors.Wrapf(err, "parsing api url ('%s'): %s", apiConfig.URL, err)
}
password := strfmt.Password(apiConfig.Password)
Client, err := apiclient.NewClient(&apiclient.Config{
MachineID: apiConfig.Login,
Password: password,
Scenarios: scenarios,
UserAgent: fmt.Sprintf("crowdsec/%s", cwversion.VersionStr()),
URL: apiURL,
VersionPrefix: "v1",
UpdateScenario: cwhub.GetInstalledScenariosAsString,
})
if err != nil {
return errors.Wrapf(err, "new client api: %s", err)
}
if _, err = Client.Auth.AuthenticateWatcher(context.Background(), models.WatcherAuthRequest{
MachineID: &apiConfig.Login,
Password: &password,
Scenarios: scenarios,
}); err != nil {
return errors.Wrapf(err, "authenticate watcher (%s)", apiConfig.Login)
}
//start the heartbeat service
log.Debugf("Starting HeartBeat service")
Client.HeartBeat.StartHeartBeat(context.Background(), &outputsTomb)
LOOP:
for {
select {
case <-ticker.C:
if len(cache) > 0 {
cacheMutex.Lock()
cachecopy := cache
newcache := make([]types.RuntimeAlert, 0)
cache = newcache
cacheMutex.Unlock()
if err := PushAlerts(cachecopy, Client); err != nil {
log.Errorf("while pushing to api : %s", err)
//just push back the events to the queue
cacheMutex.Lock()
cache = append(cache, cachecopy...)
cacheMutex.Unlock()
}
}
case <-outputsTomb.Dying():
if len(cache) > 0 {
cacheMutex.Lock()
cachecopy := cache
cacheMutex.Unlock()
if err := PushAlerts(cachecopy, Client); err != nil {
log.Errorf("while pushing leftovers to api : %s", err)
}
}
break LOOP
case event := <-overflow:
//if the Alert is nil, it's to signal bucket is ready for GC, don't track this
if dumpStates && event.Overflow.Alert != nil {
if bucketOverflows == nil {
bucketOverflows = make([]types.Event, 0)
}
bucketOverflows = append(bucketOverflows, event)
}
/*if alert is empty and mapKey is present, the overflow is just to cleanup bucket*/
if event.Overflow.Alert == nil && event.Overflow.Mapkey != "" {
buckets.Bucket_map.Delete(event.Overflow.Mapkey)
break
}
/* process post overflow parser nodes */
event, err := parser.Parse(postOverflowCTX, event, postOverflowNodes)
if err != nil {
return fmt.Errorf("postoverflow failed : %s", err)
}
log.Printf("%s", *event.Overflow.Alert.Message)
if event.Overflow.Whitelisted {
log.Printf("[%s] is whitelisted, skip.", *event.Overflow.Alert.Message)
continue
}
if event.Overflow.Reprocess {
log.Debugf("Overflow being reprocessed.")
input <- event
}
if dumpStates {
continue
}
cacheMutex.Lock()
cache = append(cache, event.Overflow)
cacheMutex.Unlock()
}
}
ticker.Stop()
return nil
}
crowdsec-1.4.6/cmd/crowdsec/parse.go 0000664 0000000 0000000 00000003031 14371172257 0017356 0 ustar 00root root 0000000 0000000 package main
import (
"time"
"github.com/prometheus/client_golang/prometheus"
log "github.com/sirupsen/logrus"
"github.com/crowdsecurity/crowdsec/pkg/parser"
"github.com/crowdsecurity/crowdsec/pkg/types"
)
func runParse(input chan types.Event, output chan types.Event, parserCTX parser.UnixParserCtx, nodes []parser.Node) error {
LOOP:
for {
select {
case <-parsersTomb.Dying():
log.Infof("Killing parser routines")
break LOOP
case event := <-input:
if !event.Process {
continue
}
if event.Line.Module == "" {
log.Errorf("empty event.Line.Module field, the acquisition module must set it ! : %+v", event.Line)
continue
}
globalParserHits.With(prometheus.Labels{"source": event.Line.Src, "type": event.Line.Module}).Inc()
startParsing := time.Now()
/* parse the log using magic */
parsed, err := parser.Parse(parserCTX, event, nodes)
if err != nil {
log.Errorf("failed parsing : %v\n", err)
}
elapsed := time.Since(startParsing)
globalParsingHistogram.With(prometheus.Labels{"source": event.Line.Src, "type": event.Line.Module}).Observe(elapsed.Seconds())
if !parsed.Process {
globalParserHitsKo.With(prometheus.Labels{"source": event.Line.Src, "type": event.Line.Module}).Inc()
log.Debugf("Discarding line %+v", parsed)
continue
}
globalParserHitsOk.With(prometheus.Labels{"source": event.Line.Src, "type": event.Line.Module}).Inc()
if parsed.Whitelisted {
log.Debugf("event whitelisted, discard")
continue
}
output <- parsed
}
}
return nil
}
crowdsec-1.4.6/cmd/crowdsec/pour.go 0000664 0000000 0000000 00000003731 14371172257 0017240 0 ustar 00root root 0000000 0000000 package main
import (
"fmt"
"time"
"github.com/crowdsecurity/crowdsec/pkg/csconfig"
leaky "github.com/crowdsecurity/crowdsec/pkg/leakybucket"
"github.com/crowdsecurity/crowdsec/pkg/types"
"github.com/prometheus/client_golang/prometheus"
log "github.com/sirupsen/logrus"
)
func runPour(input chan types.Event, holders []leaky.BucketFactory, buckets *leaky.Buckets, cConfig *csconfig.Config) error {
var (
count int
)
for {
//bucket is now ready
select {
case <-bucketsTomb.Dying():
log.Infof("Bucket routine exiting")
return nil
case parsed := <-input:
startTime := time.Now()
count++
if count%5000 == 0 {
log.Infof("%d existing buckets", leaky.LeakyRoutineCount)
//when in forensics mode, garbage collect buckets
if cConfig.Crowdsec.BucketsGCEnabled {
if parsed.MarshaledTime != "" {
var z *time.Time = &time.Time{}
if err := z.UnmarshalText([]byte(parsed.MarshaledTime)); err != nil {
log.Warningf("Failed to unmarshal time from event '%s' : %s", parsed.MarshaledTime, err)
} else {
log.Warning("Starting buckets garbage collection ...")
if err = leaky.GarbageCollectBuckets(*z, buckets); err != nil {
return fmt.Errorf("failed to start bucket GC : %s", err)
}
}
}
}
}
//here we can bucketify with parsed
poured, err := leaky.PourItemToHolders(parsed, holders, buckets)
if err != nil {
log.Errorf("bucketify failed for: %v", parsed)
return fmt.Errorf("process of event failed : %v", err)
}
elapsed := time.Since(startTime)
globalPourHistogram.With(prometheus.Labels{"type": parsed.Line.Module, "source": parsed.Line.Src}).Observe(elapsed.Seconds())
if poured {
globalBucketPourOk.Inc()
} else {
globalBucketPourKo.Inc()
}
if len(parsed.MarshaledTime) != 0 {
if err := lastProcessedItem.UnmarshalText([]byte(parsed.MarshaledTime)); err != nil {
log.Warningf("failed to unmarshal time from event : %s", err)
}
}
}
}
}
crowdsec-1.4.6/cmd/crowdsec/run_in_svc.go 0000664 0000000 0000000 00000003532 14371172257 0020417 0 ustar 00root root 0000000 0000000 //go:build linux || freebsd || netbsd || openbsd || solaris || !windows
// +build linux freebsd netbsd openbsd solaris !windows
package main
import (
"os"
"github.com/crowdsecurity/crowdsec/pkg/csconfig"
"github.com/crowdsecurity/crowdsec/pkg/cwversion"
"github.com/crowdsecurity/crowdsec/pkg/database"
"github.com/crowdsecurity/crowdsec/pkg/types"
log "github.com/sirupsen/logrus"
"github.com/sirupsen/logrus/hooks/writer"
)
func StartRunSvc() error {
var (
cConfig *csconfig.Config
err error
)
// Set a default logger with level=fatal on stderr,
// in addition to the one we configure afterwards
log.AddHook(&writer.Hook{
Writer: os.Stderr,
LogLevels: []log.Level{
log.PanicLevel,
log.FatalLevel,
},
})
cConfig, err = csconfig.NewConfig(flags.ConfigFile, flags.DisableAgent, flags.DisableAPI)
if err != nil {
return err
}
if err := LoadConfig(cConfig); err != nil {
return err
}
// Configure logging
if err = types.SetDefaultLoggerConfig(cConfig.Common.LogMedia, cConfig.Common.LogDir, *cConfig.Common.LogLevel,
cConfig.Common.LogMaxSize, cConfig.Common.LogMaxFiles, cConfig.Common.LogMaxAge, cConfig.Common.CompressLogs, cConfig.Common.ForceColorLogs); err != nil {
log.Fatal(err)
}
log.Infof("Crowdsec %s", cwversion.VersionStr())
if bincoverTesting != "" {
log.Debug("coverage report is enabled")
}
apiReady := make(chan bool, 1)
agentReady := make(chan bool, 1)
// Enable profiling early
if cConfig.Prometheus != nil {
var dbClient *database.Client
var err error
if cConfig.DbConfig != nil {
dbClient, err = database.NewClient(cConfig.DbConfig)
if err != nil {
log.Fatalf("unable to create database client: %s", err)
}
}
registerPrometheus(cConfig.Prometheus)
go servePrometheus(cConfig.Prometheus, dbClient, apiReady, agentReady)
}
return Serve(cConfig, apiReady, agentReady)
}
crowdsec-1.4.6/cmd/crowdsec/run_in_svc_windows.go 0000664 0000000 0000000 00000005201 14371172257 0022164 0 ustar 00root root 0000000 0000000 package main
import (
"fmt"
"github.com/crowdsecurity/crowdsec/pkg/csconfig"
"github.com/crowdsecurity/crowdsec/pkg/cwversion"
"github.com/crowdsecurity/crowdsec/pkg/database"
"github.com/crowdsecurity/crowdsec/pkg/types"
"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
"golang.org/x/sys/windows/svc"
)
func StartRunSvc() error {
const svcName = "CrowdSec"
const svcDescription = "Crowdsec IPS/IDS"
isRunninginService, err := svc.IsWindowsService()
if err != nil {
return errors.Wrap(err, "failed to determine if we are running in windows service mode")
}
if isRunninginService {
return runService(svcName)
}
if flags.WinSvc == "Install" {
err = installService(svcName, svcDescription)
if err != nil {
return errors.Wrapf(err, "failed to %s %s", flags.WinSvc, svcName)
}
} else if flags.WinSvc == "Remove" {
err = removeService(svcName)
if err != nil {
return errors.Wrapf(err, "failed to %s %s", flags.WinSvc, svcName)
}
} else if flags.WinSvc == "Start" {
err = startService(svcName)
if err != nil {
return errors.Wrapf(err, "failed to %s %s", flags.WinSvc, svcName)
}
} else if flags.WinSvc == "Stop" {
err = controlService(svcName, svc.Stop, svc.Stopped)
if err != nil {
return errors.Wrapf(err, "failed to %s %s", flags.WinSvc, svcName)
}
} else if flags.WinSvc == "" {
return WindowsRun()
} else {
return fmt.Errorf("Invalid value for winsvc parameter: %s", flags.WinSvc)
}
return nil
}
func WindowsRun() error {
var (
cConfig *csconfig.Config
err error
)
cConfig, err = csconfig.NewConfig(flags.ConfigFile, flags.DisableAgent, flags.DisableAPI)
if err != nil {
return err
}
if err := LoadConfig(cConfig); err != nil {
return err
}
// Configure logging
if err = types.SetDefaultLoggerConfig(cConfig.Common.LogMedia, cConfig.Common.LogDir, *cConfig.Common.LogLevel,
cConfig.Common.LogMaxSize, cConfig.Common.LogMaxFiles, cConfig.Common.LogMaxAge, cConfig.Common.CompressLogs, cConfig.Common.ForceColorLogs); err != nil {
return err
}
log.Infof("Crowdsec %s", cwversion.VersionStr())
if bincoverTesting != "" {
log.Debug("coverage report is enabled")
}
apiReady := make(chan bool, 1)
agentReady := make(chan bool, 1)
// Enable profiling early
if cConfig.Prometheus != nil {
var dbClient *database.Client
var err error
if cConfig.DbConfig != nil {
dbClient, err = database.NewClient(cConfig.DbConfig)
if err != nil {
log.Fatalf("unable to create database client: %s", err)
}
}
registerPrometheus(cConfig.Prometheus)
go servePrometheus(cConfig.Prometheus, dbClient, apiReady, agentReady)
}
return Serve(cConfig, apiReady, agentReady)
}
crowdsec-1.4.6/cmd/crowdsec/serve.go 0000664 0000000 0000000 00000020605 14371172257 0017376 0 ustar 00root root 0000000 0000000 package main
import (
"os"
"os/signal"
"syscall"
"time"
"github.com/coreos/go-systemd/daemon"
"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
"gopkg.in/tomb.v2"
"github.com/crowdsecurity/crowdsec/pkg/csconfig"
"github.com/crowdsecurity/crowdsec/pkg/database"
"github.com/crowdsecurity/crowdsec/pkg/exprhelpers"
leaky "github.com/crowdsecurity/crowdsec/pkg/leakybucket"
"github.com/crowdsecurity/crowdsec/pkg/types"
)
//nolint: deadcode,unused // debugHandler is kept as a dev convenience: it shuts down and serialize internal state
func debugHandler(sig os.Signal, cConfig *csconfig.Config) error {
var (
tmpFile string
err error
)
// stop goroutines
if err = ShutdownCrowdsecRoutines(); err != nil {
log.Warningf("Failed to shut down routines: %s", err)
}
// todo: properly stop acquis with the tail readers
if tmpFile, err = leaky.DumpBucketsStateAt(time.Now().UTC(), cConfig.Crowdsec.BucketStateDumpDir, buckets); err != nil {
log.Warningf("Failed to dump bucket state : %s", err)
}
if err := leaky.ShutdownAllBuckets(buckets); err != nil {
log.Warningf("Failed to shut down routines : %s", err)
}
log.Printf("Shutdown is finished, buckets are in %s", tmpFile)
return nil
}
func reloadHandler(sig os.Signal) (*csconfig.Config, error) {
var tmpFile string
// re-initialize tombs
acquisTomb = tomb.Tomb{}
parsersTomb = tomb.Tomb{}
bucketsTomb = tomb.Tomb{}
outputsTomb = tomb.Tomb{}
apiTomb = tomb.Tomb{}
crowdsecTomb = tomb.Tomb{}
pluginTomb = tomb.Tomb{}
cConfig, err := csconfig.NewConfig(flags.ConfigFile, flags.DisableAgent, flags.DisableAPI)
if err != nil {
return nil, err
}
if err = LoadConfig(cConfig); err != nil {
return nil, err
}
// Configure logging
if err = types.SetDefaultLoggerConfig(cConfig.Common.LogMedia,
cConfig.Common.LogDir, *cConfig.Common.LogLevel,
cConfig.Common.LogMaxSize, cConfig.Common.LogMaxFiles,
cConfig.Common.LogMaxAge, cConfig.Common.CompressLogs,
cConfig.Common.ForceColorLogs); err != nil {
return nil, err
}
if !cConfig.DisableAPI {
if flags.DisableCAPI {
log.Warningf("Communication with CrowdSec Central API disabled from args")
cConfig.API.Server.OnlineClient = nil
}
apiServer, err := initAPIServer(cConfig)
if err != nil {
return nil, errors.Wrap(err, "unable to init api server")
}
apiReady := make(chan bool, 1)
serveAPIServer(apiServer, apiReady)
}
if !cConfig.DisableAgent {
csParsers, err := initCrowdsec(cConfig)
if err != nil {
return nil, errors.Wrap(err, "unable to init crowdsec")
}
// restore bucket state
if tmpFile != "" {
log.Warningf("we are now using %s as a state file", tmpFile)
cConfig.Crowdsec.BucketStateFile = tmpFile
}
// reload the simulation state
if err := cConfig.LoadSimulation(); err != nil {
log.Errorf("reload error (simulation) : %s", err)
}
agentReady := make(chan bool, 1)
serveCrowdsec(csParsers, cConfig, agentReady)
}
log.Printf("Reload is finished")
// delete the tmp file, it's safe now :)
if tmpFile != "" {
if err := os.Remove(tmpFile); err != nil {
log.Warningf("Failed to delete temp file (%s) : %s", tmpFile, err)
}
}
return cConfig, nil
}
func ShutdownCrowdsecRoutines() error {
var reterr error
log.Debugf("Shutting down crowdsec sub-routines")
if len(dataSources) > 0 {
acquisTomb.Kill(nil)
log.Debugf("waiting for acquisition to finish")
if err := acquisTomb.Wait(); err != nil {
log.Warningf("Acquisition returned error : %s", err)
reterr = err
}
}
log.Debugf("acquisition is finished, wait for parser/bucket/ouputs.")
parsersTomb.Kill(nil)
if err := parsersTomb.Wait(); err != nil {
log.Warningf("Parsers returned error : %s", err)
reterr = err
}
log.Debugf("parsers is done")
time.Sleep(1 * time.Second) // ugly workaround for now to ensure PourItemtoholders are finished
bucketsTomb.Kill(nil)
if err := bucketsTomb.Wait(); err != nil {
log.Warningf("Buckets returned error : %s", err)
reterr = err
}
log.Debugf("buckets is done")
time.Sleep(1 * time.Second) // ugly workaround for now
outputsTomb.Kill(nil)
if err := outputsTomb.Wait(); err != nil {
log.Warningf("Ouputs returned error : %s", err)
reterr = err
}
log.Debugf("outputs are done")
// He's dead, Jim.
crowdsecTomb.Kill(nil)
return reterr
}
func shutdownAPI() error {
log.Debugf("shutting down api via Tomb")
apiTomb.Kill(nil)
if err := apiTomb.Wait(); err != nil {
return err
}
log.Debugf("done")
return nil
}
func shutdownCrowdsec() error {
log.Debugf("shutting down crowdsec via Tomb")
crowdsecTomb.Kill(nil)
if err := crowdsecTomb.Wait(); err != nil {
return err
}
log.Debugf("done")
return nil
}
func shutdown(sig os.Signal, cConfig *csconfig.Config) error {
if !cConfig.DisableAgent {
if err := shutdownCrowdsec(); err != nil {
return errors.Wrap(err, "failed to shut down crowdsec")
}
}
if !cConfig.DisableAPI {
if err := shutdownAPI(); err != nil {
return errors.Wrap(err, "failed to shut down api routines")
}
}
return nil
}
func HandleSignals(cConfig *csconfig.Config) error {
var (
newConfig *csconfig.Config
err error
)
signalChan := make(chan os.Signal, 1)
// We add os.Interrupt mostly to ease windows development,
// it allows to simulate a clean shutdown when running in the console
signal.Notify(signalChan,
syscall.SIGHUP,
syscall.SIGTERM,
os.Interrupt)
exitChan := make(chan error)
go func() {
defer types.CatchPanic("crowdsec/HandleSignals")
Loop:
for {
s := <-signalChan
switch s {
// kill -SIGHUP XXXX
case syscall.SIGHUP:
log.Warning("SIGHUP received, reloading")
if err = shutdown(s, cConfig); err != nil {
exitChan <- errors.Wrap(err, "failed shutdown")
break Loop
}
if newConfig, err = reloadHandler(s); err != nil {
exitChan <- errors.Wrap(err, "reload handler failure")
break Loop
}
if newConfig != nil {
cConfig = newConfig
}
// ctrl+C, kill -SIGINT XXXX, kill -SIGTERM XXXX
case os.Interrupt, syscall.SIGTERM:
log.Warning("SIGTERM received, shutting down")
if err = shutdown(s, cConfig); err != nil {
exitChan <- errors.Wrap(err, "failed shutdown")
break Loop
}
exitChan <- nil
}
}
}()
err = <-exitChan
if err == nil {
log.Warning("Crowdsec service shutting down")
}
return err
}
func Serve(cConfig *csconfig.Config, apiReady chan bool, agentReady chan bool) error {
acquisTomb = tomb.Tomb{}
parsersTomb = tomb.Tomb{}
bucketsTomb = tomb.Tomb{}
outputsTomb = tomb.Tomb{}
apiTomb = tomb.Tomb{}
crowdsecTomb = tomb.Tomb{}
pluginTomb = tomb.Tomb{}
if cConfig.API.Server != nil && cConfig.API.Server.DbConfig != nil {
dbClient, err := database.NewClient(cConfig.API.Server.DbConfig)
if err != nil {
return errors.Wrap(err, "failed to get database client")
}
err = exprhelpers.Init(dbClient)
if err != nil {
return errors.Wrap(err, "failed to init expr helpers")
}
} else {
err := exprhelpers.Init(nil)
if err != nil {
return errors.Wrap(err, "failed to init expr helpers")
}
log.Warningln("Exprhelpers loaded without database client.")
}
if !cConfig.DisableAPI {
if cConfig.API.Server.OnlineClient == nil || cConfig.API.Server.OnlineClient.Credentials == nil {
log.Warningf("Communication with CrowdSec Central API disabled from configuration file")
}
if flags.DisableCAPI {
log.Warningf("Communication with CrowdSec Central API disabled from args")
cConfig.API.Server.OnlineClient = nil
}
apiServer, err := initAPIServer(cConfig)
if err != nil {
return errors.Wrap(err, "api server init")
}
if !flags.TestMode {
serveAPIServer(apiServer, apiReady)
}
} else {
apiReady <- true
}
if !cConfig.DisableAgent {
csParsers, err := initCrowdsec(cConfig)
if err != nil {
return errors.Wrap(err, "crowdsec init")
}
// if it's just linting, we're done
if !flags.TestMode {
serveCrowdsec(csParsers, cConfig, agentReady)
}
} else {
agentReady <- true
}
if flags.TestMode {
log.Infof("test done")
pluginBroker.Kill()
os.Exit(0)
}
if cConfig.Common != nil && cConfig.Common.Daemonize {
sent, err := daemon.SdNotify(false, daemon.SdNotifyReady)
if !sent || err != nil {
log.Errorf("Failed to notify(sent: %v): %v", sent, err)
}
// wait for signals
return HandleSignals(cConfig)
}
for {
select {
case <-apiTomb.Dead():
log.Infof("api shutdown")
return nil
case <-crowdsecTomb.Dead():
log.Infof("crowdsec shutdown")
return nil
}
}
}
crowdsec-1.4.6/cmd/crowdsec/win_service.go 0000664 0000000 0000000 00000006672 14371172257 0020577 0 ustar 00root root 0000000 0000000 // Copyright 2012 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
//go:build windows
// +build windows
package main
import (
"syscall"
"time"
"github.com/crowdsecurity/crowdsec/pkg/csconfig"
"github.com/crowdsecurity/crowdsec/pkg/types"
"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
"golang.org/x/sys/windows"
"golang.org/x/sys/windows/svc"
"golang.org/x/sys/windows/svc/eventlog"
)
type crowdsec_winservice struct {
config *csconfig.Config
}
func (m *crowdsec_winservice) Execute(args []string, r <-chan svc.ChangeRequest, changes chan<- svc.Status) (ssec bool, errno uint32) {
const cmdsAccepted = svc.AcceptStop | svc.AcceptShutdown
changes <- svc.Status{State: svc.StartPending}
tick := time.Tick(500 * time.Millisecond)
changes <- svc.Status{State: svc.Running, Accepts: cmdsAccepted}
go func() {
loop:
for {
select {
case <-tick:
case c := <-r:
switch c.Cmd {
case svc.Interrogate:
changes <- c.CurrentStatus
case svc.Stop, svc.Shutdown:
changes <- svc.Status{State: svc.StopPending}
err := shutdown(nil, m.config)
if err != nil {
log.Errorf("Error while shutting down: %s", err)
//don't return, we still want to notify windows that we are stopped ?
}
break loop
default:
log.Errorf("unexpected control request #%d", c)
}
}
}
}()
err := WindowsRun()
changes <- svc.Status{State: svc.Stopped}
if err != nil {
log.Fatal(err)
}
return
}
func runService(name string) error {
//All the calls to logging before the logger is configured are pretty much useless, but we keep them for clarity
err := eventlog.InstallAsEventCreate("CrowdSec", eventlog.Error|eventlog.Warning|eventlog.Info)
if err != nil {
if errno, ok := err.(syscall.Errno); ok {
if errno == windows.ERROR_ACCESS_DENIED {
log.Warnf("Access denied when installing event source, running as non-admin ?")
} else {
log.Warnf("Failed to install event log: %s (%d)", err, errno)
}
} else {
log.Warnf("Failed to install event log: %s", err)
}
}
//Let's use our source even if we could not install it:
// - It could have been created earlier
// - No permission to create it (e.g. running as non-admin when working on crowdsec)
//It will still work, windows will just display some additional errors in the event log
evtlog, err := eventlog.Open("CrowdSec")
if err == nil {
//Send panic and fatal to event log, as they can happen before the logger is configured.
log.AddHook(&EventLogHook{
LogLevels: []log.Level{
log.PanicLevel,
log.FatalLevel,
},
evtlog: evtlog,
})
} else {
log.Warnf("Failed to open event log: %s", err)
}
cConfig, err := csconfig.NewConfig(flags.ConfigFile, flags.DisableAgent, flags.DisableAPI)
if err != nil {
return err
}
if err := LoadConfig(cConfig); err != nil {
return err
}
// Configure logging
if err := types.SetDefaultLoggerConfig(cConfig.Common.LogMedia, cConfig.Common.LogDir, *cConfig.Common.LogLevel,
cConfig.Common.LogMaxSize, cConfig.Common.LogMaxFiles, cConfig.Common.LogMaxAge, cConfig.Common.CompressLogs, cConfig.Common.ForceColorLogs); err != nil {
return err
}
log.Infof("starting %s service", name)
winsvc := crowdsec_winservice{config: cConfig}
if err := svc.Run(name, &winsvc); err != nil {
return errors.Wrapf(err, "%s service failed", name)
}
log.Infof("%s service stopped", name)
return nil
}
crowdsec-1.4.6/cmd/crowdsec/win_service_install.go 0000664 0000000 0000000 00000003501 14371172257 0022311 0 ustar 00root root 0000000 0000000 // Copyright 2012 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
//go:build windows
// +build windows
package main
import (
"fmt"
"os"
"path/filepath"
"golang.org/x/sys/windows/svc/mgr"
)
func exePath() (string, error) {
var err error
prog := os.Args[0]
p, err := filepath.Abs(prog)
if err != nil {
return "", err
}
fi, err := os.Stat(p)
if err == nil {
if !fi.Mode().IsDir() {
return p, nil
}
err = fmt.Errorf("%s is directory", p)
}
if filepath.Ext(p) == "" {
var fi os.FileInfo
p += ".exe"
fi, err = os.Stat(p)
if err == nil {
if !fi.Mode().IsDir() {
return p, nil
}
err = fmt.Errorf("%s is directory", p)
}
}
return "", err
}
func installService(name, desc string) error {
exepath, err := exePath()
if err != nil {
return err
}
m, err := mgr.Connect()
if err != nil {
return err
}
defer m.Disconnect()
s, err := m.OpenService(name)
if err == nil {
s.Close()
return fmt.Errorf("service %s already exists", name)
}
s, err = m.CreateService(name, exepath, mgr.Config{DisplayName: desc}, "is", "auto-started")
if err != nil {
return err
}
defer s.Close()
/*err = eventlog.InstallAsEventCreate(name, eventlog.Error|eventlog.Warning|eventlog.Info)
if err != nil {
s.Delete()
return fmt.Errorf("SetupEventLogSource() failed: %s", err)
}*/
return nil
}
func removeService(name string) error {
m, err := mgr.Connect()
if err != nil {
return err
}
defer m.Disconnect()
s, err := m.OpenService(name)
if err != nil {
return fmt.Errorf("service %s is not installed", name)
}
defer s.Close()
err = s.Delete()
if err != nil {
return err
}
/*err = eventlog.Remove(name)
if err != nil {
return fmt.Errorf("RemoveEventLogSource() failed: %s", err)
}*/
return nil
}
crowdsec-1.4.6/cmd/crowdsec/win_service_manage.go 0000664 0000000 0000000 00000002577 14371172257 0022107 0 ustar 00root root 0000000 0000000 // Copyright 2012 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// +build windows
package main
import (
"fmt"
"time"
//"time"
"golang.org/x/sys/windows/svc"
"golang.org/x/sys/windows/svc/mgr"
)
func startService(name string) error {
m, err := mgr.Connect()
if err != nil {
return err
}
defer m.Disconnect()
s, err := m.OpenService(name)
if err != nil {
return fmt.Errorf("could not access service: %v", err)
}
defer s.Close()
err = s.Start("is", "manual-started")
if err != nil {
return fmt.Errorf("could not start service: %v", err)
}
return nil
}
func controlService(name string, c svc.Cmd, to svc.State) error {
m, err := mgr.Connect()
if err != nil {
return err
}
defer m.Disconnect()
s, err := m.OpenService(name)
if err != nil {
return fmt.Errorf("could not access service: %v", err)
}
defer s.Close()
status, err := s.Control(c)
if err != nil {
return fmt.Errorf("could not send control=%d: %v", c, err)
}
timeout := time.Now().Add(10 * time.Second)
for status.State != to {
if timeout.Before(time.Now()) {
return fmt.Errorf("timeout waiting for service to go to state=%d", to)
}
time.Sleep(300 * time.Millisecond)
status, err = s.Query()
if err != nil {
return fmt.Errorf("could not retrieve service status: %v", err)
}
}
return nil
}
crowdsec-1.4.6/config/ 0000775 0000000 0000000 00000000000 14371172257 0014611 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/config/acquis.yaml 0000664 0000000 0000000 00000000436 14371172257 0016765 0 ustar 00root root 0000000 0000000 filenames:
- /var/log/nginx/*.log
- ./tests/nginx/nginx.log
#this is not a syslog log, indicate which kind of logs it is
labels:
type: nginx
---
filenames:
- /var/log/auth.log
- /var/log/syslog
labels:
type: syslog
---
filename: /var/log/apache2/*.log
labels:
type: apache2
crowdsec-1.4.6/config/acquis_win.yaml 0000664 0000000 0000000 00000000167 14371172257 0017643 0 ustar 00root root 0000000 0000000 source: wineventlog
event_channel: Security
event_ids:
- 4625
- 4623
event_level: information
labels:
type: eventlog crowdsec-1.4.6/config/config.yaml 0000664 0000000 0000000 00000003251 14371172257 0016743 0 ustar 00root root 0000000 0000000 common:
daemonize: true
pid_dir: /var/run/
log_media: file
log_level: info
log_dir: /var/log/
log_max_size: 20
compress_logs: true
log_max_files: 10
working_dir: .
config_paths:
config_dir: /etc/crowdsec/
data_dir: /var/lib/crowdsec/data/
simulation_path: /etc/crowdsec/simulation.yaml
hub_dir: /etc/crowdsec/hub/
index_path: /etc/crowdsec/hub/.index.json
notification_dir: /etc/crowdsec/notifications/
plugin_dir: /usr/local/lib/crowdsec/plugins/
crowdsec_service:
acquisition_path: /etc/crowdsec/acquis.yaml
acquisition_dir: /etc/crowdsec/acquis.d
parser_routines: 1
cscli:
output: human
color: auto
db_config:
log_level: info
type: sqlite
db_path: /var/lib/crowdsec/data/crowdsec.db
#max_open_conns: 100
#user:
#password:
#db_name:
#host:
#port:
flush:
max_items: 5000
max_age: 7d
plugin_config:
user: nobody # plugin process would be ran on behalf of this user
group: nogroup # plugin process would be ran on behalf of this group
api:
client:
insecure_skip_verify: false
credentials_path: /etc/crowdsec/local_api_credentials.yaml
server:
log_level: info
listen_uri: 127.0.0.1:8080
profiles_path: /etc/crowdsec/profiles.yaml
console_path: /etc/crowdsec/console.yaml
online_client: # Central API credentials (to push signals and receive bad IPs)
credentials_path: /etc/crowdsec/online_api_credentials.yaml
trusted_ips: # IP ranges, or IPs which can have admin API access
- 127.0.0.1
- ::1
# tls:
# cert_file: /etc/crowdsec/ssl/cert.pem
# key_file: /etc/crowdsec/ssl/key.pem
prometheus:
enabled: true
level: full
listen_addr: 127.0.0.1
listen_port: 6060
crowdsec-1.4.6/config/config_win.yaml 0000664 0000000 0000000 00000002626 14371172257 0017625 0 ustar 00root root 0000000 0000000 common:
daemonize: false
log_media: file
log_level: info
log_dir: C:\ProgramData\CrowdSec\log\
working_dir: .
config_paths:
config_dir: C:\ProgramData\CrowdSec\config\
data_dir: C:\ProgramData\CrowdSec\data\
simulation_path: C:\ProgramData\CrowdSec\config\simulation.yaml
hub_dir: C:\ProgramData\CrowdSec\hub\
index_path: C:\ProgramData\CrowdSec\hub\.index.json
plugin_dir: C:\ProgramData\CrowdSec\plugins\
notification_dir: C:\ProgramData\CrowdSec\config\notifications\
crowdsec_service:
acquisition_path: C:\ProgramData\CrowdSec\config\acquis.yaml
parser_routines: 1
cscli:
output: human
db_config:
log_level: info
type: sqlite
db_path: C:\ProgramData\CrowdSec\data\crowdsec.db
#user:
#password:
#db_name:
#host:
#port:
flush:
max_items: 5000
max_age: 7d
api:
client:
insecure_skip_verify: false
credentials_path: C:\ProgramData\CrowdSec\config\local_api_credentials.yaml
server:
log_level: info
listen_uri: 127.0.0.1:8080
profiles_path: C:\ProgramData\Crowdsec\config\profiles.yaml
online_client: # Crowdsec API credentials (to push signals and receive bad IPs)
credentials_path: C:\ProgramData\CrowdSec\config\online_api_credentials.yaml
# tls:
# cert_file: /etc/crowdsec/ssl/cert.pem
# key_file: /etc/crowdsec/ssl/key.pem
prometheus:
enabled: true
level: full
listen_addr: 127.0.0.1
listen_port: 6060
crowdsec-1.4.6/config/config_win_no_lapi.yaml 0000664 0000000 0000000 00000001513 14371172257 0021320 0 ustar 00root root 0000000 0000000 common:
daemonize: true
log_media: file
log_level: info
log_dir: C:\ProgramData\CrowdSec\log\
working_dir: .
config_paths:
config_dir: C:\ProgramData\CrowdSec\config\
data_dir: C:\ProgramData\CrowdSec\data\
simulation_path: C:\ProgramData\CrowdSec\config\simulation.yaml
hub_dir: C:\ProgramData\CrowdSec\hub\
index_path: C:\ProgramData\CrowdSec\hub\.index.json
plugin_dir: C:\ProgramData\CrowdSec\plugins\
notification_dir: C:\ProgramData\CrowdSec\config\notifications\
crowdsec_service:
acquisition_path: C:\ProgramData\CrowdSec\config\acquis.yaml
parser_routines: 1
cscli:
output: human
api:
client:
insecure_skip_verify: false
credentials_path: C:\ProgramData\CrowdSec\config\local_api_credentials.yaml
prometheus:
enabled: true
level: full
listen_addr: 127.0.0.1
listen_port: 6060
crowdsec-1.4.6/config/console.yaml 0000664 0000000 0000000 00000000105 14371172257 0017133 0 ustar 00root root 0000000 0000000 share_manual_decisions: false
share_custom: true
share_tainted: true
crowdsec-1.4.6/config/crowdsec.cron.daily 0000664 0000000 0000000 00000000407 14371172257 0020407 0 ustar 00root root 0000000 0000000 #!/bin/sh
test -x /usr/bin/cscli || exit 0
/usr/bin/cscli --error hub update
upgraded=$(/usr/bin/cscli --error hub upgrade)
if [ -n "$upgraded" ]; then
# splay initial metrics push
sleep $(seq 1 90 | shuf -n 1)
systemctl reload crowdsec
fi
exit 0
crowdsec-1.4.6/config/crowdsec.service 0000664 0000000 0000000 00000000606 14371172257 0020006 0 ustar 00root root 0000000 0000000 [Unit]
Description=Crowdsec agent
After=syslog.target network.target remote-fs.target nss-lookup.target
[Service]
Type=notify
Environment=LC_ALL=C LANG=C
ExecStartPre=/usr/local/bin/crowdsec -c /etc/crowdsec/config.yaml -t
ExecStart=/usr/local/bin/crowdsec -c /etc/crowdsec/config.yaml
#ExecStartPost=/bin/sleep 0.1
ExecReload=/bin/kill -HUP $MAINPID
[Install]
WantedBy=multi-user.target
crowdsec-1.4.6/config/dev.yaml 0000664 0000000 0000000 00000002250 14371172257 0016252 0 ustar 00root root 0000000 0000000 common:
daemonize: true
pid_dir: /tmp/
log_media: stdout
log_level: info
working_dir: .
config_paths:
config_dir: ./config
data_dir: ./data/
notification_dir: ./config/notifications/
plugin_dir: ./plugins/
#simulation_path: /etc/crowdsec/config/simulation.yaml
#hub_dir: /etc/crowdsec/hub/
#index_path: ./config/hub/.index.json
crowdsec_service:
acquisition_path: ./config/acquis.yaml
parser_routines: 1
plugin_config:
user: $USER # plugin process would be ran on behalf of this user
group: $USER # plugin process would be ran on behalf of this group
cscli:
output: human
db_config:
type: sqlite
db_path: ./data/crowdsec.db
user: root
password: crowdsec
db_name: crowdsec
host: "172.17.0.2"
port: 3306
flush:
#max_items: 10000
#max_age: 168h
api:
client:
credentials_path: ./config/local_api_credentials.yaml
server:
#insecure_skip_verify: true
listen_uri: 127.0.0.1:8081
profiles_path: ./config/profiles.yaml
tls:
#cert_file: ./cert.pem
#key_file: ./key.pem
online_client: # Central API
credentials_path: ./config/online_api_credentials.yaml
prometheus:
enabled: true
level: full
crowdsec-1.4.6/config/local_api_credentials.yaml 0000664 0000000 0000000 00000000032 14371172257 0021770 0 ustar 00root root 0000000 0000000 url: http://127.0.0.1:8080 crowdsec-1.4.6/config/online_api_credentials.yaml 0000664 0000000 0000000 00000000000 14371172257 0022155 0 ustar 00root root 0000000 0000000 crowdsec-1.4.6/config/patterns/ 0000775 0000000 0000000 00000000000 14371172257 0016451 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/config/patterns/aws 0000664 0000000 0000000 00000002254 14371172257 0017171 0 ustar 00root root 0000000 0000000 S3_REQUEST_LINE (?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})
S3_ACCESS_LOG %{WORD:owner} %{NOTSPACE:bucket} \[%{HTTPDATE:timestamp}\] %{IP:clientip} %{NOTSPACE:requester} %{NOTSPACE:request_id} %{NOTSPACE:operation} %{NOTSPACE:key} (?:"%{S3_REQUEST_LINE}"|-) (?:%{INT:response:int}|-) (?:-|%{NOTSPACE:error_code}) (?:%{INT:bytes:int}|-) (?:%{INT:object_size:int}|-) (?:%{INT:request_time_ms:int}|-) (?:%{INT:turnaround_time_ms:int}|-) (?:%{QS:referrer}|-) (?:"?%{QS:agent}"?|-) (?:-|%{NOTSPACE:version_id})
ELB_URIPATHPARAM %{URIPATH:path}(?:%{URIPARAM:params})?
ELB_URI %{URIPROTO:proto}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST:urihost})?(?:%{ELB_URIPATHPARAM})?
ELB_REQUEST_LINE (?:%{WORD:verb} %{ELB_URI:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})
ELB_ACCESS_LOG %{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:elb} %{IP:clientip}:%{INT:clientport:int} (?:(%{IP:backendip}:?:%{INT:backendport:int})|-) %{NUMBER:request_processing_time:float} %{NUMBER:backend_processing_time:float} %{NUMBER:response_processing_time:float} %{INT:response:int} %{INT:backend_response:int} %{INT:received_bytes:int} %{INT:bytes:int} "%{ELB_REQUEST_LINE}" crowdsec-1.4.6/config/patterns/bacula 0000664 0000000 0000000 00000011326 14371172257 0017626 0 ustar 00root root 0000000 0000000 BACULA_TIMESTAMP %{MONTHDAY}-%{MONTH} %{HOUR}:%{MINUTE}
BACULA_HOST [a-zA-Z0-9-]+
BACULA_VOLUME %{USER}
BACULA_DEVICE %{USER}
BACULA_DEVICEPATH %{UNIXPATH}
BACULA_CAPACITY %{INT}{1,3}(,%{INT}{3})*
BACULA_VERSION %{USER}
BACULA_JOB %{USER}
BACULA_LOG_MAX_CAPACITY User defined maximum volume capacity %{BACULA_CAPACITY} exceeded on device \"%{BACULA_DEVICE:device}\" \(%{BACULA_DEVICEPATH}\)
BACULA_LOG_END_VOLUME End of medium on Volume \"%{BACULA_VOLUME:volume}\" Bytes=%{BACULA_CAPACITY} Blocks=%{BACULA_CAPACITY} at %{MONTHDAY}-%{MONTH}-%{YEAR} %{HOUR}:%{MINUTE}.
BACULA_LOG_NEW_VOLUME Created new Volume \"%{BACULA_VOLUME:volume}\" in catalog.
BACULA_LOG_NEW_LABEL Labeled new Volume \"%{BACULA_VOLUME:volume}\" on device \"%{BACULA_DEVICE:device}\" \(%{BACULA_DEVICEPATH}\).
BACULA_LOG_WROTE_LABEL Wrote label to prelabeled Volume \"%{BACULA_VOLUME:volume}\" on device \"%{BACULA_DEVICE}\" \(%{BACULA_DEVICEPATH}\)
BACULA_LOG_NEW_MOUNT New volume \"%{BACULA_VOLUME:volume}\" mounted on device \"%{BACULA_DEVICE:device}\" \(%{BACULA_DEVICEPATH}\) at %{MONTHDAY}-%{MONTH}-%{YEAR} %{HOUR}:%{MINUTE}.
BACULA_LOG_NOOPEN \s+Cannot open %{DATA}: ERR=%{GREEDYDATA:berror}
BACULA_LOG_NOOPENDIR \s+Could not open directory %{DATA}: ERR=%{GREEDYDATA:berror}
BACULA_LOG_NOSTAT \s+Could not stat %{DATA}: ERR=%{GREEDYDATA:berror}
BACULA_LOG_NOJOBS There are no more Jobs associated with Volume \"%{BACULA_VOLUME:volume}\". Marking it purged.
BACULA_LOG_ALL_RECORDS_PRUNED All records pruned from Volume \"%{BACULA_VOLUME:volume}\"; marking it \"Purged\"
BACULA_LOG_BEGIN_PRUNE_JOBS Begin pruning Jobs older than %{INT} month %{INT} days .
BACULA_LOG_BEGIN_PRUNE_FILES Begin pruning Files.
BACULA_LOG_PRUNED_JOBS Pruned %{INT} Jobs* for client %{BACULA_HOST:client} from catalog.
BACULA_LOG_PRUNED_FILES Pruned Files from %{INT} Jobs* for client %{BACULA_HOST:client} from catalog.
BACULA_LOG_ENDPRUNE End auto prune.
BACULA_LOG_STARTJOB Start Backup JobId %{INT}, Job=%{BACULA_JOB:job}
BACULA_LOG_STARTRESTORE Start Restore Job %{BACULA_JOB:job}
BACULA_LOG_USEDEVICE Using Device \"%{BACULA_DEVICE:device}\"
BACULA_LOG_DIFF_FS \s+%{UNIXPATH} is a different filesystem. Will not descend from %{UNIXPATH} into it.
BACULA_LOG_JOBEND Job write elapsed time = %{DATA:elapsed}, Transfer rate = %{NUMBER} (K|M|G)? Bytes/second
BACULA_LOG_NOPRUNE_JOBS No Jobs found to prune.
BACULA_LOG_NOPRUNE_FILES No Files found to prune.
BACULA_LOG_VOLUME_PREVWRITTEN Volume \"%{BACULA_VOLUME:volume}\" previously written, moving to end of data.
BACULA_LOG_READYAPPEND Ready to append to end of Volume \"%{BACULA_VOLUME:volume}\" size=%{INT}
BACULA_LOG_CANCELLING Cancelling duplicate JobId=%{INT}.
BACULA_LOG_MARKCANCEL JobId %{INT}, Job %{BACULA_JOB:job} marked to be canceled.
BACULA_LOG_CLIENT_RBJ shell command: run ClientRunBeforeJob \"%{GREEDYDATA:runjob}\"
BACULA_LOG_VSS (Generate )?VSS (Writer)?
BACULA_LOG_MAXSTART Fatal error: Job canceled because max start delay time exceeded.
BACULA_LOG_DUPLICATE Fatal error: JobId %{INT:duplicate} already running. Duplicate job not allowed.
BACULA_LOG_NOJOBSTAT Fatal error: No Job status returned from FD.
BACULA_LOG_FATAL_CONN Fatal error: bsock.c:133 Unable to connect to (Client: %{BACULA_HOST:client}|Storage daemon) on %{HOSTNAME}:%{POSINT}. ERR=%{GREEDYDATA:berror}
BACULA_LOG_NO_CONNECT Warning: bsock.c:127 Could not connect to (Client: %{BACULA_HOST:client}|Storage daemon) on %{HOSTNAME}:%{POSINT}. ERR=%{GREEDYDATA:berror}
BACULA_LOG_NO_AUTH Fatal error: Unable to authenticate with File daemon at %{HOSTNAME}. Possible causes:
BACULA_LOG_NOSUIT No prior or suitable Full backup found in catalog. Doing FULL backup.
BACULA_LOG_NOPRIOR No prior Full backup Job record found.
BACULA_LOG_JOB (Error: )?Bacula %{BACULA_HOST} %{BACULA_VERSION} \(%{BACULA_VERSION}\):
BACULA_LOGLINE %{BACULA_TIMESTAMP:bts} %{BACULA_HOST:hostname} JobId %{INT:jobid}: (%{BACULA_LOG_MAX_CAPACITY}|%{BACULA_LOG_END_VOLUME}|%{BACULA_LOG_NEW_VOLUME}|%{BACULA_LOG_NEW_LABEL}|%{BACULA_LOG_WROTE_LABEL}|%{BACULA_LOG_NEW_MOUNT}|%{BACULA_LOG_NOOPEN}|%{BACULA_LOG_NOOPENDIR}|%{BACULA_LOG_NOSTAT}|%{BACULA_LOG_NOJOBS}|%{BACULA_LOG_ALL_RECORDS_PRUNED}|%{BACULA_LOG_BEGIN_PRUNE_JOBS}|%{BACULA_LOG_BEGIN_PRUNE_FILES}|%{BACULA_LOG_PRUNED_JOBS}|%{BACULA_LOG_PRUNED_FILES}|%{BACULA_LOG_ENDPRUNE}|%{BACULA_LOG_STARTJOB}|%{BACULA_LOG_STARTRESTORE}|%{BACULA_LOG_USEDEVICE}|%{BACULA_LOG_DIFF_FS}|%{BACULA_LOG_JOBEND}|%{BACULA_LOG_NOPRUNE_JOBS}|%{BACULA_LOG_NOPRUNE_FILES}|%{BACULA_LOG_VOLUME_PREVWRITTEN}|%{BACULA_LOG_READYAPPEND}|%{BACULA_LOG_CANCELLING}|%{BACULA_LOG_MARKCANCEL}|%{BACULA_LOG_CLIENT_RBJ}|%{BACULA_LOG_VSS}|%{BACULA_LOG_MAXSTART}|%{BACULA_LOG_DUPLICATE}|%{BACULA_LOG_NOJOBSTAT}|%{BACULA_LOG_FATAL_CONN}|%{BACULA_LOG_NO_CONNECT}|%{BACULA_LOG_NO_AUTH}|%{BACULA_LOG_NOSUIT}|%{BACULA_LOG_JOB}|%{BACULA_LOG_NOPRIOR}) crowdsec-1.4.6/config/patterns/bro 0000664 0000000 0000000 00000004151 14371172257 0017157 0 ustar 00root root 0000000 0000000 # https://www.bro.org/sphinx/script-reference/log-files.html
# http.log
BRO_HTTP %{NUMBER:ts}\t%{NOTSPACE:uid}\t%{IP:orig_h}\t%{INT:orig_p}\t%{IP:resp_h}\t%{INT:resp_p}\t%{INT:trans_depth}\t%{GREEDYDATA:method}\t%{GREEDYDATA:domain}\t%{GREEDYDATA:uri}\t%{GREEDYDATA:referrer}\t%{GREEDYDATA:user_agent}\t%{NUMBER:request_body_len}\t%{NUMBER:response_body_len}\t%{GREEDYDATA:status_code}\t%{GREEDYDATA:status_msg}\t%{GREEDYDATA:info_code}\t%{GREEDYDATA:info_msg}\t%{GREEDYDATA:filename}\t%{GREEDYDATA:bro_tags}\t%{GREEDYDATA:username}\t%{GREEDYDATA:password}\t%{GREEDYDATA:proxied}\t%{GREEDYDATA:orig_fuids}\t%{GREEDYDATA:orig_mime_types}\t%{GREEDYDATA:resp_fuids}\t%{GREEDYDATA:resp_mime_types}
# dns.log
BRO_DNS %{NUMBER:ts}\t%{NOTSPACE:uid}\t%{IP:orig_h}\t%{INT:orig_p}\t%{IP:resp_h}\t%{INT:resp_p}\t%{WORD:proto}\t%{INT:trans_id}\t%{GREEDYDATA:query}\t%{GREEDYDATA:qclass}\t%{GREEDYDATA:qclass_name}\t%{GREEDYDATA:qtype}\t%{GREEDYDATA:qtype_name}\t%{GREEDYDATA:rcode}\t%{GREEDYDATA:rcode_name}\t%{GREEDYDATA:AA}\t%{GREEDYDATA:TC}\t%{GREEDYDATA:RD}\t%{GREEDYDATA:RA}\t%{GREEDYDATA:Z}\t%{GREEDYDATA:answers}\t%{GREEDYDATA:TTLs}\t%{GREEDYDATA:rejected}
# conn.log
BRO_CONN %{NUMBER:ts}\t%{NOTSPACE:uid}\t%{IP:orig_h}\t%{INT:orig_p}\t%{IP:resp_h}\t%{INT:resp_p}\t%{WORD:proto}\t%{GREEDYDATA:service}\t%{NUMBER:duration}\t%{NUMBER:orig_bytes}\t%{NUMBER:resp_bytes}\t%{GREEDYDATA:conn_state}\t%{GREEDYDATA:local_orig}\t%{GREEDYDATA:missed_bytes}\t%{GREEDYDATA:history}\t%{GREEDYDATA:orig_pkts}\t%{GREEDYDATA:orig_ip_bytes}\t%{GREEDYDATA:resp_pkts}\t%{GREEDYDATA:resp_ip_bytes}\t%{GREEDYDATA:tunnel_parents}
# files.log
BRO_FILES %{NUMBER:ts}\t%{NOTSPACE:fuid}\t%{IP:tx_hosts}\t%{IP:rx_hosts}\t%{NOTSPACE:conn_uids}\t%{GREEDYDATA:source}\t%{GREEDYDATA:depth}\t%{GREEDYDATA:analyzers}\t%{GREEDYDATA:mime_type}\t%{GREEDYDATA:filename}\t%{GREEDYDATA:duration}\t%{GREEDYDATA:local_orig}\t%{GREEDYDATA:is_orig}\t%{GREEDYDATA:seen_bytes}\t%{GREEDYDATA:total_bytes}\t%{GREEDYDATA:missing_bytes}\t%{GREEDYDATA:overflow_bytes}\t%{GREEDYDATA:timedout}\t%{GREEDYDATA:parent_fuid}\t%{GREEDYDATA:md5}\t%{GREEDYDATA:sha1}\t%{GREEDYDATA:sha256}\t%{GREEDYDATA:extracted} crowdsec-1.4.6/config/patterns/cowrie_honeypot 0000664 0000000 0000000 00000000200 14371172257 0021601 0 ustar 00root root 0000000 0000000 COWRIE_NEW_CO New connection: %{IPV4:source_ip}:[0-9]+ \(%{IPV4:dest_ip}:%{INT:dest_port}\) \[session: %{DATA:telnet_session}\]$ crowdsec-1.4.6/config/patterns/exim 0000664 0000000 0000000 00000001555 14371172257 0017344 0 ustar 00root root 0000000 0000000 EXIM_MSGID [0-9A-Za-z]{6}-[0-9A-Za-z]{6}-[0-9A-Za-z]{2}
EXIM_FLAGS (<=|[-=>*]>|[*]{2}|==)
EXIM_DATE %{YEAR:exim_year}-%{MONTHNUM:exim_month}-%{MONTHDAY:exim_day} %{TIME:exim_time}
EXIM_PID \[%{POSINT}\]
EXIM_QT ((\d+y)?(\d+w)?(\d+d)?(\d+h)?(\d+m)?(\d+s)?)
EXIM_EXCLUDE_TERMS (Message is frozen|(Start|End) queue run| Warning: | retry time not reached | no (IP address|host name) found for (IP address|host) | unexpected disconnection while reading SMTP command | no immediate delivery: |another process is handling this message)
EXIM_REMOTE_HOST (H=(%{NOTSPACE:remote_hostname} )?(\(%{NOTSPACE:remote_heloname}\) )?\[%{IP:remote_host}\])
EXIM_INTERFACE (I=\[%{IP:exim_interface}\](:%{NUMBER:exim_interface_port}))
EXIM_PROTOCOL (P=%{NOTSPACE:protocol})
EXIM_MSG_SIZE (S=%{NUMBER:exim_msg_size})
EXIM_HEADER_ID (id=%{NOTSPACE:exim_header_id})
EXIM_SUBJECT (T=%{QS:exim_subject}) crowdsec-1.4.6/config/patterns/firewalls 0000664 0000000 0000000 00000022507 14371172257 0020372 0 ustar 00root root 0000000 0000000 # NetScreen firewall logs
NETSCREENSESSIONLOG %{SYSLOGTIMESTAMP:date} %{IPORHOST:device} %{IPORHOST}: NetScreen device_id=%{WORD:device_id}%{DATA}: start_time=%{QUOTEDSTRING:start_time} duration=%{INT:duration} policy_id=%{INT:policy_id} service=%{DATA:service} proto=%{INT:proto} src zone=%{WORD:src_zone} dst zone=%{WORD:dst_zone} action=%{WORD:action} sent=%{INT:sent} rcvd=%{INT:rcvd} src=%{IPORHOST:src_ip} dst=%{IPORHOST:dst_ip} src_port=%{INT:src_port} dst_port=%{INT:dst_port} src-xlated ip=%{IPORHOST:src_xlated_ip} port=%{INT:src_xlated_port} dst-xlated ip=%{IPORHOST:dst_xlated_ip} port=%{INT:dst_xlated_port} session_id=%{INT:session_id} reason=%{GREEDYDATA:reason}
#== Cisco ASA ==
CISCOTAG [A-Z0-9]+-%{INT}-(?:[A-Z0-9_]+)
CISCOTIMESTAMP %{MONTH} +%{MONTHDAY}(?: %{YEAR})? %{TIME}
CISCO_TAGGED_SYSLOG ^<%{POSINT:syslog_pri}>%{CISCOTIMESTAMP:timestamp}( %{SYSLOGHOST:sysloghost})? ?: %%{CISCOTAG:ciscotag}:
# Common Particles
CISCO_ACTION Built|Teardown|Deny|Denied|denied|requested|permitted|denied by ACL|discarded|est-allowed|Dropping|created|deleted
CISCO_REASON Duplicate TCP SYN|Failed to locate egress interface|Invalid transport field|No matching connection|DNS Response|DNS Query|(?:%{WORD}\s*)*
CISCO_DIRECTION Inbound|inbound|Outbound|outbound
CISCO_INTERVAL first hit|%{INT}-second interval
CISCO_XLATE_TYPE static|dynamic
# ASA-1-104001
CISCOFW104001 \((?:Primary|Secondary)\) Switching to ACTIVE - %{GREEDYDATA:switch_reason}
# ASA-1-104002
CISCOFW104002 \((?:Primary|Secondary)\) Switching to STANDBY - %{GREEDYDATA:switch_reason}
# ASA-1-104003
CISCOFW104003 \((?:Primary|Secondary)\) Switching to FAILED\.
# ASA-1-104004
CISCOFW104004 \((?:Primary|Secondary)\) Switching to OK\.
# ASA-1-105003
CISCOFW105003 \((?:Primary|Secondary)\) Monitoring on [Ii]nterface %{GREEDYDATA:interface_name} waiting
# ASA-1-105004
CISCOFW105004 \((?:Primary|Secondary)\) Monitoring on [Ii]nterface %{GREEDYDATA:interface_name} normal
# ASA-1-105005
CISCOFW105005 \((?:Primary|Secondary)\) Lost Failover communications with mate on [Ii]nterface %{GREEDYDATA:interface_name}
# ASA-1-105008
CISCOFW105008 \((?:Primary|Secondary)\) Testing [Ii]nterface %{GREEDYDATA:interface_name}
# ASA-1-105009
CISCOFW105009 \((?:Primary|Secondary)\) Testing on [Ii]nterface %{GREEDYDATA:interface_name} (?:Passed|Failed)
# ASA-2-106001
CISCOFW106001 %{CISCO_DIRECTION:direction} %{WORD:protocol} connection %{CISCO_ACTION:action} from %{IP:src_ip}/%{INT:src_port} to %{IP:dst_ip}/%{INT:dst_port} flags %{GREEDYDATA:tcp_flags} on interface %{GREEDYDATA:interface}
# ASA-2-106006, ASA-2-106007, ASA-2-106010
CISCOFW106006_106007_106010 %{CISCO_ACTION:action} %{CISCO_DIRECTION:direction} %{WORD:protocol} (?:from|src) %{IP:src_ip}/%{INT:src_port}(\(%{DATA:src_fwuser}\))? (?:to|dst) %{IP:dst_ip}/%{INT:dst_port}(\(%{DATA:dst_fwuser}\))? (?:on interface %{DATA:interface}|due to %{CISCO_REASON:reason})
# ASA-3-106014
CISCOFW106014 %{CISCO_ACTION:action} %{CISCO_DIRECTION:direction} %{WORD:protocol} src %{DATA:src_interface}:%{IP:src_ip}(\(%{DATA:src_fwuser}\))? dst %{DATA:dst_interface}:%{IP:dst_ip}(\(%{DATA:dst_fwuser}\))? \(type %{INT:icmp_type}, code %{INT:icmp_code}\)
# ASA-6-106015
CISCOFW106015 %{CISCO_ACTION:action} %{WORD:protocol} \(%{DATA:policy_id}\) from %{IP:src_ip}/%{INT:src_port} to %{IP:dst_ip}/%{INT:dst_port} flags %{DATA:tcp_flags} on interface %{GREEDYDATA:interface}
# ASA-1-106021
CISCOFW106021 %{CISCO_ACTION:action} %{WORD:protocol} reverse path check from %{IP:src_ip} to %{IP:dst_ip} on interface %{GREEDYDATA:interface}
# ASA-4-106023
CISCOFW106023 %{CISCO_ACTION:action}( protocol)? %{WORD:protocol} src %{DATA:src_interface}:%{DATA:src_ip}(/%{INT:src_port})?(\(%{DATA:src_fwuser}\))? dst %{DATA:dst_interface}:%{DATA:dst_ip}(/%{INT:dst_port})?(\(%{DATA:dst_fwuser}\))?( \(type %{INT:icmp_type}, code %{INT:icmp_code}\))? by access-group "?%{DATA:policy_id}"? \[%{DATA:hashcode1}, %{DATA:hashcode2}\]
# ASA-4-106100, ASA-4-106102, ASA-4-106103
CISCOFW106100_2_3 access-list %{NOTSPACE:policy_id} %{CISCO_ACTION:action} %{WORD:protocol} for user '%{DATA:src_fwuser}' %{DATA:src_interface}/%{IP:src_ip}\(%{INT:src_port}\) -> %{DATA:dst_interface}/%{IP:dst_ip}\(%{INT:dst_port}\) hit-cnt %{INT:hit_count} %{CISCO_INTERVAL:interval} \[%{DATA:hashcode1}, %{DATA:hashcode2}\]
# ASA-5-106100
CISCOFW106100 access-list %{NOTSPACE:policy_id} %{CISCO_ACTION:action} %{WORD:protocol} %{DATA:src_interface}/%{IP:src_ip}\(%{INT:src_port}\)(\(%{DATA:src_fwuser}\))? -> %{DATA:dst_interface}/%{IP:dst_ip}\(%{INT:dst_port}\)(\(%{DATA:src_fwuser}\))? hit-cnt %{INT:hit_count} %{CISCO_INTERVAL:interval} \[%{DATA:hashcode1}, %{DATA:hashcode2}\]
# ASA-6-110002
CISCOFW110002 %{CISCO_REASON:reason} for %{WORD:protocol} from %{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port} to %{IP:dst_ip}/%{INT:dst_port}
# ASA-6-302010
CISCOFW302010 %{INT:connection_count} in use, %{INT:connection_count_max} most used
# ASA-6-302013, ASA-6-302014, ASA-6-302015, ASA-6-302016
CISCOFW302013_302014_302015_302016 %{CISCO_ACTION:action}(?: %{CISCO_DIRECTION:direction})? %{WORD:protocol} connection %{INT:connection_id} for %{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port}( \(%{IP:src_mapped_ip}/%{INT:src_mapped_port}\))?(\(%{DATA:src_fwuser}\))? to %{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port}( \(%{IP:dst_mapped_ip}/%{INT:dst_mapped_port}\))?(\(%{DATA:dst_fwuser}\))?( duration %{TIME:duration} bytes %{INT:bytes})?(?: %{CISCO_REASON:reason})?( \(%{DATA:user}\))?
# ASA-6-302020, ASA-6-302021
CISCOFW302020_302021 %{CISCO_ACTION:action}(?: %{CISCO_DIRECTION:direction})? %{WORD:protocol} connection for faddr %{IP:dst_ip}/%{INT:icmp_seq_num}(?:\(%{DATA:fwuser}\))? gaddr %{IP:src_xlated_ip}/%{INT:icmp_code_xlated} laddr %{IP:src_ip}/%{INT:icmp_code}( \(%{DATA:user}\))?
# ASA-6-305011
CISCOFW305011 %{CISCO_ACTION:action} %{CISCO_XLATE_TYPE:xlate_type} %{WORD:protocol} translation from %{DATA:src_interface}:%{IP:src_ip}(/%{INT:src_port})?(\(%{DATA:src_fwuser}\))? to %{DATA:src_xlated_interface}:%{IP:src_xlated_ip}/%{DATA:src_xlated_port}
# ASA-3-313001, ASA-3-313004, ASA-3-313008
CISCOFW313001_313004_313008 %{CISCO_ACTION:action} %{WORD:protocol} type=%{INT:icmp_type}, code=%{INT:icmp_code} from %{IP:src_ip} on interface %{DATA:interface}( to %{IP:dst_ip})?
# ASA-4-313005
CISCOFW313005 %{CISCO_REASON:reason} for %{WORD:protocol} error message: %{WORD:err_protocol} src %{DATA:err_src_interface}:%{IP:err_src_ip}(\(%{DATA:err_src_fwuser}\))? dst %{DATA:err_dst_interface}:%{IP:err_dst_ip}(\(%{DATA:err_dst_fwuser}\))? \(type %{INT:err_icmp_type}, code %{INT:err_icmp_code}\) on %{DATA:interface} interface\. Original IP payload: %{WORD:protocol} src %{IP:orig_src_ip}/%{INT:orig_src_port}(\(%{DATA:orig_src_fwuser}\))? dst %{IP:orig_dst_ip}/%{INT:orig_dst_port}(\(%{DATA:orig_dst_fwuser}\))?
# ASA-5-321001
CISCOFW321001 Resource '%{WORD:resource_name}' limit of %{POSINT:resource_limit} reached for system
# ASA-4-402117
CISCOFW402117 %{WORD:protocol}: Received a non-IPSec packet \(protocol= %{WORD:orig_protocol}\) from %{IP:src_ip} to %{IP:dst_ip}
# ASA-4-402119
CISCOFW402119 %{WORD:protocol}: Received an %{WORD:orig_protocol} packet \(SPI= %{DATA:spi}, sequence number= %{DATA:seq_num}\) from %{IP:src_ip} \(user= %{DATA:user}\) to %{IP:dst_ip} that failed anti-replay checking
# ASA-4-419001
CISCOFW419001 %{CISCO_ACTION:action} %{WORD:protocol} packet from %{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port} to %{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port}, reason: %{GREEDYDATA:reason}
# ASA-4-419002
CISCOFW419002 %{CISCO_REASON:reason} from %{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port} to %{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port} with different initial sequence number
# ASA-4-500004
CISCOFW500004 %{CISCO_REASON:reason} for protocol=%{WORD:protocol}, from %{IP:src_ip}/%{INT:src_port} to %{IP:dst_ip}/%{INT:dst_port}
# ASA-6-602303, ASA-6-602304
CISCOFW602303_602304 %{WORD:protocol}: An %{CISCO_DIRECTION:direction} %{GREEDYDATA:tunnel_type} SA \(SPI= %{DATA:spi}\) between %{IP:src_ip} and %{IP:dst_ip} \(user= %{DATA:user}\) has been %{CISCO_ACTION:action}
# ASA-7-710001, ASA-7-710002, ASA-7-710003, ASA-7-710005, ASA-7-710006
CISCOFW710001_710002_710003_710005_710006 %{WORD:protocol} (?:request|access) %{CISCO_ACTION:action} from %{IP:src_ip}/%{INT:src_port} to %{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port}
# ASA-6-713172
CISCOFW713172 Group = %{GREEDYDATA:group}, IP = %{IP:src_ip}, Automatic NAT Detection Status:\s+Remote end\s*%{DATA:is_remote_natted}\s*behind a NAT device\s+This\s+end\s*%{DATA:is_local_natted}\s*behind a NAT device
# ASA-4-733100
CISCOFW733100 \[\s*%{DATA:drop_type}\s*\] drop %{DATA:drop_rate_id} exceeded. Current burst rate is %{INT:drop_rate_current_burst} per second, max configured rate is %{INT:drop_rate_max_burst}; Current average rate is %{INT:drop_rate_current_avg} per second, max configured rate is %{INT:drop_rate_max_avg}; Cumulative total count is %{INT:drop_total_count}
#== End Cisco ASA ==
# Shorewall firewall logs
SHOREWALL (%{SYSLOGTIMESTAMP:timestamp}) (%{WORD:nf_host}) kernel:.*Shorewall:(%{WORD:nf_action1})?:(%{WORD:nf_action2})?.*IN=(%{USERNAME:nf_in_interface})?.*(OUT= *MAC=(%{COMMONMAC:nf_dst_mac}):(%{COMMONMAC:nf_src_mac})?|OUT=%{USERNAME:nf_out_interface}).*SRC=(%{IPV4:nf_src_ip}).*DST=(%{IPV4:nf_dst_ip}).*LEN=(%{WORD:nf_len}).*?TOS=(%{WORD:nf_tos}).*?PREC=(%{WORD:nf_prec}).*?TTL=(%{INT:nf_ttl}).*?ID=(%{INT:nf_id}).*?PROTO=(%{WORD:nf_protocol}).*?SPT=(%{INT:nf_src_port}?.*DPT=%{INT:nf_dst_port}?.*)
#== End Shorewall crowdsec-1.4.6/config/patterns/haproxy 0000664 0000000 0000000 00000006237 14371172257 0020076 0 ustar 00root root 0000000 0000000 ## These patterns were tested w/ haproxy-1.4.15
## Documentation of the haproxy log formats can be found at the following links:
## http://code.google.com/p/haproxy-docs/wiki/HTTPLogFormat
## http://code.google.com/p/haproxy-docs/wiki/TCPLogFormat
HAPROXYTIME %{HOUR:haproxy_hour}:%{MINUTE:haproxy_minute}(?::%{SECOND:haproxy_second})
HAPROXYDATE %{MONTHDAY:haproxy_monthday}/%{MONTH:haproxy_month}/%{YEAR:haproxy_year}:%{HAPROXYTIME:haproxy_time}.%{INT:haproxy_milliseconds}
# Override these default patterns to parse out what is captured in your haproxy.cfg
HAPROXYCAPTUREDREQUESTHEADERS %{DATA:captured_request_headers}
HAPROXYCAPTUREDRESPONSEHEADERS %{DATA:captured_response_headers}
# Example:
# These haproxy config lines will add data to the logs that are captured
# by the patterns below. Place them in your custom patterns directory to
# override the defaults.
#
# capture request header Host len 40
# capture request header X-Forwarded-For len 50
# capture request header Accept-Language len 50
# capture request header Referer len 200
# capture request header User-Agent len 200
#
# capture response header Content-Type len 30
# capture response header Content-Encoding len 10
# capture response header Cache-Control len 200
# capture response header Last-Modified len 200
#
# HAPROXYCAPTUREDREQUESTHEADERS %{DATA:request_header_host}\|%{DATA:request_header_x_forwarded_for}\|%{DATA:request_header_accept_language}\|%{DATA:request_header_referer}\|%{DATA:request_header_user_agent}
# HAPROXYCAPTUREDRESPONSEHEADERS %{DATA:response_header_content_type}\|%{DATA:response_header_content_encoding}\|%{DATA:response_header_cache_control}\|%{DATA:response_header_last_modified}
# parse a haproxy 'httplog' line
HAPROXYHTTPBASE %{IP:client_ip}:%{INT:client_port} \[%{HAPROXYDATE:accept_date}\] %{NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{INT:time_request}/%{INT:time_queue}/%{INT:time_backend_connect}/%{INT:time_backend_response}/%{NOTSPACE:time_duration} %{INT:http_status_code} %{NOTSPACE:bytes_read} %{DATA:captured_request_cookie} %{DATA:captured_response_cookie} %{NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/%{INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} %{INT:srv_queue}/%{INT:backend_queue} (\{%{HAPROXYCAPTUREDREQUESTHEADERS}\})?( )?(\{%{HAPROXYCAPTUREDRESPONSEHEADERS}\})?( )?"(|(%{WORD:http_verb} (%{URIPROTO:http_proto}://)?(?:%{USER:http_user}(?::[^@]*)?@)?(?:%{URIHOST:http_host})?(?:%{URIPATHPARAM:http_request})?( HTTP/%{NUMBER:http_version})?))?"
HAPROXYHTTP (?:%{SYSLOGTIMESTAMP:syslog_timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) %{IPORHOST:syslog_server} %{SYSLOGPROG}: %{HAPROXYHTTPBASE}
# parse a haproxy 'tcplog' line
HAPROXYTCP (?:%{SYSLOGTIMESTAMP:syslog_timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) %{IPORHOST:syslog_server} %{SYSLOGPROG}: %{IP:client_ip}:%{INT:client_port} \[%{HAPROXYDATE:accept_date}\] %{NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{INT:time_queue}/%{INT:time_backend_connect}/%{NOTSPACE:time_duration} %{NOTSPACE:bytes_read} %{NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/%{INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} %{INT:srv_queue}/%{INT:backend_queue} crowdsec-1.4.6/config/patterns/java 0000664 0000000 0000000 00000002500 14371172257 0017312 0 ustar 00root root 0000000 0000000 JAVACLASS (?:[a-zA-Z$_][a-zA-Z$_0-9]*\.)*[a-zA-Z$_][a-zA-Z$_0-9]*
#Space is an allowed character to match special cases like 'Native Method' or 'Unknown Source'
JAVAFILE (?:[A-Za-z0-9_. -]+)
#Allow special method
JAVAMETHOD (?:()|[a-zA-Z$_][a-zA-Z$_0-9]*)
#Line number is optional in special cases 'Native method' or 'Unknown source'
JAVASTACKTRACEPART %{SPACE}at %{JAVACLASS:class}\.%{JAVAMETHOD:method}\(%{JAVAFILE:file}(?::%{NUMBER:line})?\)
# Java Logs
JAVATHREAD (?:[A-Z]{2}-Processor[\d]+)
##JAVACLASS (?:[a-zA-Z0-9-]+\.)+[A-Za-z0-9$]+
##JAVAFILE (?:[A-Za-z0-9_.-]+)
##JAVASTACKTRACEPART at %{JAVACLASS:class}\.%{WORD:method}\(%{JAVAFILE:file}:%{NUMBER:line}\)
JAVALOGMESSAGE (.*)
# MMM dd, yyyy HH:mm:ss eg: Jan 9, 2014 7:13:13 AM
CATALINA_DATESTAMP %{MONTH} %{MONTHDAY}, 20%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) (?:AM|PM)
# yyyy-MM-dd HH:mm:ss,SSS ZZZ eg: 2014-01-09 17:32:25,527 -0800
TOMCAT_DATESTAMP 20%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) %{ISO8601_TIMEZONE}
CATALINALOG %{CATALINA_DATESTAMP:timestamp} %{JAVACLASS:class} %{JAVALOGMESSAGE:logmessage}
# 2014-01-09 20:03:28,269 -0800 | ERROR | com.example.service.ExampleService - something compeletely unexpected happened...
TOMCATLOG %{TOMCAT_DATESTAMP:timestamp} \| %{LOGLEVEL:level} \| %{JAVACLASS:class} - %{JAVALOGMESSAGE:logmessage} crowdsec-1.4.6/config/patterns/junos 0000664 0000000 0000000 00000002076 14371172257 0017537 0 ustar 00root root 0000000 0000000 # JUNOS 11.4 RT_FLOW patterns
RT_FLOW_EVENT (RT_FLOW_SESSION_CREATE|RT_FLOW_SESSION_CLOSE|RT_FLOW_SESSION_DENY)
RT_FLOW1 %{RT_FLOW_EVENT:event}: %{GREEDYDATA:close-reason}: %{IP:src-ip}/%{INT:src-port}->%{IP:dst-ip}/%{INT:dst-port} %{DATA:service} %{IP:nat-src-ip}/%{INT:nat-src-port}->%{IP:nat-dst-ip}/%{INT:nat-dst-port} %{DATA:src-nat-rule-name} %{DATA:dst-nat-rule-name} %{INT:protocol-id} %{DATA:policy-name} %{DATA:from-zone} %{DATA:to-zone} %{INT:session-id} \d+\(%{DATA:sent}\) \d+\(%{DATA:received}\) %{INT:elapsed-time} .*
RT_FLOW2 %{RT_FLOW_EVENT:event}: session created %{IP:src-ip}/%{INT:src-port}->%{IP:dst-ip}/%{INT:dst-port} %{DATA:service} %{IP:nat-src-ip}/%{INT:nat-src-port}->%{IP:nat-dst-ip}/%{INT:nat-dst-port} %{DATA:src-nat-rule-name} %{DATA:dst-nat-rule-name} %{INT:protocol-id} %{DATA:policy-name} %{DATA:from-zone} %{DATA:to-zone} %{INT:session-id} .*
RT_FLOW3 %{RT_FLOW_EVENT:event}: session denied %{IP:src-ip}/%{INT:src-port}->%{IP:dst-ip}/%{INT:dst-port} %{DATA:service} %{INT:protocol-id}\(\d\) %{DATA:policy-name} %{DATA:from-zone} %{DATA:to-zone} .*
crowdsec-1.4.6/config/patterns/linux-syslog 0000664 0000000 0000000 00000002010 14371172257 0021042 0 ustar 00root root 0000000 0000000 SYSLOG5424PRINTASCII [!-~]+
SYSLOGBASE2 (?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource}+(?: %{SYSLOGPROG}:|)
SYSLOGPAMSESSION %{SYSLOGBASE} %{GREEDYDATA:message}%{WORD:pam_module}\(%{DATA:pam_caller}\): session %{WORD:pam_session_state} for user %{USERNAME:username}(?: by %{GREEDYDATA:pam_by})?
CRON_ACTION [A-Z ]+
CRONLOG %{SYSLOGBASE} \(%{USER:user}\) %{CRON_ACTION:action} \(%{DATA:message}\)
SYSLOGLINE %{SYSLOGBASE2} %{GREEDYDATA:message}
# IETF 5424 syslog(8) format (see http://www.rfc-editor.org/info/rfc5424)
SYSLOG5424PRI <%{NONNEGINT:syslog5424_pri}>
SYSLOG5424SD \[%{DATA}\]+
SYSLOG5424BASE %{SYSLOG5424PRI}%{NONNEGINT:syslog5424_ver} +(?:%{TIMESTAMP_ISO8601:syslog5424_ts}|-) +(?:%{HOSTNAME:syslog5424_host}|-) +(-|%{SYSLOG5424PRINTASCII:syslog5424_app}) +(-|%{SYSLOG5424PRINTASCII:syslog5424_proc}) +(-|%{SYSLOG5424PRINTASCII:syslog5424_msgid}) +(?:%{SYSLOG5424SD:syslog5424_sd}|-|)
SYSLOG5424LINE %{SYSLOG5424BASE} +%{GREEDYDATA:syslog5424_msg} crowdsec-1.4.6/config/patterns/mcollective 0000664 0000000 0000000 00000000275 14371172257 0020706 0 ustar 00root root 0000000 0000000 # Remember, these can be multi-line events.
MCOLLECTIVE ., \[%{TIMESTAMP_ISO8601:timestamp} #%{POSINT:pid}\]%{SPACE}%{LOGLEVEL:event_level}
MCOLLECTIVEAUDIT %{TIMESTAMP_ISO8601:timestamp}: crowdsec-1.4.6/config/patterns/modsecurity 0000664 0000000 0000000 00000003271 14371172257 0020746 0 ustar 00root root 0000000 0000000 APACHEERRORTIME %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}
APACHEERRORPREFIX \[%{APACHEERRORTIME:timestamp}\] \[%{NOTSPACE:apacheseverity}\] (\[pid %{INT}:tid %{INT}\] )?\[client %{IPORHOST:sourcehost}(:%{INT:source_port})?\] (\[client %{IPORHOST}\])?
GENERICAPACHEERROR %{APACHEERRORPREFIX} %{GREEDYDATA:message}
MODSECPREFIX %{APACHEERRORPREFIX} ModSecurity: %{NOTSPACE:modsecseverity}\. %{GREEDYDATA:modsecmessage}
MODSECRULEFILE \[file %{QUOTEDSTRING:rulefile}\]
MODSECRULELINE \[line %{QUOTEDSTRING:ruleline}\]
MODSECMATCHOFFSET \[offset %{QUOTEDSTRING:matchoffset}\]
MODSECRULEID \[id %{QUOTEDSTRING:ruleid}\]
MODSECRULEREV \[rev %{QUOTEDSTRING:rulerev}\]
MODSECRULEMSG \[msg %{QUOTEDSTRING:rulemessage}\]
MODSECRULEDATA \[data %{QUOTEDSTRING:ruledata}\]
MODSECRULESEVERITY \[severity ["']%{WORD:ruleseverity}["']\]
MODSECRULEVERS \[ver "[^"]+"\]
MODSECRULETAGS (?:\[tag %{QUOTEDSTRING:ruletag0}\] )?(?:\[tag %{QUOTEDSTRING:ruletag1}\] )?(?:\[tag %{QUOTEDSTRING:ruletag2}\] )?(?:\[tag %{QUOTEDSTRING:ruletag3}\] )?(?:\[tag %{QUOTEDSTRING:ruletag4}\] )?(?:\[tag %{QUOTEDSTRING:ruletag5}\] )?(?:\[tag %{QUOTEDSTRING:ruletag6}\] )?(?:\[tag %{QUOTEDSTRING:ruletag7}\] )?(?:\[tag %{QUOTEDSTRING:ruletag8}\] )?(?:\[tag %{QUOTEDSTRING:ruletag9}\] )?(?:\[tag %{QUOTEDSTRING}\] )*
MODSECHOSTNAME \[hostname ['"]%{DATA:targethost}["']\]
MODSECURI \[uri ["']%{DATA:targeturi}["']\]
MODSECUID \[unique_id %{QUOTEDSTRING:uniqueid}\]
MODSECAPACHEERROR %{MODSECPREFIX} %{MODSECRULEFILE} %{MODSECRULELINE} (?:%{MODSECMATCHOFFSET} )?(?:%{MODSECRULEID} )?(?:%{MODSECRULEREV} )?(?:%{MODSECRULEMSG} )?(?:%{MODSECRULEDATA} )?(?:%{MODSECRULESEVERITY} )?(?:%{MODSECRULEVERS} )?%{MODSECRULETAGS}%{MODSECHOSTNAME} %{MODSECURI} %{MODSECUID} crowdsec-1.4.6/config/patterns/mongodb 0000664 0000000 0000000 00000001135 14371172257 0020021 0 ustar 00root root 0000000 0000000 MONGO_LOG %{SYSLOGTIMESTAMP:timestamp} \[%{WORD:component}\] %{GREEDYDATA:message}
MONGO_QUERY \{ \{ .* \} ntoreturn: \}
MONGO_WORDDASH \b[\w-]+\b
MONGO_SLOWQUERY %{WORD} %{MONGO_WORDDASH:database}\.%{MONGO_WORDDASH:collection} %{WORD}: %{MONGO_QUERY:query} %{WORD}:%{NONNEGINT:ntoreturn} %{WORD}:%{NONNEGINT:ntoskip} %{WORD}:%{NONNEGINT:nscanned}.*nreturned:%{NONNEGINT:nreturned}..+ %{POSINT:duration}ms
MONGO3_SEVERITY \w
MONGO3_COMPONENT %{WORD}|-
MONGO3_LOG %{TIMESTAMP_ISO8601:timestamp} %{MONGO3_SEVERITY:severity} %{MONGO3_COMPONENT:component}%{SPACE}(?:\[%{DATA:context}\])? %{GREEDYDATA:message} crowdsec-1.4.6/config/patterns/mysql 0000664 0000000 0000000 00000000240 14371172257 0017535 0 ustar 00root root 0000000 0000000 MYSQL_AUTH_FAIL %{TIMESTAMP_ISO8601:time} %{NUMBER} \[Note\] Access denied for user '%{DATA:user}'@'%{IP:source_ip}' \(using password: %{WORD:using_password}\)
crowdsec-1.4.6/config/patterns/nagios 0000664 0000000 0000000 00000022574 14371172257 0017666 0 ustar 00root root 0000000 0000000 ##################################################################################
##################################################################################
# Chop Nagios log files to smithereens!
#
# A set of GROK filters to process logfiles generated by Nagios.
# While it does not, this set intends to cover all possible Nagios logs.
#
# Some more work needs to be done to cover all External Commands:
# http://old.nagios.org/developerinfo/externalcommands/commandlist.php
#
# If you need some support on these rules please contact:
# Jelle Smet http://smetj.net
#
#################################################################################
#################################################################################
NAGIOSTIME \[%{NUMBER:nagios_epoch}\]
###############################################
######## Begin nagios log types
###############################################
NAGIOS_TYPE_CURRENT_SERVICE_STATE CURRENT SERVICE STATE
NAGIOS_TYPE_CURRENT_HOST_STATE CURRENT HOST STATE
NAGIOS_TYPE_SERVICE_NOTIFICATION SERVICE NOTIFICATION
NAGIOS_TYPE_HOST_NOTIFICATION HOST NOTIFICATION
NAGIOS_TYPE_SERVICE_ALERT SERVICE ALERT
NAGIOS_TYPE_HOST_ALERT HOST ALERT
NAGIOS_TYPE_SERVICE_FLAPPING_ALERT SERVICE FLAPPING ALERT
NAGIOS_TYPE_HOST_FLAPPING_ALERT HOST FLAPPING ALERT
NAGIOS_TYPE_SERVICE_DOWNTIME_ALERT SERVICE DOWNTIME ALERT
NAGIOS_TYPE_HOST_DOWNTIME_ALERT HOST DOWNTIME ALERT
NAGIOS_TYPE_PASSIVE_SERVICE_CHECK PASSIVE SERVICE CHECK
NAGIOS_TYPE_PASSIVE_HOST_CHECK PASSIVE HOST CHECK
NAGIOS_TYPE_SERVICE_EVENT_HANDLER SERVICE EVENT HANDLER
NAGIOS_TYPE_HOST_EVENT_HANDLER HOST EVENT HANDLER
NAGIOS_TYPE_EXTERNAL_COMMAND EXTERNAL COMMAND
NAGIOS_TYPE_TIMEPERIOD_TRANSITION TIMEPERIOD TRANSITION
###############################################
######## End nagios log types
###############################################
###############################################
######## Begin external check types
###############################################
NAGIOS_EC_DISABLE_SVC_CHECK DISABLE_SVC_CHECK
NAGIOS_EC_ENABLE_SVC_CHECK ENABLE_SVC_CHECK
NAGIOS_EC_DISABLE_HOST_CHECK DISABLE_HOST_CHECK
NAGIOS_EC_ENABLE_HOST_CHECK ENABLE_HOST_CHECK
NAGIOS_EC_PROCESS_SERVICE_CHECK_RESULT PROCESS_SERVICE_CHECK_RESULT
NAGIOS_EC_PROCESS_HOST_CHECK_RESULT PROCESS_HOST_CHECK_RESULT
NAGIOS_EC_SCHEDULE_SERVICE_DOWNTIME SCHEDULE_SERVICE_DOWNTIME
NAGIOS_EC_SCHEDULE_HOST_DOWNTIME SCHEDULE_HOST_DOWNTIME
NAGIOS_EC_DISABLE_HOST_SVC_NOTIFICATIONS DISABLE_HOST_SVC_NOTIFICATIONS
NAGIOS_EC_ENABLE_HOST_SVC_NOTIFICATIONS ENABLE_HOST_SVC_NOTIFICATIONS
NAGIOS_EC_DISABLE_HOST_NOTIFICATIONS DISABLE_HOST_NOTIFICATIONS
NAGIOS_EC_ENABLE_HOST_NOTIFICATIONS ENABLE_HOST_NOTIFICATIONS
NAGIOS_EC_DISABLE_SVC_NOTIFICATIONS DISABLE_SVC_NOTIFICATIONS
NAGIOS_EC_ENABLE_SVC_NOTIFICATIONS ENABLE_SVC_NOTIFICATIONS
###############################################
######## End external check types
###############################################
NAGIOS_WARNING Warning:%{SPACE}%{GREEDYDATA:nagios_message}
NAGIOS_CURRENT_SERVICE_STATE %{NAGIOS_TYPE_CURRENT_SERVICE_STATE:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{DATA:nagios_statetype};%{DATA:nagios_statecode};%{GREEDYDATA:nagios_message}
NAGIOS_CURRENT_HOST_STATE %{NAGIOS_TYPE_CURRENT_HOST_STATE:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{DATA:nagios_statetype};%{DATA:nagios_statecode};%{GREEDYDATA:nagios_message}
NAGIOS_SERVICE_NOTIFICATION %{NAGIOS_TYPE_SERVICE_NOTIFICATION:nagios_type}: %{DATA:nagios_notifyname};%{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{DATA:nagios_contact};%{GREEDYDATA:nagios_message}
NAGIOS_HOST_NOTIFICATION %{NAGIOS_TYPE_HOST_NOTIFICATION:nagios_type}: %{DATA:nagios_notifyname};%{DATA:nagios_hostname};%{DATA:nagios_state};%{DATA:nagios_contact};%{GREEDYDATA:nagios_message}
NAGIOS_SERVICE_ALERT %{NAGIOS_TYPE_SERVICE_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{DATA:nagios_statelevel};%{NUMBER:nagios_attempt};%{GREEDYDATA:nagios_message}
NAGIOS_HOST_ALERT %{NAGIOS_TYPE_HOST_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{DATA:nagios_statelevel};%{NUMBER:nagios_attempt};%{GREEDYDATA:nagios_message}
NAGIOS_SERVICE_FLAPPING_ALERT %{NAGIOS_TYPE_SERVICE_FLAPPING_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{GREEDYDATA:nagios_message}
NAGIOS_HOST_FLAPPING_ALERT %{NAGIOS_TYPE_HOST_FLAPPING_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{GREEDYDATA:nagios_message}
NAGIOS_SERVICE_DOWNTIME_ALERT %{NAGIOS_TYPE_SERVICE_DOWNTIME_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{GREEDYDATA:nagios_comment}
NAGIOS_HOST_DOWNTIME_ALERT %{NAGIOS_TYPE_HOST_DOWNTIME_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{GREEDYDATA:nagios_comment}
NAGIOS_PASSIVE_SERVICE_CHECK %{NAGIOS_TYPE_PASSIVE_SERVICE_CHECK:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{GREEDYDATA:nagios_comment}
NAGIOS_PASSIVE_HOST_CHECK %{NAGIOS_TYPE_PASSIVE_HOST_CHECK:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{GREEDYDATA:nagios_comment}
NAGIOS_SERVICE_EVENT_HANDLER %{NAGIOS_TYPE_SERVICE_EVENT_HANDLER:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{DATA:nagios_statelevel};%{DATA:nagios_event_handler_name}
NAGIOS_HOST_EVENT_HANDLER %{NAGIOS_TYPE_HOST_EVENT_HANDLER:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{DATA:nagios_statelevel};%{DATA:nagios_event_handler_name}
NAGIOS_TIMEPERIOD_TRANSITION %{NAGIOS_TYPE_TIMEPERIOD_TRANSITION:nagios_type}: %{DATA:nagios_service};%{DATA:nagios_unknown1};%{DATA:nagios_unknown2}
####################
#### External checks
####################
#Disable host & service check
NAGIOS_EC_LINE_DISABLE_SVC_CHECK %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_DISABLE_SVC_CHECK:nagios_command};%{DATA:nagios_hostname};%{DATA:nagios_service}
NAGIOS_EC_LINE_DISABLE_HOST_CHECK %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_DISABLE_HOST_CHECK:nagios_command};%{DATA:nagios_hostname}
#Enable host & service check
NAGIOS_EC_LINE_ENABLE_SVC_CHECK %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_ENABLE_SVC_CHECK:nagios_command};%{DATA:nagios_hostname};%{DATA:nagios_service}
NAGIOS_EC_LINE_ENABLE_HOST_CHECK %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_ENABLE_HOST_CHECK:nagios_command};%{DATA:nagios_hostname}
#Process host & service check
NAGIOS_EC_LINE_PROCESS_SERVICE_CHECK_RESULT %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_PROCESS_SERVICE_CHECK_RESULT:nagios_command};%{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{GREEDYDATA:nagios_check_result}
NAGIOS_EC_LINE_PROCESS_HOST_CHECK_RESULT %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_PROCESS_HOST_CHECK_RESULT:nagios_command};%{DATA:nagios_hostname};%{DATA:nagios_state};%{GREEDYDATA:nagios_check_result}
#Disable host & service notifications
NAGIOS_EC_LINE_DISABLE_HOST_SVC_NOTIFICATIONS %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_DISABLE_HOST_SVC_NOTIFICATIONS:nagios_command};%{GREEDYDATA:nagios_hostname}
NAGIOS_EC_LINE_DISABLE_HOST_NOTIFICATIONS %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_DISABLE_HOST_NOTIFICATIONS:nagios_command};%{GREEDYDATA:nagios_hostname}
NAGIOS_EC_LINE_DISABLE_SVC_NOTIFICATIONS %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_DISABLE_SVC_NOTIFICATIONS:nagios_command};%{DATA:nagios_hostname};%{GREEDYDATA:nagios_service}
#Enable host & service notifications
NAGIOS_EC_LINE_ENABLE_HOST_SVC_NOTIFICATIONS %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_ENABLE_HOST_SVC_NOTIFICATIONS:nagios_command};%{GREEDYDATA:nagios_hostname}
NAGIOS_EC_LINE_ENABLE_HOST_NOTIFICATIONS %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_ENABLE_HOST_NOTIFICATIONS:nagios_command};%{GREEDYDATA:nagios_hostname}
NAGIOS_EC_LINE_ENABLE_SVC_NOTIFICATIONS %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_ENABLE_SVC_NOTIFICATIONS:nagios_command};%{DATA:nagios_hostname};%{GREEDYDATA:nagios_service}
#Schedule host & service downtime
NAGIOS_EC_LINE_SCHEDULE_HOST_DOWNTIME %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_SCHEDULE_HOST_DOWNTIME:nagios_command};%{DATA:nagios_hostname};%{NUMBER:nagios_start_time};%{NUMBER:nagios_end_time};%{NUMBER:nagios_fixed};%{NUMBER:nagios_trigger_id};%{NUMBER:nagios_duration};%{DATA:author};%{DATA:comment}
#End matching line
NAGIOSLOGLINE %{NAGIOSTIME} (?:%{NAGIOS_WARNING}|%{NAGIOS_CURRENT_SERVICE_STATE}|%{NAGIOS_CURRENT_HOST_STATE}|%{NAGIOS_SERVICE_NOTIFICATION}|%{NAGIOS_HOST_NOTIFICATION}|%{NAGIOS_SERVICE_ALERT}|%{NAGIOS_HOST_ALERT}|%{NAGIOS_SERVICE_FLAPPING_ALERT}|%{NAGIOS_HOST_FLAPPING_ALERT}|%{NAGIOS_SERVICE_DOWNTIME_ALERT}|%{NAGIOS_HOST_DOWNTIME_ALERT}|%{NAGIOS_PASSIVE_SERVICE_CHECK}|%{NAGIOS_PASSIVE_HOST_CHECK}|%{NAGIOS_SERVICE_EVENT_HANDLER}|%{NAGIOS_HOST_EVENT_HANDLER}|%{NAGIOS_TIMEPERIOD_TRANSITION}|%{NAGIOS_EC_LINE_DISABLE_SVC_CHECK}|%{NAGIOS_EC_LINE_ENABLE_SVC_CHECK}|%{NAGIOS_EC_LINE_DISABLE_HOST_CHECK}|%{NAGIOS_EC_LINE_ENABLE_HOST_CHECK}|%{NAGIOS_EC_LINE_PROCESS_HOST_CHECK_RESULT}|%{NAGIOS_EC_LINE_PROCESS_SERVICE_CHECK_RESULT}|%{NAGIOS_EC_LINE_SCHEDULE_HOST_DOWNTIME}|%{NAGIOS_EC_LINE_DISABLE_HOST_SVC_NOTIFICATIONS}|%{NAGIOS_EC_LINE_ENABLE_HOST_SVC_NOTIFICATIONS}|%{NAGIOS_EC_LINE_DISABLE_HOST_NOTIFICATIONS}|%{NAGIOS_EC_LINE_ENABLE_HOST_NOTIFICATIONS}|%{NAGIOS_EC_LINE_DISABLE_SVC_NOTIFICATIONS}|%{NAGIOS_EC_LINE_ENABLE_SVC_NOTIFICATIONS}) crowdsec-1.4.6/config/patterns/nginx 0000664 0000000 0000000 00000002047 14371172257 0017522 0 ustar 00root root 0000000 0000000 NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
# '$remote_addr - $remote_user [$time_local] '
# '"$request" $status $body_bytes_sent '
# '"$http_referer" "$http_user_agent"';
# 127.0.0.1 - - [28/Jan/2016:14:19:36 +0300] "GET /zero.html HTTP/1.1" 200 398 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36"
NOTDQUOTE [^"]*
DAY2 \d{2}
#NGINXERRTIME %{YEAR:year}/%{MONTHNUM2:month}/%{DAY2:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second}
NGINXERRTIME %{YEAR}/%{MONTHNUM2}/%{DAY2} %{HOUR}:%{MINUTE}:%{SECOND}
NGINXACCESS %{IPORHOST:remote_addr} - %{NGUSER:remote_user} \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status} %{NUMBER:body_bytes_sent} "%{NOTDQUOTE:http_referer}" "%{NOTDQUOTE:http_user_agent}"
# YYYY/MM/DD HH:MM:SS [LEVEL] PID#TID: *CID MESSAGE
NGINXERROR %{NGINXERRTIME:time} \[%{LOGLEVEL:loglevel}\] %{NONNEGINT:pid}#%{NONNEGINT:tid}: (\*%{NONNEGINT:cid} )?%{GREEDYDATA:message}
crowdsec-1.4.6/config/patterns/paths 0000664 0000000 0000000 00000002272 14371172257 0017516 0 ustar 00root root 0000000 0000000
#DIR ^.*/
#FILE [^/].*$
#URI_SPLIT ^%{GREEDYDATA:request}\?%{GREEDYDATA:http_args}$
#FULLPATH_SPLITTER %{DIR:prefix_directory}%{FILE:file_name}
NAXSI_FMT ^NAXSI_FMT: ip=%{IPORHOST:src_ip}&server=%{IPORHOST:target_ip}&uri=%{PATH:http_path}&learning=\d&vers=%{DATA:naxsi_version}&total_processed=\d+&total_blocked=\d+&block=\d+(&cscore\d=%{WORD:score_label}&score\d=%{INT:score})+&zone0=%{WORD:zone}
#^NAXSI_FMT: ip=%{IPORHOST:src_ip}&server=%{IPORHOST:target_ip}&uri=%{PATH:http_path}&learning=\d&vers=%{DATA:naxsi_version}&total_processed=\d+&total_blocked=\d+&block=\d+(&cscore\d=%{WORD:score_label}&score\d=%{INT:score})+&cscore2
#^NAXSI_FMT: ip=%{IPORHOST:src_ip}&server=%{IPORHOST:target_ip}&uri=%{PATH:http_path}(&cscore\d=%{WORD:score_label}&score\d=%{INT:score})+&cscore2
#^NAXSI_FMT: ip=%{IPORHOST:src_ip}&server=%{IPORHOST:target_ip}&uri=%{PATH:http_path}&learning=\d&vers=%{DATA:naxsi_version}&total_processed=\d+&total_blocked=\d+&block=\d+(&cscore\d=%{WORD:score_label}&score\d=%{INT:score})+&cscore2
NAXSI_EXLOG ^NAXSI_EXLOG: ip=%{IPORHOST:naxsi_src_ip}&server=%{IPORHOST:naxsi_dst_ip}&uri=%{PATH:http_path}&id=%{INT:naxsi_id}&zone=%{WORD:naxsi_zone}&var_name=%{DATA:naxsi_var_name}&content=
crowdsec-1.4.6/config/patterns/postgresql 0000664 0000000 0000000 00000000214 14371172257 0020574 0 ustar 00root root 0000000 0000000 # Default postgresql pg_log format pattern
POSTGRESQL %{DATESTAMP:timestamp} %{TZ} %{DATA:user_id} %{GREEDYDATA:connection_id} %{POSINT:pid} crowdsec-1.4.6/config/patterns/rails 0000664 0000000 0000000 00000001705 14371172257 0017511 0 ustar 00root root 0000000 0000000 RUUID \s{32}
# rails controller with action
RAILS_CONSTROLLER [^#]+
RAIL_ACTION \w+
RCONTROLLER %{RAILS_CONSTROLLER:controller}#%{RAIL_ACTION:action}
# this will often be the only line:
RAILS_TIMESTAMP %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND} %{ISO8601_TIMEZONE}
RAILS3HEAD (?m)Started %{WORD:verb} "%{URIPATHPARAM:request}" for %{IPORHOST:clientip} at %{RAILS_TIMESTAMP:timestamp}
# for some a strange reason, params are stripped of {} - not sure that's a good idea.
RPROCESSING \W*Processing by %{RCONTROLLER} as %{NOTSPACE:format}(?:\W*Parameters: {%{DATA:params}}\W*)?
RAILS3PROFILE (?:\(Views: %{NUMBER:viewms}ms \| ActiveRecord: %{NUMBER:activerecordms}ms|\(ActiveRecord: %{NUMBER:activerecordms}ms)?
RAILS3FOOT Completed %{NUMBER:response}%{DATA} in %{NUMBER:totalms}ms %{RAILS3PROFILE}%{GREEDYDATA}
RAILS_CONTEXT (?:%{DATA}\n)*
# putting it all together
RAILS3 %{RAILS3HEAD}(?:%{RPROCESSING})?%{RAILS_CONTEXT:context}(?:%{RAILS3FOOT})? crowdsec-1.4.6/config/patterns/redis 0000664 0000000 0000000 00000001652 14371172257 0017506 0 ustar 00root root 0000000 0000000
#
# Format 1:
#
# [43569] 27 Aug 12:38:58.471 * RDB: 12 MB of memory used by copy-on-write
#
#
# Format 2:
#
# 31493:M 17 Sep 09:02:54.807 # Server started, Redis version 3.0.2
# 31493:M 17 Sep 09:02:54.807 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm$
# 31493:M 17 Sep 09:02:54.807 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
# 31493:M 17 Sep 09:02:54.807 * DB loaded from disk: 0.000 seconds
# 31493:M 17 Sep 09:02:54.807 * The server is now ready to accept connections on port 6379
#
REDISTIMESTAMP %{MONTHDAY} %{MONTH} %{TIME}
REDISLOG \[%{POSINT:pid}\] %{REDISTIMESTAMP:time} \*\s
REDISLOG1 %{REDISLOG}
REDISLOG2 %{POSINT:pid}:M %{REDISTIMESTAMP:time} [*#] %{GREEDYDATA:message} crowdsec-1.4.6/config/patterns/ruby 0000664 0000000 0000000 00000000267 14371172257 0017362 0 ustar 00root root 0000000 0000000 RUBY_LOGLEVEL DEBUG|FATAL|ERROR|WARN|INFO
RUBY_LOGGER [DFEWI], \[%{TIMESTAMP_ISO8601:timestamp} #%{POSINT:pid}\] *%{RUBY_LOGLEVEL:loglevel} -- +%{DATA:progname}: %{GREEDYDATA:message} crowdsec-1.4.6/config/patterns/smb 0000664 0000000 0000000 00000000255 14371172257 0017157 0 ustar 00root root 0000000 0000000 SMB_AUTH_FAIL Auth:%{GREEDYDATA} user \[%{DATA:smb_domain}\]\\\[%{DATA:user}\]%{GREEDYDATA} status \[NT_STATUS_NO_SUCH_USER\]%{GREEDYDATA} remote host \[ipv4:%{IP:ip_source} crowdsec-1.4.6/config/patterns/ssh 0000664 0000000 0000000 00000013076 14371172257 0017200 0 ustar 00root root 0000000 0000000 # sshd grok pattern
# Start/Stop
SSHD_LISTEN Server listening on %{IP:sshd_listen_ip} port %{NUMBER:sshd_listen_port}.
SSHD_TERMINATE Received signal %{NUMBER:sshd_signal}; terminating.
# SSH Tunnel
SSHD_TUNN_ERR1 error: connect_to %{IP:sshd_listen_ip} port %{NUMBER:sshd_listen_port}: failed.
SSHD_TUNN_ERR2 error: channel_setup_fwd_listener: cannot listen to port: %{NUMBER:sshd_listen_port}
SSHD_TUNN_ERR3 error: bind: Address already in use
SSHD_TUNN_ERR4 error: channel_setup_fwd_listener_tcpip: cannot listen to port: %{NUMBER:sshd_listen_port}
SSHD_TUNN_TIMEOUT Timeout, client not responding.
# Normal
SSHD_SUCCESS Accepted %{WORD:sshd_auth_type} for %{USERNAME:sshd_user} from %{IP:sshd_client_ip} port %{NUMBER:sshd_port} %{WORD:sshd_protocol}: %{GREEDYDATA:sshd_cipher}
SSHD_DISCONNECT Received disconnect from %{IP:sshd_client_ip} port %{NUMBER:sshd_port}:%{NUMBER:sshd_disconnect_code}: %{GREEDYDATA:sshd_disconnect_status}
SSHD_CONN_CLOSE Connection closed by %{IP:sshd_client_ip}$
SSHD_SESSION_OPEN pam_unix\(sshd:session\): session opened for user %{USERNAME:sshd_user} by \(uid=\d+\)
SSHD_SESSION_CLOSE pam_unix\(sshd:session\): session closed for user %{USERNAME:sshd_user}
SSHD_SESSION_FAIL pam_systemd\(sshd:session\): Failed to release session: %{GREEDYDATA:sshd_disconnect_status}
SSHD_LOGOUT_ERR syslogin_perform_logout: logout\(\) returned an error
# Probe
SSHD_REFUSE_CONN refused connect from %{DATA:sshd_client_hostname} \(%{IPORHOST:sshd_client_ip}\)
SSHD_TCPWRAP_FAIL1 warning: %{DATA:sshd_tcpd_file}, line %{NUMBER}: can't verify hostname: getaddrinfo\(%{DATA:sshd_paranoid_hostname}, %{DATA:sshd_sa_family}\) failed
SSHD_TCPWRAP_FAIL2 warning: %{DATA:sshd_tcpd_file}, line %{NUMBER}: host name/address mismatch: %{IPORHOST:sshd_client_ip} != %{HOSTNAME:sshd_paranoid_hostname}
SSHD_TCPWRAP_FAIL3 warning: %{DATA:sshd_tcpd_file}, line %{NUMBER}: host name/name mismatch: %{HOSTNAME:sshd_paranoid_hostname_1} != %{HOSTNAME:sshd_paranoid_hostname_2}
SSHD_TCPWRAP_FAIL4 warning: %{DATA:sshd_tcpd_file}, line %{NUMBER}: host name/name mismatch: reverse lookup results in non-FQDN %{HOSTNAME:sshd_paranoid_hostname}
SSHD_TCPWRAP_FAIL5 warning: can't get client address: Connection reset by peer
SSHD_FAIL Failed %{WORD:sshd_auth_type} for %{USERNAME:sshd_invalid_user} from %{IP:sshd_client_ip} port %{NUMBER:sshd_port} %{WORD:sshd_protocol}
SSHD_USER_FAIL Failed password for invalid user %{USERNAME:sshd_invalid_user} from %{IP:sshd_client_ip} port %{NUMBER:sshd_port} %{WORD:sshd_protocol}
SSHD_INVAL_USER Invalid user\s*%{USERNAME:sshd_invalid_user}? from %{IP:sshd_client_ip}
# preauth
SSHD_DISC_PREAUTH Disconnected from %{IP:sshd_client_ip} port %{NUMBER:sshd_port}\s*(?:\[%{GREEDYDATA:sshd_privsep}\]|)
SSHD_MAXE_PREAUTH error: maximum authentication attempts exceeded for (?:invalid user |)%{USERNAME:sshd_invalid_user} from %{IP:sshd_client_ip} port %{NUMBER:sshd_port} %{WORD:sshd_protocol}\s*(?:\[%{GREEDYDATA:sshd_privsep}\]|)
SSHD_DISR_PREAUTH Disconnecting: %{GREEDYDATA:sshd_disconnect_status} \[%{GREEDYDATA:sshd_privsep}\]
SSHD_INVA_PREAUTH input_userauth_request: invalid user %{USERNAME:sshd_invalid_user}?\s*(?:\[%{GREEDYDATA:sshd_privsep}\]|)
SSHD_REST_PREAUTH Connection reset by %{IP:sshd_client_ip} port %{NUMBER:sshd_port}\s*(?:\[%{GREEDYDATA:sshd_privsep}\]|)
SSHD_CLOS_PREAUTH Connection closed by %{IP:sshd_client_ip} port %{NUMBER:sshd_port}\s*(?:\[%{GREEDYDATA:sshd_privsep}\]|)
SSHD_FAIL_PREAUTH fatal: Unable to negotiate with %{IP:sshd_client_ip} port %{NUMBER:sshd_port}:\s*%{GREEDYDATA:sshd_disconnect_status}? \[%{GREEDYDATA:sshd_privsep}\]
SSHD_FAI2_PREAUTH fatal: %{GREEDYDATA:sshd_fatal_status}: Connection from %{IP:sshd_client_ip} port %{NUMBER:sshd_port}:\s*%{GREEDYDATA:sshd_disconnect_status}? \[%{GREEDYDATA:sshd_privsep}\]
SSHD_BADL_PREAUTH Bad packet length %{NUMBER:sshd_packet_length}. \[%{GREEDYDATA:sshd_privsep}\]
# Corrupted
SSHD_IDENT_FAIL Did not receive identification string from %{IP:sshd_client_ip}
SSHD_MAPB_FAIL Address %{IP:sshd_client_ip} maps to %{HOSTNAME:sshd_client_hostname}, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
SSHD_RMAP_FAIL reverse mapping checking getaddrinfo for %{HOSTNAME:sshd_client_hostname} \[%{IP:sshd_client_ip}\] failed - POSSIBLE BREAK-IN ATTEMPT!
SSHD_TOOMANY_AUTH Disconnecting: Too many authentication failures for %{USERNAME:sshd_invalid_user}
SSHD_CORRUPT_MAC Corrupted MAC on input
SSHD_PACKET_CORRUPT Disconnecting: Packet corrupt
SSHD_BAD_VERSION Bad protocol version identification '%{GREEDYDATA}' from %{IP:sshd_client_ip}
####
SSHD_INIT %{SSHD_LISTEN}|%{SSHD_TERMINATE}
SSHD_TUNN %{SSHD_TUNN_ERR1}|%{SSHD_TUNN_ERR2}|%{SSHD_TUNN_ERR3}|%{SSHD_TUNN_ERR4}|%{SSHD_TUNN_TIMEOUT}
SSHD_NORMAL_LOG %{SSHD_SUCCESS}|%{SSHD_DISCONNECT}|%{SSHD_CONN_CLOSE}|%{SSHD_SESSION_OPEN}|%{SSHD_SESSION_CLOSE}|%{SSHD_SESSION_FAIL}|%{SSHD_LOGOUT_ERR}
SSHD_PROBE_LOG %{SSHD_REFUSE_CONN}|%{SSHD_TCPWRAP_FAIL1}|%{SSHD_TCPWRAP_FAIL2}|%{SSHD_TCPWRAP_FAIL3}|%{SSHD_TCPWRAP_FAIL4}|%{SSHD_TCPWRAP_FAIL5}|%{SSHD_FAIL}|%{SSHD_USER_FAIL}|%{SSHD_INVAL_USER}
SSHD_PREAUTH %{SSHD_DISC_PREAUTH}|%{SSHD_MAXE_PREAUTH}|%{SSHD_DISR_PREAUTH}|%{SSHD_INVA_PREAUTH}|%{SSHD_REST_PREAUTH}|%{SSHD_FAIL_PREAUTH}|%{SSHD_CLOS_PREAUTH}|%{SSHD_FAI2_PREAUTH}|%{SSHD_BADL_PREAUTH}
SSHD_CORRUPTED %{SSHD_IDENT_FAIL}|%{SSHD_MAPB_FAIL}|%{SSHD_RMAP_FAIL}|%{SSHD_TOOMANY_AUTH}|%{SSHD_CORRUPT_MAC}|%{SSHD_PACKET_CORRUPT}|%{SSHD_BAD_VERSION}
SSHD_LOG %{SSHD_INIT}|%{SSHD_NORMAL_LOG}|%{SSHD_PROBE_LOG}|%{SSHD_CORRUPTED}|%{SSHD_TUNN}|%{SSHD_PREAUTH}
crowdsec-1.4.6/config/patterns/tcpdump 0000664 0000000 0000000 00000000243 14371172257 0020047 0 ustar 00root root 0000000 0000000 TCPDUMP_OUTPUT %{GREEDYDATA:timestamp} IP %{IPORHOST:source_ip}\.%{INT:source_port} > %{IPORHOST:dest_ip}\.%{INT:dest_port}: Flags \[%{GREEDYDATA:tcpflags}\], seq
crowdsec-1.4.6/config/profiles.yaml 0000664 0000000 0000000 00000001321 14371172257 0017315 0 ustar 00root root 0000000 0000000 name: default_ip_remediation
#debug: true
filters:
- Alert.Remediation == true && Alert.GetScope() == "Ip"
decisions:
- type: ban
duration: 4h
#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4)
# notifications:
# - slack_default # Set the webhook in /etc/crowdsec/notifications/slack.yaml before enabling this.
# - splunk_default # Set the splunk url and token in /etc/crowdsec/notifications/splunk.yaml before enabling this.
# - http_default # Set the required http parameters in /etc/crowdsec/notifications/http.yaml before enabling this.
# - email_default # Set the required email parameters in /etc/crowdsec/notifications/email.yaml before enabling this.
on_success: break
crowdsec-1.4.6/config/simulation.yaml 0000664 0000000 0000000 00000000071 14371172257 0017657 0 ustar 00root root 0000000 0000000 simulation: off
# exclusions:
# - crowdsecurity/ssh-bf
crowdsec-1.4.6/config/user.yaml 0000664 0000000 0000000 00000001737 14371172257 0016463 0 ustar 00root root 0000000 0000000 common:
daemonize: false
pid_dir: /var/run/
log_media: stdout
log_level: info
log_dir: /var/log/
working_dir: .
config_paths:
config_dir: /etc/crowdsec/
data_dir: /var/lib/crowdsec/data
#simulation_path: /etc/crowdsec/config/simulation.yaml
#hub_dir: /etc/crowdsec/hub/
#index_path: ./config/hub/.index.json
crowdsec_service:
#acquisition_path: ./config/acquis.yaml
parser_routines: 1
cscli:
output: human
db_config:
type: sqlite
db_path: /var/lib/crowdsec/data/crowdsec.db
user: crowdsec
#log_level: info
password: crowdsec
db_name: crowdsec
host: "127.0.0.1"
port: 3306
api:
client:
insecure_skip_verify: false # default true
credentials_path: /etc/crowdsec/local_api_credentials.yaml
server:
#log_level: info
listen_uri: 127.0.0.1:8080
profiles_path: /etc/crowdsec/profiles.yaml
online_client: # Central API
credentials_path: /etc/crowdsec/online_api_credentials.yaml
prometheus:
enabled: true
level: full
crowdsec-1.4.6/debian/ 0000775 0000000 0000000 00000000000 14371172257 0014566 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/debian/.gitignore 0000664 0000000 0000000 00000000130 14371172257 0016550 0 ustar 00root root 0000000 0000000 # Generated during the build
/crowdsec
/files
/*.substvars
/*.log
/*.debhelper
/*-stamp
crowdsec-1.4.6/debian/README.md 0000664 0000000 0000000 00000000650 14371172257 0016046 0 ustar 00root root 0000000 0000000
# Building Debian/Ubuntu packages
It is not recommended to build your own packages for production environments.
However, if you want to experiment and contribute:
* Update the changelog (at least give it a correct version number)
* Run "QUILT_PATCHES=debian/patches quilt push -a && quilt refresh"
We do the above in the build pipeline, so you'll have to do it manually before running:
* dpkg-buildpackage -uc -us -b
crowdsec-1.4.6/debian/changelog 0000664 0000000 0000000 00000004347 14371172257 0016450 0 ustar 00root root 0000000 0000000 crowdsec (1.0.13-3) UNRELEASED; urgency=medium
* Fix small stuff
-- Manuel Sabban Wed, 26 Apr 2021 09:30:14 +0100
crowdsec (1.0.13-2) UNRELEASED; urgency=medium
* Fix version advertisement
-- Manuel Sabban Wed, 26 Apr 2021 09:30:14 +0100
crowdsec (1.0.13-1) UNRELEASED; urgency=medium
* Bump to 1.0.13
-- Manuel Sabban Wed, 26 Apr 2021 09:30:14 +0100
crowdsec (1.0.12-1) UNRELEASED; urgency=medium
* Bump to 1.0.12
-- Manuel Sabban Wed, 07 Apr 2021 15:10:11 +0100
crowdsec (1.0.9-5) UNRELEASED; urgency=medium
* Fix cleanup on remove --purge
-- Manuel Sabban Wed, 23 Mar 2021 15:17:09 +0100
crowdsec (1.0.9-4) UNRELEASED; urgency=medium
* Fix a bunch a bugs in package management
-- Manuel Sabban Wed, 23 Mar 2021 15:17:09 +0100
crowdsec (1.0.9-3) UNRELEASED; urgency=medium
* truly fix error generating acquis.yaml
-- Manuel Sabban Wed, 10 Feb 2021 10:41:06 +0100
crowdsec (1.0.9-2) UNRELEASED; urgency=medium
* fix error generating acquis.yaml
-- Manuel Sabban Wed, 10 Feb 2021 10:41:06 +0100
crowdsec (1.0.9-1) UNRELEASED; urgency=medium
* bump to 1.0.9
-- Manuel Sabban Wed, 10 Feb 2021 10:41:06 +0100
crowdsec (1.0.7-4) UNRELEASED; urgency=medium
* fix crowdsec -version output
-- Manuel Sabban Wed, 10 Feb 2021 10:41:06 +0100
crowdsec (1.0.7-3) UNRELEASED; urgency=medium
* fix error when docker metabase doesn't exist
* fix crowdsec not stopped when uninstalling
-- Manuel Sabban Wed, 10 Feb 2021 10:41:06 +0100
crowdsec (1.0.7-2) UNRELEASED; urgency=medium
* remove stop metabase docker container
* purge removes the metabase container once and for all
-- Manuel Sabban Wed, 10 Feb 2021 10:41:06 +0100
crowdsec (1.0.7-1) UNRELEASED; urgency=medium
* debian package
* bump to 1.0.7
-- Manuel Sabban Wed, 10 Feb 2021 10:41:06 +0100
crowdsec (1.0.4-1) UNRELEASED; urgency=medium
* debian package
* bump to 1.0.4
-- Manuel Sabban Mon, 08 Feb 2021 09:38:06 +0100
crowdsec-1.4.6/debian/compat 0000664 0000000 0000000 00000000003 14371172257 0015765 0 ustar 00root root 0000000 0000000 11
crowdsec-1.4.6/debian/control 0000664 0000000 0000000 00000000530 14371172257 0016167 0 ustar 00root root 0000000 0000000 Source: crowdsec
Maintainer: Crowdsec Team
Build-Depends: debhelper, jq, bash, git
Package: crowdsec
Architecture: any
Description: Crowdsec - An open-source, lightweight agent to detect and respond to bad behaviours. It also automatically benefits from our global community-wide IP reputation database
Depends: coreutils crowdsec-1.4.6/debian/crowdsec.cron.daily 0000777 0000000 0000000 00000000000 14371172257 0025633 2../config/crowdsec.cron.daily ustar 00root root 0000000 0000000 crowdsec-1.4.6/debian/crowdsec.service 0000664 0000000 0000000 00000000627 14371172257 0017766 0 ustar 00root root 0000000 0000000 [Unit]
Description=Crowdsec agent
After=syslog.target network.target remote-fs.target nss-lookup.target
[Service]
Type=notify
Environment=LC_ALL=C LANG=C
ExecStartPre=/usr/bin/crowdsec -c /etc/crowdsec/config.yaml -t
ExecStart=/usr/bin/crowdsec -c /etc/crowdsec/config.yaml
#ExecStartPost=/bin/sleep 0.1
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
RestartSec=60
[Install]
WantedBy=multi-user.target
crowdsec-1.4.6/debian/install 0000664 0000000 0000000 00000001003 14371172257 0016151 0 ustar 00root root 0000000 0000000 config/config.yaml etc/crowdsec/
config/profiles.yaml etc/crowdsec/
config/simulation.yaml etc/crowdsec/
config/patterns/* etc/crowdsec/patterns
config/crowdsec.service lib/systemd/system
# Referenced configs:
plugins/notifications/slack/slack.yaml etc/crowdsec/notifications/
plugins/notifications/http/http.yaml etc/crowdsec/notifications/
plugins/notifications/splunk/splunk.yaml etc/crowdsec/notifications/
plugins/notifications/email/email.yaml etc/crowdsec/notifications/
crowdsec-1.4.6/debian/patches/ 0000775 0000000 0000000 00000000000 14371172257 0016215 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/debian/patches/config_plugins 0000664 0000000 0000000 00000000774 14371172257 0021156 0 ustar 00root root 0000000 0000000 Index: crowdsec/config/config.yaml
===================================================================
--- crowdsec.orig/config/config.yaml
+++ crowdsec/config/config.yaml
@@ -12,7 +12,7 @@ config_paths:
hub_dir: /etc/crowdsec/hub/
index_path: /etc/crowdsec/hub/.index.json
notification_dir: /etc/crowdsec/notifications/
- plugin_dir: /usr/local/lib/crowdsec/plugins/
+ plugin_dir: /usr/lib/crowdsec/plugins/
crowdsec_service:
acquisition_path: /etc/crowdsec/acquis.yaml
parser_routines: 1
crowdsec-1.4.6/debian/patches/series 0000664 0000000 0000000 00000000017 14371172257 0017430 0 ustar 00root root 0000000 0000000 config_plugins
crowdsec-1.4.6/debian/postinst 0000664 0000000 0000000 00000006445 14371172257 0016405 0 ustar 00root root 0000000 0000000 #!/bin/bash
COLLECTIONS=false
set -e
# Source debconf library.
. /usr/share/debconf/confmodule
if [ "$1" = configure ]; then
if [[ ! -d /var/lib/crowdsec/data ]]; then
mkdir -p /var/lib/crowdsec/data
fi
if [[ -d /var/lib/crowdsec/backup ]]; then
cscli config restore /var/lib/crowdsec/backup/backup.config
rm -rf /var/lib/crowdsec/backup
/usr/bin/cscli hub update
/usr/bin/cscli hub upgrade
systemctl start crowdsec
fi
. /usr/share/crowdsec/wizard.sh -n
if ! [[ -f /etc/crowdsec/acquis.yaml ]]; then
echo Creating /etc/crowdsec/acquis.yaml
set +e
SILENT=true detect_services
SILENT=true TMP_ACQUIS_FILE_SKIP=skip genacquisition
set -e
COLLECTIONS=true
fi
if [[ -f /etc/crowdsec/local_api_credentials.yaml ]] ; then
chmod 600 /etc/crowdsec/local_api_credentials.yaml
fi
if [[ -f /etc/crowdsec/online_api_credentials.yaml ]]; then
chmod 600 /etc/crowdsec/online_api_credentials.yaml
fi
if [[ ! -f /etc/crowdsec/local_api_credentials.yaml ]] || [[ ! -f /etc/crowdsec/online_api_credentials.yaml ]]; then
if [[ ! -f /etc/crowdsec/local_api_credentials.yaml ]] ; then
install -m 600 /dev/null /etc/crowdsec/local_api_credentials.yaml
fi
if [[ ! -f /etc/crowdsec/online_api_credentials.yaml ]] ; then
install -m 600 /dev/null /etc/crowdsec/online_api_credentials.yaml
fi
db_input medium crowdsec/lapi || true
db_go || true
db_get crowdsec/lapi
LAPI=$RET
if [ "$LAPI" = true ]; then
db_input medium crowdsec/capi || true
db_go || true
db_get crowdsec/capi
CAPI=$RET
cscli machines add -a
if [ "$CAPI" = true ]; then
cscli capi register
fi
else
db_input medium crowdsec/lapi_host || true
db_go || true
db_get crowdsec/lapi_host
LAPI_HOST=$RET
sed -i "s/127.0.0.1:8080/$LAPI_HOST/g" /etc/crowdsec/config.yaml
fi
fi
echo Updating hub
/usr/bin/cscli hub update
if [ "$COLLECTIONS" = true ]; then
set +e
CSCLI_BIN_INSTALLED="/usr/bin/cscli" SILENT=true install_collection
set -e
fi
if [[ -f /var/lib/crowdsec/data/crowdsec.db.backup ]]; then
cp /var/lib/crowdsec/data/crowdsec.db.backup /var/lib/crowdsec/data/crowdsec.db
rm -f /var/lib/crowdsec/data/crowdsec.db.backup
fi
systemctl --quiet is-enabled crowdsec || systemctl unmask crowdsec && systemctl enable crowdsec
API=$(cscli config show --key "Config.API.Server")
if [ "$API" = "" ] ; then
LAPI=false
else
PORT=$(cscli config show --key "Config.API.Server.ListenURI"|cut -d ":" -f2)
fi
if [ "$LAPI" = false ] || [ -z "$(ss -nlt "sport = ${PORT}" | grep -v ^State)" ] ; then
systemctl start crowdsec
else
echo "Not attempting to start crowdsec, port ${PORT} is already used or lapi was disabled"
echo "This port is configured through /etc/crowdsec/config.yaml and /etc/crowdsec/local_api_credentials.yaml"
fi
fi
echo "You can always run the configuration again interactively by using '/usr/share/crowdsec/wizard.sh -c"
crowdsec-1.4.6/debian/postrm 0000664 0000000 0000000 00000000227 14371172257 0016036 0 ustar 00root root 0000000 0000000 if [ "$1" = "purge" ]; then
find /etc/crowdsec -maxdepth 1 -mindepth 1 | grep -v "bouncer" | xargs rm -rf || echo ""
rm -rf /var/lib/crowdsec
fi
crowdsec-1.4.6/debian/preinst 0000664 0000000 0000000 00000002457 14371172257 0016205 0 ustar 00root root 0000000 0000000 #!/bin/bash
set -e
# Source debconf library.
. /usr/share/debconf/confmodule
OLD_MAJOR_VERSION=$(echo $2 | cut -d'.' -f1)
OLD_MINOR_VERSION=$(echo $2 | cut -d'.' -f2)
OLD_PATCH_VERSION=$(echo $2 | cut -d'.' -f3|cut -d'-' -f1)
NEW_MAJOR_VERSION=$(echo $3 | cut -d'.' -f1)
NEW_MINOR_VERSION=$(echo $3 | cut -d'.' -f2)
NEW_PATCH_VERSION=$(echo $3 | cut -d'.' -f3|cut -d'-' -f1)
if [ "$1" = upgrade ]; then
OLD_MAJOR_VERSION=$(echo $2 | cut -d'.' -f1)
OLD_MINOR_VERSION=$(echo $2 | cut -d'.' -f2)
OLD_PATCH_VERSION=$(echo $2 | cut -d'.' -f3|cut -d'-' -f1)
NEW_MAJOR_VERSION=$(echo $3 | cut -d'.' -f1)
NEW_MINOR_VERSION=$(echo $3 | cut -d'.' -f2)
NEW_PATCH_VERSION=$(echo $3 | cut -d'.' -f3|cut -d'-' -f1)
if [[ $OLD_MAJOR_VERSION -eq "1" ]] && [[ $OLD_MINOR_VERSION -eq "0" ]] && [[ $OLD_PATCH_VERSION -lt "9" ]]; then
if [[ -f /var/lib/crowdsec/data/crowdsec.db ]]; then
cp /var/lib/crowdsec/data/crowdsec.db /var/lib/crowdsec/data/crowdsec.db.backup
fi
fi
if [[ $NEW_MAJOR_VERSION -gt $OLD_MAJOR_VERSION ]]; then
echo "Stopping crowdsec"
systemctl stop crowdsec || true
cscli config backup /var/lib/crowdsec/backup
fi
fi
echo "You can always run the configuration again interactively by using '/usr/share/crowdsec/wizard.sh -c"
crowdsec-1.4.6/debian/prerm 0000664 0000000 0000000 00000000275 14371172257 0015642 0 ustar 00root root 0000000 0000000 if [ "$1" = "remove" ]; then
cscli dashboard remove -f -y || true
systemctl stop crowdsec
systemctl disable crowdsec
fi
if [ "$1" = "upgrade" ]; then
systemctl stop crowdsec
fi crowdsec-1.4.6/debian/rules 0000775 0000000 0000000 00000004477 14371172257 0015662 0 ustar 00root root 0000000 0000000 #!/usr/bin/make -f
export DEB_VERSION=$(shell dpkg-parsechangelog | egrep '^Version:' | cut -f 2 -d ' ')
export BUILD_VERSION=v${DEB_VERSION}-debian-pragmatic
export GO111MODULE=on
# LD_OPTS=-ldflags "-s -w -X github.com/crowdsecurity/crowdsec/pkg/cwversion.Version=$(BUILD_VERSION) \
# -X github.com/crowdsecurity/crowdsec/pkg/cwversion.BuildDate=$(BUILD_TIMESTAMP) \
# -X github.com/crowdsecurity/crowdsec/pkg/cwversion.Codename=$(BUILD_CODENAME) \
# -X github.com/crowdsecurity/crowdsec/pkg/cwversion.Tag=$(BUILD_TAG) \
# -X github.com/crowdsecurity/crowdsec/pkg/cwversion.GoVersion=$(BUILD_GOVERSION)"
%:
dh $@
override_dh_auto_clean:
override_dh_auto_test:
override_dh_auto_build:
override_dh_auto_install:
# mkdir /tmp/go
# echo $(go version)
# echo $($GOCMD version)
# cd cmd/crowdsec && GOROOT=/tmp/go GO111MODULE=on $(GOBUILD) $(LD_OPTS) -o $(CROWDSEC_BIN) -v && cd ..
# cd cmd/crowdsec-cli && GOROOT=/tmp/go GO111MODULE=on $(GOBUILD) $(LD_OPTS) -o cscli -v && cd ..
make build
mkdir -p debian/crowdsec/usr/bin
mkdir -p debian/crowdsec/etc/crowdsec
mkdir -p debian/crowdsec/usr/share/crowdsec
mkdir -p debian/crowdsec/etc/crowdsec/hub/
mkdir -p debian/crowdsec/usr/share/crowdsec/config
mkdir -p debian/crowdsec/usr/lib/crowdsec/plugins/
mkdir -p debian/crowdsec/etc/crowdsec/notifications/
install -m 551 plugins/notifications/slack/notification-slack debian/crowdsec/usr/lib/crowdsec/plugins/
install -m 551 plugins/notifications/http/notification-http debian/crowdsec/usr/lib/crowdsec/plugins/
install -m 551 plugins/notifications/splunk/notification-splunk debian/crowdsec/usr/lib/crowdsec/plugins/
install -m 551 plugins/notifications/email/notification-email debian/crowdsec/usr/lib/crowdsec/plugins/
cp cmd/crowdsec/crowdsec debian/crowdsec/usr/bin
cp cmd/crowdsec-cli/cscli debian/crowdsec/usr/bin
cp wizard.sh debian/crowdsec/usr/share/crowdsec
install -m 600 config/config.yaml debian/crowdsec/etc/crowdsec/config.yaml
cp config/simulation.yaml debian/crowdsec/etc/crowdsec/simulation.yaml
cp config/profiles.yaml debian/crowdsec/etc/crowdsec/profiles.yaml
cp config/console.yaml debian/crowdsec/etc/crowdsec/console.yaml
cp -a config/patterns debian/crowdsec/etc/crowdsec
override_dh_fixperms:
dh_fixperms
chmod 600 debian/crowdsec/etc/crowdsec/notifications/*
chmod 600 debian/crowdsec/etc/crowdsec/config.yaml
crowdsec-1.4.6/debian/templates 0000664 0000000 0000000 00000001342 14371172257 0016507 0 ustar 00root root 0000000 0000000 Template: crowdsec/lapi
Type: boolean
Default: true
Description: Do you want to run the local API server ?
A local API is required to run crowdsec, but another installation can be used.
.
If you don't know what to do, answer yes.
Template: crowdsec/lapi_host
Type: string
Default: 127.0.0.1:8080
Description: Address of the local API server
A local API is required to run crowdsec, but another installation can be used.
.
Please add the address of the local API server
Template: crowdsec/capi
Type: boolean
Default: true
Description: Do you want to the centralized remote API server ?
To share information with other crowdsec you can register to the centralized remote API server.
.
If you don't know what to do, answer yes.
crowdsec-1.4.6/docker/ 0000775 0000000 0000000 00000000000 14371172257 0014613 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/docker/README.md 0000664 0000000 0000000 00000031477 14371172257 0016106 0 ustar 00root root 0000000 0000000 # Quick reference
* Documentation and help: https://docs.crowdsec.net/
* Crowdsec concepts: https://docs.crowdsec.net/docs/concepts
* Where to file issues: https://github.com/crowdsecurity/crowdsec
# What is Crowdsec
Crowdsec - An open-source, lightweight agent to detect and respond to bad behaviours. It also automatically benefits from our global community-wide IP reputation database.
# How to use this image
## Image flavors
All the following images are available on Docker Hub for the architectures
386, amd64, arm/v6, arm/v7, arm64.
### Alpine
- `crowdsecurity/crowdsec:{version}`
Recommended for production usage. Also available on GitHub (ghrc.io).
- `crowdsecurity/crowdsec:latest`
For development and testing.
since v1.4.2:
- `crowdsecurity/crowdsec:slim`
Reduced size by 60%, does not include notifier plugins nor the GeoIP database.
If you need these details on decisions, running `cscli hub upgrade` inside the
container downloads the GeoIP database at runtime.
### Debian (since v1.3.3)
- `crowdsecurity/crowdsec:{version}-debian`
- `crowdsecurity/crowdsec:latest-debian`
The debian version includes support for systemd and journalctl.
### Custom
You can build your images with Dockerfile and Dockerfile-debian.
For example, if you want a Debian version without plugin notifiers:
```console
$ docker build -f Dockerfile.debian --build-arg=BUILD_ENV=slim
```
supported values for BUILD_ENV are: full, with-geoip, with-plugins, slim.
## Required configuration
### Journalctl (only for debian image)
To use journalctl as a log stream, with or without the `DSN` environment variable, it's important to mount the journal log from the host to the container itself.
This can be done by adding the following volume mount to the docker command:
```
-v /var/log/journal:/run/log/journal
```
### Logs ingestion and processing
Collections are a good place to start: https://docs.crowdsec.net/docs/collections/intro
Find collections, scenarios, parsers and postoverflows in the hub: https://hub.crowdsec.net
* Specify collections | scenarios | parsers | postoverflows to install via the environment variables (by default [`crowdsecurity/linux`](https://hub.crowdsec.net/author/crowdsecurity/collections/linux) is installed)
* Mount volumes to specify your log files that should be ingested by crowdsec
### Acquisition
`/etc/crowdsec/acquis.yaml` maps logs to the provided parsers. Find out more here: https://docs.crowdsec.net/docs/concepts/#acquisition
acquis.yaml example:
```shell
filenames:
- /logs/auth.log
- /logs/syslog
labels:
type: syslog
---
filename: /logs/apache2/*.log
labels:
type: apache2
```
`labels.type`: use `syslog` if the logs come from syslog, otherwise check the collection's documentation for the relevant type.
## Recommended configuration
### Volumes
We strongly suggest mounting **named volumes** for Crowdsec configuration and database to avoid credentials and decisions loss in case of container destruction and recreation, version update, etc.
* Credentials and configuration: `/etc/crowdsec`
* Database when using SQLite (default): `/var/lib/crowdsec/data`
## Start a Crowdsec instance
```shell
docker run -d \
-v local_path_to_crowdsec_config/acquis.yaml:/etc/crowdsec/acquis.yaml \
-v crowdsec_config:/etc/crowdsec \
-v crowdsec_data:/var/lib/crowdsec/data \
-v /var/log/auth.log:/logs/auth.log:ro \
-v /var/log/syslog.log:/logs/syslog.log:ro \
-v /var/log/apache:/logs/apache:ro \
-e COLLECTIONS="crowdsecurity/apache2 crowdsecurity/sshd" \
-p 8080:8080 -p 6060:6060 \
--name crowdsec crowdsecurity/crowdsec
```
## ... or docker-compose
Check this full-stack example using docker-compose: https://github.com/crowdsecurity/example-docker-compose
# How to extend this image
## Full configuration
The container is built with a specific docker
[configuration](https://github.com/crowdsecurity/crowdsec/blob/master/docker/config.yaml).
If you need to change it and the docker variables (see below) are not enough,
you can bind `/etc/crowdsec/config.yaml` to your a configuration file.
## Notifications
If you wish to use the [notification system](https://docs.crowdsec.net/docs/notification_plugins/intro), you will need to mount at least a custom `profiles.yaml` and a notification configuration to `/etc/crowdsec/notifications`
# Deployment use cases
Crowdsec is composed of an `agent` that parses logs and creates `alerts`, and a
`local API (LAPI)` that transforms these alerts into decisions. Both functions
are provided by the same executables, so the agent and the LAPI can run in the
same or separate containers. In complex configurations, it makes sense to have
agents on each machine that runs the protected applications, and a LAPI that
gathers all signals from agents and communicates with the `central API`.
## Register a new agent with LAPI
Without TLS authentication:
```shell
docker exec -it crowdsec_lapi_container_name cscli machines add agent_user_name --password agent_password
```
With TLS authentication:
Agents are automatically registered and don't need a username or password. The
agents' names are derived from the IP address from which they connect.
## Run an agent connected to LAPI
Add the following environment variables to the docker run command:
* `DISABLE_LOCAL_API=true`
* `AGENT_USERNAME="agent_user_name"` - agent_user_name previously registered with LAPI
* `AGENT_PASSWORD="agent_password"` - agent_password previously registered with LAPI
* `LOCAL_API_URL="http://LAPI_host:LAPI_port"`
# Next steps
## Bouncers
Crowdsec being a detection component, the remediation is implemented using `bouncers`. Each bouncer protects a specific component. Find out more:
https://hub.crowdsec.net/browse/#bouncers
https://docs.crowdsec.net/docs/user_guides/bouncers_configuration/
### Automatic Bouncer Registration
Without TLS authentication:
You can register bouncers with the crowdsec container at startup, using environment variables or Docker secrets. You cannot use this process to update an existing bouncer without first deleting it.
To use environment variables, they should be in the format `BOUNCER_KEY_=`. e.g. `BOUNCER_KEY_nginx=mysecretkey12345`.
To use Docker secrets, the secret should be named `bouncer_key_` with a content of ``. e.g. `bouncer_key_nginx` with content `mysecretkey12345`.
A bouncer key can be any string but we recommend an alphanumeric value for consistency with the crowdsec-generated keys and to avoid problems with escaping special characters.
With TLS authentication:
Bouncers are automatically registered and don't need an API key. The
bouncers' names are derived from the IP address from which they connect.
## Console
We provide a web-based interface to get more from Crowdsec: https://docs.crowdsec.net/docs/console
Subscribe here: https://app.crowdsec.net
# Caveats
Using binds rather than named volumes ([complete explanation here](https://docs.docker.com/storage/volumes/)) results in more complexity as you'll have to bind the relevant files one by one whereas with named volumes you can mount full configuration and data folders. On the other hand, named volumes are less straightforward to navigate.
# Reference
## Environment Variables
Note for persistent configurations (i.e. bind mount or volumes): when a
variable is set, its value may be written to the appropriate file (usually
config.yaml) each time the container is run.
| Variable | Default | Description |
| ----------------------- | ------------------------- | ----------- |
| `CONFIG_FILE` | `/etc/crowdsec/config.yaml` | Configuration file location |
| `DISABLE_AGENT` | false | Disable the agent, run a LAPI-only container |
| `DISABLE_LOCAL_API` | false | Disable LAPI, run an agent-only container |
| `DISABLE_ONLINE_API` | false | Disable online API registration for signal sharing |
| `TEST_MODE` | false | Don't run the service, only test the configuration: `-e TEST_MODE=true` |
| `TZ` | | Set the [timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) to ensure the logs have a local timestamp. |
| `LOCAL_API_URL` | `http://0.0.0.0:8080` | The LAPI URL, you need to change this when `DISABLE_LOCAL_API` is true: `-e LOCAL_API_URL="http://lapi-address:8080"` |
| `PLUGIN_DIR` | `/usr/local/lib/crowdsec/plugins/` | Directory for plugins: `-e PLUGIN_DIR=""` |
| `METRICS_PORT` | 6060 | Port to expose Prometheus metrics |
| | | |
| __LAPI__ | | (useless with DISABLE_LOCAL_API) |
| `USE_WAL` | false | Enable Write-Ahead Logging with SQLite |
| `CUSTOM_HOSTNAME` | localhost | Name for the local agent (running in the container with LAPI) |
| | | |
| __Agent__ | | (these don't work with DISABLE_AGENT) |
| `TYPE` | | [`Labels.type`](https://docs.crowdsec.net/Crowdsec/v1/references/acquisition/) for file in time-machine: `-e TYPE=""` |
| `DSN` | | Process a single source in time-machine: `-e DSN="file:///var/log/toto.log"` or `-e DSN="cloudwatch:///your/group/path:stream_name?profile=dev&backlog=16h"` or `-e DSN="journalctl://filters=_SYSTEMD_UNIT=ssh.service"` |
| | | |
| __Bouncers__ | | |
| `BOUNCER_KEY_` | | Register a bouncer with the name `` and a key equal to the value of the environment variable. |
| | | |
| __Console__ | | |
| `ENROLL_KEY` | | Enroll key retrieved from [the console](https://app.crowdsec.net/) to enroll the instance. |
| `ENROLL_INSTANCE_NAME` | | To set an instance name and see it on [the console](https://app.crowdsec.net/) |
| `ENROLL_TAGS` | | Tags of the enrolled instance, for search and filter |
| | | |
| __Password Auth__ | | |
| `AGENT_USERNAME` | | Agent username (to register if is LAPI or to use if it's an agent): `-e AGENT_USERNAME="machine_id"` |
| `AGENT_PASSWORD` | | Agent password (to register if is LAPI or to use if it's an agent): `-e AGENT_PASSWORD="machine_password"` |
| | | |
| __TLS Encryption__ | | |
| `USE_TLS` | false | Enable TLS encryption (either as a LAPI or agent) |
| `CACERT_FILE` | | CA certificate bundle (for self-signed certificates) |
| `INSECURE_SKIP_VERIFY` | | Skip LAPI certificate validation |
| `LAPI_CERT_FILE` | | LAPI TLS Certificate path |
| `LAPI_KEY_FILE` | | LAPI TLS Key path |
| | | |
| __TLS Authentication__ | | (these require USE_TLS=true) |
| `CLIENT_CERT_FILE` | | Client TLS Certificate path (enable TLS authentication) |
| `CLIENT_KEY_FILE` | | Client TLS Key path |
| `AGENTS_ALLOWED_OU` | agent-ou | OU values allowed for agents, separated by comma |
| `BOUNCERS_ALLOWED_OU` | bouncer-ou | OU values allowed for bouncers, separated by comma |
| | | |
| __Hub management__ | | |
| `COLLECTIONS` | | Collections to install, separated by space: `-e COLLECTIONS="crowdsecurity/linux crowdsecurity/apache2"` |
| `PARSERS` | | Parsers to install, separated by space |
| `SCENARIOS` | | Scenarios to install, separated by space |
| `POSTOVERFLOWS` | | Postoverflows to install, separated by space |
| `DISABLE_COLLECTIONS` | | Collections to remove, separated by space: `-e DISABLE_COLLECTIONS="crowdsecurity/linux crowdsecurity/nginx"` |
| `DISABLE_PARSERS` | | Parsers to remove, separated by space |
| `DISABLE_SCENARIOS` | | Scenarios to remove, separated by space |
| `DISABLE_POSTOVERFLOWS` | | Postoverflows to remove, separated by space |
| | | |
| __Log verbosity__ | | |
| `LEVEL_INFO` | false | Force INFO level for the container log |
| `LEVEL_DEBUG` | false | Force DEBUG level for the container log |
| `LEVEL_TRACE` | false | Force TRACE level (VERY verbose) for the container log |
| | | |
| __Developer options__ | | |
| `CI_TESTING` | false | Used during functional tests |
| `DEBUG` | false | Trace the entrypoint |
## Volumes
* `/var/lib/crowdsec/data/` - Directory where all crowdsec data (Databases) is located
* `/etc/crowdsec/` - Directory where all crowdsec configurations are located
## File Locations
* `/usr/local/bin/crowdsec` - Crowdsec binary
* `/usr/local/bin/cscli` - Crowdsec CLI binary to interact with crowdsec
# Find Us
* [GitHub](https://github.com/crowdsecurity/crowdsec)
# Contributing
Please read [contributing](https://docs.crowdsec.net/Crowdsec/v1/contributing/) for details on our code of conduct, and the process for submitting pull requests to us.
# License
This project is licensed under the MIT License - see the [LICENSE](https://github.com/crowdsecurity/crowdsec/blob/master/LICENSE) file for details.
crowdsec-1.4.6/docker/config.yaml 0000664 0000000 0000000 00000002537 14371172257 0016753 0 ustar 00root root 0000000 0000000 common:
daemonize: false
pid_dir: /var/run/
log_media: stdout
log_level: info
log_dir: /var/log/
working_dir: .
config_paths:
config_dir: /etc/crowdsec/
data_dir: /var/lib/crowdsec/data/
simulation_path: /etc/crowdsec/simulation.yaml
hub_dir: /etc/crowdsec/hub/
index_path: /etc/crowdsec/hub/.index.json
notification_dir: /etc/crowdsec/notifications/
plugin_dir: /usr/local/lib/crowdsec/plugins/
crowdsec_service:
acquisition_path: /etc/crowdsec/acquis.yaml
parser_routines: 1
plugin_config:
user: nobody
group: nobody
cscli:
output: human
db_config:
log_level: info
type: sqlite
db_path: /var/lib/crowdsec/data/crowdsec.db
flush:
max_items: 5000
max_age: 7d
use_wal: false
api:
client:
insecure_skip_verify: false
credentials_path: /etc/crowdsec/local_api_credentials.yaml
server:
log_level: info
listen_uri: 0.0.0.0:8080
profiles_path: /etc/crowdsec/profiles.yaml
trusted_ips: # IP ranges, or IPs which can have admin API access
- 127.0.0.1
- ::1
online_client: # Central API credentials (to push signals and receive bad IPs)
#credentials_path: /etc/crowdsec/online_api_credentials.yaml
tls:
agents_allowed_ou:
- agent-ou
bouncers_allowed_ou:
- bouncer-ou
prometheus:
enabled: true
level: full
listen_addr: 0.0.0.0
listen_port: 6060
crowdsec-1.4.6/docker/docker_start.sh 0000775 0000000 0000000 00000026137 14371172257 0017647 0 ustar 00root root 0000000 0000000 #!/bin/bash
# shellcheck disable=SC2292 # allow [ test ] syntax
# shellcheck disable=SC2310 # allow "if function..." syntax with -e
set -e
shopt -s inherit_errexit
# match true, TRUE, True, tRuE, etc.
istrue() {
case "$(echo "$1" | tr '[:upper:]' '[:lower:]')" in
true) return 0 ;;
*) return 1 ;;
esac
}
isfalse() {
if istrue "$1"; then
return 1
else
return 0
fi
}
if istrue "$DEBUG"; then
set -x
export PS4='+(${BASH_SOURCE}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'
fi
if istrue "$CI_TESTING"; then
echo "githubciXXXXXXXXXXXXXXXXXXXXXXXX" >/etc/machine-id
fi
#- DEFAULTS -----------------------#
export CONFIG_FILE="${CONFIG_FILE:=/etc/crowdsec/config.yaml}"
export CUSTOM_HOSTNAME="${CUSTOM_HOSTNAME:=localhost}"
#- HELPER FUNCTIONS ----------------#
# csv2yaml
# generate a yaml list from a comma-separated string of values
csv2yaml() {
[ -z "$1" ] && return
echo "$1" | sed 's/,/\n- /g;s/^/- /g'
}
# wrap cscli with the correct config file location
cscli() {
command cscli -c "$CONFIG_FILE" "$@"
}
# conf_get [file_path]
# retrieve a value from a file (by default $CONFIG_FILE)
conf_get() {
if [ $# -ge 2 ]; then
yq e "$1" "$2"
else
yq e "$1" "$CONFIG_FILE"
fi
}
# conf_set [file_path]
# evaluate a yq command (by default on $CONFIG_FILE),
# create the file if it doesn't exist
conf_set() {
if [ $# -ge 2 ]; then
YAML_FILE="$2"
else
YAML_FILE="$CONFIG_FILE"
fi
YAML_CONTENT=$(cat "$YAML_FILE" 2>/dev/null || true)
NEW_CONTENT=$(echo "$YAML_CONTENT" | yq e "$1")
echo "$NEW_CONTENT" | install -m 0600 /dev/stdin "$YAML_FILE"
}
# conf_set_if(): used to update the configuration
# only if a given variable is provided
# conf_set_if "$VAR" [file_path]
conf_set_if() {
if [ "$1" != "" ]; then
shift
conf_set "$@"
fi
}
# register_bouncer
register_bouncer() {
if ! cscli bouncers list -o json | sed '/^ *"name"/!d;s/^ *"name": "\(.*\)",/\1/' | grep -q "^${1}$"; then
if cscli bouncers add "$1" -k "$2" > /dev/null; then
echo "Registered bouncer for $1"
else
echo "Failed to register bouncer for $1"
fi
fi
}
# Call cscli to manage objects ignoring taint errors
# $1 can be collections, parsers, etc.
# $2 can be install, remove, upgrade
# $3 is a list of object names separated by space
cscli_if_clean() {
# loop over all objects
for obj in $3; do
if cscli "$1" inspect "$obj" -o json | yq -e '.tainted // false' >/dev/null 2>&1; then
echo "Object $1/$obj is tainted, skipping"
else
cscli "$1" "$2" "$obj"
fi
done
}
#-----------------------------------#
if [ -n "$CERT_FILE" ] || [ -n "$KEY_FILE" ] ; then
printf '%b' '\033[0;33m'
echo "Warning: the variables CERT_FILE and KEY_FILE have been deprecated." >&2
echo "Please use LAPI_CERT_FILE and LAPI_KEY_FILE insted." >&2
echo "The old variables will be removed in a future release." >&2
printf '%b' '\033[0m'
LAPI_CERT_FILE=${LAPI_CERT_FILE:-$CERT_FILE}
LAPI_KEY_FILE=${LAPI_KEY_FILE:-$KEY_FILE}
fi
# Check and prestage databases
for geodb in GeoLite2-ASN.mmdb GeoLite2-City.mmdb; do
# We keep the pre-populated geoip databases in /staging instead of /var,
# because if the data directory is bind-mounted from the host, it will be
# empty and the files will be out of reach, requiring a runtime download.
# We link to them to save about 80Mb compared to cp/mv.
if [ ! -e "/var/lib/crowdsec/data/$geodb" ] && [ -e "/staging/var/lib/crowdsec/data/$geodb" ]; then
mkdir -p /var/lib/crowdsec/data
ln -s "/staging/var/lib/crowdsec/data/$geodb" /var/lib/crowdsec/data/
fi
done
# Check and prestage /etc/crowdsec
if [ ! -e "/etc/crowdsec/local_api_credentials.yaml" ] && [ ! -e "/etc/crowdsec/config.yaml" ]; then
echo "Populating configuration directory..."
# don't overwrite existing configuration files, which may come
# from bind-mount or even be read-only (configmaps)
if [ -e /staging/etc/crowdsec ]; then
mkdir -p /etc/crowdsec/
# if you change this, check that it still works
# under alpine and k8s, with and without tls
cp -an /staging/etc/crowdsec/* /etc/crowdsec/
fi
fi
# do this as soon as we have a config.yaml, to avoid useless warnings
if istrue "$USE_WAL"; then
conf_set '.db_config.use_wal = true'
elif [ -n "$USE_WAL" ] && isfalse "$USE_WAL"; then
conf_set '.db_config.use_wal = false'
fi
# regenerate local agent credentials (even if agent is disabled, cscli needs a
# connection to the API)
cscli machines delete "$CUSTOM_HOSTNAME" 2>/dev/null || true
if isfalse "$DISABLE_LOCAL_API"; then
if isfalse "$USE_TLS" || [ "$CLIENT_CERT_FILE" = "" ]; then
echo "Regenerate local agent credentials"
cscli machines add "$CUSTOM_HOSTNAME" --auto
fi
echo "Check if lapi needs to register an additional agent"
# pre-registration is not needed with TLS authentication, but we can have TLS transport with user/pw
if [ "$AGENT_USERNAME" != "" ] && [ "$AGENT_PASSWORD" != "" ] ; then
# re-register because pw may have been changed
cscli machines add "$AGENT_USERNAME" --password "$AGENT_PASSWORD" -f /dev/null --force
echo "Agent registered to lapi"
fi
fi
# ----------------
lapi_credentials_path=$(conf_get '.api.client.credentials_path')
conf_set_if "$LOCAL_API_URL" '.url = strenv(LOCAL_API_URL)' "$lapi_credentials_path"
if istrue "$DISABLE_LOCAL_API"; then
# we only use the envvars that are actually defined
# in case of persistent configuration
conf_set_if "$AGENT_USERNAME" '.login = strenv(AGENT_USERNAME)' "$lapi_credentials_path"
conf_set_if "$AGENT_PASSWORD" '.password = strenv(AGENT_PASSWORD)' "$lapi_credentials_path"
fi
conf_set_if "$INSECURE_SKIP_VERIFY" '.api.client.insecure_skip_verify = env(INSECURE_SKIP_VERIFY)'
# agent-only containers still require USE_TLS
if istrue "$USE_TLS"; then
# shellcheck disable=SC2153
conf_set_if "$CACERT_FILE" '.ca_cert_path = strenv(CACERT_FILE)' "$lapi_credentials_path"
conf_set_if "$CLIENT_KEY_FILE" '.key_path = strenv(CLIENT_KEY_FILE)' "$lapi_credentials_path"
conf_set_if "$CLIENT_CERT_FILE" '.cert_path = strenv(CLIENT_CERT_FILE)' "$lapi_credentials_path"
else
conf_set '
del(.ca_cert_path) |
del(.key_path) |
del(.cert_path)
' "$lapi_credentials_path"
fi
if istrue "$DISABLE_ONLINE_API"; then
conf_set 'del(.api.server.online_client)'
fi
# registration to online API for signal push
if isfalse "$DISABLE_ONLINE_API" ; then
CONFIG_DIR=$(conf_get '.config_paths.config_dir')
config_exists=$(conf_get '.api.server.online_client | has("credentials_path")')
if isfalse "$config_exists"; then
export CONFIG_DIR
conf_set '.api.server.online_client = {"credentials_path": strenv(CONFIG_DIR) + "/online_api_credentials.yaml"}'
cscli capi register > "$CONFIG_DIR/online_api_credentials.yaml"
echo "Registration to online API done"
fi
fi
# Enroll instance if enroll key is provided
if isfalse "$DISABLE_ONLINE_API" && [ "$ENROLL_KEY" != "" ]; then
enroll_args=""
if [ "$ENROLL_INSTANCE_NAME" != "" ]; then
enroll_args="--name $ENROLL_INSTANCE_NAME"
fi
if [ "$ENROLL_TAGS" != "" ]; then
# shellcheck disable=SC2086
for tag in ${ENROLL_TAGS}; do
enroll_args="$enroll_args --tags $tag"
done
fi
# shellcheck disable=SC2086
cscli console enroll $enroll_args "$ENROLL_KEY"
fi
# crowdsec sqlite database permissions
if [ "$GID" != "" ]; then
if istrue "$(conf_get '.db_config.type == "sqlite"')"; then
chown ":$GID" "$(conf_get '.db_config.db_path')"
echo "sqlite database permissions updated"
fi
fi
# XXX only with LAPI
if istrue "$USE_TLS"; then
agents_allowed_yaml=$(csv2yaml "$AGENTS_ALLOWED_OU") \
bouncers_allowed_yaml=$(csv2yaml "$BOUNCERS_ALLOWED_OU") \
conf_set_if "$CACERT_FILE" '.api.server.tls.ca_cert_path = strenv(CACERT_FILE)'
conf_set_if "$LAPI_CERT_FILE" '.api.server.tls.cert_file = strenv(LAPI_CERT_FILE)'
conf_set_if "$LAPI_KEY_FILE" '.api.server.tls.key_file = strenv(LAPI_KEY_FILE)'
conf_set_if "$BOUNCERS_ALLOWED_OU" '.api.server.tls.bouncers_allowed_ou = env(bouncers_allowed_yaml)'
conf_set_if "$AGENTS_ALLOWED_OU" '.api.server.tls.agents_allowed_ou = env(agents_allowed_yaml)'
else
conf_set 'del(.api.server.tls)'
fi
conf_set_if "$PLUGIN_DIR" '.config_paths.plugin_dir = strenv(PLUGIN_DIR)'
## Install collections, parsers, scenarios & postoverflows
cscli hub update
cscli_if_clean collections upgrade crowdsecurity/linux
cscli_if_clean parsers upgrade crowdsecurity/whitelists
cscli_if_clean parsers install crowdsecurity/docker-logs
if [ "$COLLECTIONS" != "" ]; then
# shellcheck disable=SC2086
cscli_if_clean collections install "$COLLECTIONS"
fi
if [ "$PARSERS" != "" ]; then
# shellcheck disable=SC2086
cscli_if_clean parsers install "$PARSERS"
fi
if [ "$SCENARIOS" != "" ]; then
# shellcheck disable=SC2086
cscli_if_clean scenarios install "$SCENARIOS"
fi
if [ "$POSTOVERFLOWS" != "" ]; then
# shellcheck disable=SC2086
cscli_if_clean postoverflows install "$POSTOVERFLOWS"
fi
## Remove collections, parsers, scenarios & postoverflows
if [ "$DISABLE_COLLECTIONS" != "" ]; then
# shellcheck disable=SC2086
cscli_if_clean collections remove "$DISABLE_COLLECTIONS"
fi
if [ "$DISABLE_PARSERS" != "" ]; then
# shellcheck disable=SC2086
cscli_if_clean parsers remove "$DISABLE_PARSERS"
fi
if [ "$DISABLE_SCENARIOS" != "" ]; then
# shellcheck disable=SC2086
cscli_if_clean scenarios remove "$DISABLE_SCENARIOS"
fi
if [ "$DISABLE_POSTOVERFLOWS" != "" ]; then
# shellcheck disable=SC2086
cscli_if_clean postoverflows remove "$DISABLE_POSTOVERFLOWS"
fi
## Register bouncers via env
for BOUNCER in $(compgen -A variable | grep -i BOUNCER_KEY); do
KEY=$(printf '%s' "${!BOUNCER}")
NAME=$(printf '%s' "$BOUNCER" | cut -d_ -f3-)
if [[ -n $KEY ]] && [[ -n $NAME ]]; then
register_bouncer "$NAME" "$KEY"
fi
done
## Register bouncers via secrets
shopt -s nullglob extglob
for BOUNCER in /run/secrets/@(bouncer_key|BOUNCER_KEY)* ; do
KEY=$(cat "${BOUNCER}")
NAME=$(echo "${BOUNCER}" | awk -F "/" '{printf $NF}' | cut -d_ -f2-)
if [[ -n $KEY ]] && [[ -n $NAME ]]; then
register_bouncer "$NAME" "$KEY"
fi
done
shopt -u nullglob extglob
ARGS=""
if [ "$CONFIG_FILE" != "" ]; then
ARGS="-c $CONFIG_FILE"
fi
if [ "$DSN" != "" ]; then
ARGS="$ARGS -dsn ${DSN}"
fi
if [ "$TYPE" != "" ]; then
ARGS="$ARGS -type $TYPE"
fi
if istrue "$TEST_MODE"; then
ARGS="$ARGS -t"
fi
if istrue "$DISABLE_AGENT"; then
ARGS="$ARGS -no-cs"
fi
if istrue "$DISABLE_LOCAL_API"; then
ARGS="$ARGS -no-api"
fi
if istrue "$LEVEL_TRACE"; then
ARGS="$ARGS -trace"
fi
if istrue "$LEVEL_DEBUG"; then
ARGS="$ARGS -debug"
fi
if istrue "$LEVEL_INFO"; then
ARGS="$ARGS -info"
fi
conf_set_if "$METRICS_PORT" '.prometheus.listen_port=env(METRICS_PORT)'
# shellcheck disable=SC2086
exec crowdsec $ARGS
crowdsec-1.4.6/go.mod 0000664 0000000 0000000 00000017677 14371172257 0014474 0 ustar 00root root 0000000 0000000 module github.com/crowdsecurity/crowdsec
go 1.19
require (
entgo.io/ent v0.11.3
github.com/AlecAivazis/survey/v2 v2.2.7
github.com/Microsoft/go-winio v0.5.2 // indirect
github.com/alexliesenfeld/health v0.5.1
github.com/antonmedv/expr v1.9.0
github.com/appleboy/gin-jwt/v2 v2.8.0
github.com/aws/aws-sdk-go v1.42.25
github.com/buger/jsonparser v1.1.1
github.com/c-robinson/iplib v1.0.3
github.com/confluentinc/bincover v0.2.0
github.com/coreos/go-systemd v0.0.0-20191104093116-d3cd4ed1dbcf
github.com/crowdsecurity/dlog v0.0.0-20170105205344-4fb5f8204f26
github.com/crowdsecurity/grokky v0.1.0
github.com/crowdsecurity/machineid v1.0.2
github.com/davecgh/go-spew v1.1.1
github.com/dghubble/sling v1.3.0
github.com/docker/docker v20.10.2+incompatible
github.com/docker/go-connections v0.4.0
github.com/enescakir/emoji v1.0.0
github.com/fatih/color v1.13.0
github.com/fsnotify/fsnotify v1.5.1
github.com/gin-gonic/gin v1.7.7
github.com/go-co-op/gocron v1.17.0
github.com/go-openapi/errors v0.20.1
github.com/go-openapi/strfmt v0.19.11
github.com/go-openapi/swag v0.19.12
github.com/go-openapi/validate v0.20.0
github.com/go-sql-driver/mysql v1.6.0
github.com/google/go-querystring v1.0.0
github.com/google/uuid v1.3.0
github.com/goombaio/namegenerator v0.0.0-20181006234301-989e774b106e
github.com/hashicorp/go-hclog v1.0.0
github.com/hashicorp/go-plugin v1.4.2
github.com/hashicorp/go-version v1.2.1
github.com/jackc/pgx/v4 v4.14.1
github.com/jarcoal/httpmock v1.1.0
github.com/jszwec/csvutil v1.5.1
github.com/lib/pq v1.10.7
github.com/mattn/go-sqlite3 v1.14.15
github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826
github.com/nxadm/tail v1.4.6
github.com/oschwald/geoip2-golang v1.4.0
github.com/oschwald/maxminddb-golang v1.8.0
github.com/pkg/errors v0.9.1
github.com/prometheus/client_golang v1.11.0
github.com/prometheus/client_model v0.2.0
github.com/prometheus/prom2json v1.3.0
github.com/r3labs/diff/v2 v2.14.1
github.com/sirupsen/logrus v1.8.1
github.com/spf13/cobra v1.5.0
github.com/stretchr/testify v1.8.0
golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4
google.golang.org/grpc v1.45.0
google.golang.org/protobuf v1.28.0
gopkg.in/natefinch/lumberjack.v2 v2.0.0
gopkg.in/tomb.v2 v2.0.0-20161208151619-d5d1b5820637
gopkg.in/yaml.v2 v2.4.0
gotest.tools/v3 v3.0.3
)
require (
github.com/Masterminds/sprig/v3 v3.2.2
github.com/aquasecurity/table v1.8.0
github.com/beevik/etree v1.1.0
github.com/blackfireio/osinfo v1.0.3
github.com/google/winops v0.0.0-20211216095627-f0e86eb1453b
github.com/ivanpirog/coloredcobra v1.0.1
github.com/mattn/go-isatty v0.0.14
github.com/pbnjay/memory v0.0.0-20210728143218-7b4eea64cf58
github.com/segmentio/kafka-go v0.4.34
github.com/texttheater/golang-levenshtein/levenshtein v0.0.0-20200805054039-cae8b0eaed6c
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f
)
require (
ariga.io/atlas v0.7.2-0.20220927111110-867ee0cca56a // indirect
github.com/Masterminds/goutils v1.1.1 // indirect
github.com/Masterminds/semver/v3 v3.1.1 // indirect
github.com/PuerkitoBio/purell v1.1.1 // indirect
github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 // indirect
github.com/agext/levenshtein v1.2.3 // indirect
github.com/ahmetalpbalkan/dlog v0.0.0-20170105205344-4fb5f8204f26 // indirect
github.com/apparentlymart/go-textseg/v13 v13.0.0 // indirect
github.com/asaskevich/govalidator v0.0.0-20200907205600-7a23bdc65eef // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.1.2 // indirect
github.com/containerd/containerd v1.6.2 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.2 // indirect
github.com/creack/pty v1.1.11 // indirect
github.com/docker/distribution v2.7.1+incompatible // indirect
github.com/docker/go-units v0.4.0 // indirect
github.com/gin-contrib/sse v0.1.0 // indirect
github.com/go-openapi/analysis v0.19.16 // indirect
github.com/go-openapi/inflect v0.19.0 // indirect
github.com/go-openapi/jsonpointer v0.19.5 // indirect
github.com/go-openapi/jsonreference v0.19.5 // indirect
github.com/go-openapi/loads v0.20.0 // indirect
github.com/go-openapi/runtime v0.19.24 // indirect
github.com/go-openapi/spec v0.20.0 // indirect
github.com/go-playground/locales v0.14.0 // indirect
github.com/go-playground/universal-translator v0.18.0 // indirect
github.com/go-playground/validator/v10 v10.10.0 // indirect
github.com/go-stack/stack v1.8.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang-jwt/jwt/v4 v4.2.0 // indirect
github.com/golang/glog v0.0.0-20210429001901-424d2337a529 // indirect
github.com/golang/protobuf v1.5.2 // indirect
github.com/google/go-cmp v0.5.7 // indirect
github.com/gorilla/mux v1.7.3 // indirect
github.com/hashicorp/hcl/v2 v2.13.0 // indirect
github.com/hashicorp/yamux v0.0.0-20180604194846-3520598351bb // indirect
github.com/huandu/xstrings v1.3.2 // indirect
github.com/imdario/mergo v0.3.12 // indirect
github.com/inconshreveable/mousetrap v1.0.1 // indirect
github.com/jackc/chunkreader/v2 v2.0.1 // indirect
github.com/jackc/pgconn v1.10.1 // indirect
github.com/jackc/pgio v1.0.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgproto3/v2 v2.2.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20200714003250-2b9c44734f2b // indirect
github.com/jackc/pgtype v1.9.1 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect
github.com/klauspost/compress v1.15.7 // indirect
github.com/leodido/go-urn v1.2.1 // indirect
github.com/mailru/easyjson v0.7.6 // indirect
github.com/mattn/go-colorable v0.1.12 // indirect
github.com/mattn/go-runewidth v0.0.13 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369 // indirect
github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d // indirect
github.com/mitchellh/copystructure v1.2.0 // indirect
github.com/mitchellh/go-testing-interface v1.0.0 // indirect
github.com/mitchellh/go-wordwrap v1.0.1 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/mitchellh/reflectwalk v1.0.2 // indirect
github.com/moby/term v0.0.0-20201216013528-df9cb8a40635 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/morikuni/aec v1.0.0 // indirect
github.com/oklog/run v1.0.0 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.0.2-0.20211117181255-693428a734f5 // indirect
github.com/pierrec/lz4/v4 v4.1.15 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/prometheus/common v0.30.0 // indirect
github.com/prometheus/procfs v0.7.3 // indirect
github.com/rivo/uniseg v0.2.0 // indirect
github.com/robfig/cron/v3 v3.0.1 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/shopspring/decimal v1.2.0 // indirect
github.com/spf13/cast v1.3.1 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/tidwall/gjson v1.13.0 // indirect
github.com/ugorji/go/codec v1.2.6 // indirect
github.com/vjeantet/grok v1.0.1 // indirect
github.com/vmihailenco/msgpack v4.0.4+incompatible // indirect
github.com/zclconf/go-cty v1.10.0 // indirect
go.mongodb.org/mongo-driver v1.9.0 // indirect
golang.org/x/net v0.0.0-20220706163947-c90051bbdb60 // indirect
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4 // indirect
golang.org/x/term v0.0.0-20220526004731-065cf7ba2467 // indirect
golang.org/x/text v0.3.7 // indirect
google.golang.org/appengine v1.6.7 // indirect
google.golang.org/genproto v0.0.0-20220414192740-2d67ff6cf2b4 // indirect
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
replace golang.org/x/time/rate => github.com/crowdsecurity/crowdsec/pkg/time/rate v0.0.0
crowdsec-1.4.6/go.sum 0000664 0000000 0000000 00000340716 14371172257 0014512 0 ustar 00root root 0000000 0000000 ariga.io/atlas v0.7.2-0.20220927111110-867ee0cca56a h1:6/nt4DODfgxzHTTg3tYy7YkVzruGQGZ/kRvXpA45KUo=
ariga.io/atlas v0.7.2-0.20220927111110-867ee0cca56a/go.mod h1:ft47uSh5hWGDCmQC9DsztZg6Xk+KagM5Ts/mZYKb9JE=
bitbucket.org/creachadair/stringset v0.0.9 h1:L4vld9nzPt90UZNrXjNelTshD74ps4P5NGs3Iq6yN3o=
bitbucket.org/creachadair/stringset v0.0.9/go.mod h1:t+4WcQ4+PXTa8aQdNKe40ZP6iwesoMFWAxPGd3UGjyY=
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU=
cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU=
cloud.google.com/go v0.44.2/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY=
cloud.google.com/go v0.45.1/go.mod h1:RpBamKRgapWJb87xiFSdk4g1CME7QZg3uwTez+TSTjc=
cloud.google.com/go v0.46.3/go.mod h1:a6bKKbmY7er1mI7TEI4lsAkts/mkhTSZK8w33B4RAg0=
cloud.google.com/go v0.50.0/go.mod h1:r9sluTvynVuxRIOHXQEHMFffphuXHOMZMycpNR5e6To=
cloud.google.com/go v0.52.0/go.mod h1:pXajvRH/6o3+F9jDHZWQ5PbGhn+o8w9qiu/CffaVdO4=
cloud.google.com/go v0.53.0/go.mod h1:fp/UouUEsRkN6ryDKNW/Upv/JBKnv6WDthjR6+vze6M=
cloud.google.com/go v0.54.0/go.mod h1:1rq2OEkV3YMf6n/9ZvGWI3GWw0VoqH/1x2nd8Is/bPc=
cloud.google.com/go v0.56.0/go.mod h1:jr7tqZxxKOVYizybht9+26Z/gUq7tiRzu+ACVAMbKVk=
cloud.google.com/go v0.57.0/go.mod h1:oXiQ6Rzq3RAkkY7N6t3TcE6jE+CIBBbA36lwQ1JyzZs=
cloud.google.com/go v0.62.0/go.mod h1:jmCYTdRCQuc1PHIIJ/maLInMho30T/Y0M4hTdTShOYc=
cloud.google.com/go v0.65.0/go.mod h1:O5N8zS7uWy9vkA9vayVHs65eM1ubvY4h553ofrNHObY=
cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o=
cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE=
cloud.google.com/go/bigquery v1.4.0/go.mod h1:S8dzgnTigyfTmLBfrtrhyYhwRxG72rYxvftPBK2Dvzc=
cloud.google.com/go/bigquery v1.5.0/go.mod h1:snEHRnqQbz117VIFhE8bmtwIDY80NLUZUMb4Nv6dBIg=
cloud.google.com/go/bigquery v1.7.0/go.mod h1://okPTzCYNXSlb24MZs83e2Do+h+VXtc4gLoIoXIAPc=
cloud.google.com/go/bigquery v1.8.0/go.mod h1:J5hqkt3O0uAFnINi6JXValWIb1v0goeZM77hZzJN/fQ=
cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE=
cloud.google.com/go/datastore v1.1.0/go.mod h1:umbIZjpQpHh4hmRpGhH4tLFup+FVzqBi1b3c64qFpCk=
cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I=
cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw=
cloud.google.com/go/pubsub v1.2.0/go.mod h1:jhfEVHT8odbXTkndysNHCcx0awwzvfOlguIAii9o8iA=
cloud.google.com/go/pubsub v1.3.1/go.mod h1:i+ucay31+CNRpDW4Lu78I4xXG+O1r/MAHgjpRVR+TSU=
cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw=
cloud.google.com/go/storage v1.5.0/go.mod h1:tpKbwo567HUNpVclU5sGELwQWBDZ8gh0ZeosJ0Rtdos=
cloud.google.com/go/storage v1.6.0/go.mod h1:N7U0C8pVQ/+NIKOBQyamJIeKQKkZ+mxpohlUTyfDhBk=
cloud.google.com/go/storage v1.8.0/go.mod h1:Wv1Oy7z6Yz3DshWRJFhqM/UCfaWIRTdp0RXyy7KQOVs=
cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9ullr3+Kg0=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
entgo.io/ent v0.11.3 h1:F5FBGAWiDCGder7YT+lqMnyzXl6d0xU3xMBM/SO3CMc=
entgo.io/ent v0.11.3/go.mod h1:mvDhvynOzAsOe7anH7ynPPtMjA/eeXP96kAfweevyxc=
github.com/AlecAivazis/survey/v2 v2.2.7 h1:5NbxkF4RSKmpywYdcRgUmos1o+roJY8duCLZXbVjoig=
github.com/AlecAivazis/survey/v2 v2.2.7/go.mod h1:9DYvHgXtiXm6nCn+jXnOXLKbH+Yo9u8fAS/SduGdoPk=
github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78 h1:w+iIsaOQNcT7OZ575w+acHgRric5iCyQh+xv+KJ4HB8=
github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78/go.mod h1:LmzpDX56iTiv29bbRTIsUNlaFfuhWRQBWjQdVyAevI8=
github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
github.com/DATA-DOG/go-sqlmock v1.3.3/go.mod h1:f/Ixk793poVmq4qj/V1dPUg2JEAKC73Q5eFN3EC/SaM=
github.com/DATA-DOG/go-sqlmock v1.5.0 h1:Shsta01QNfFxHCfpW6YH2STWB0MudeXXEWMr20OEh60=
github.com/Masterminds/goutils v1.1.1 h1:5nUrii3FMTL5diU80unEVvNevw1nH4+ZV4DSLVJLSYI=
github.com/Masterminds/goutils v1.1.1/go.mod h1:8cTjp+g8YejhMuvIA5y2vz3BpJxksy863GQaJW2MFNU=
github.com/Masterminds/semver/v3 v3.1.1 h1:hLg3sBzpNErnxhQtUy/mmLR2I9foDujNK030IGemrRc=
github.com/Masterminds/semver/v3 v3.1.1/go.mod h1:VPu/7SZ7ePZ3QOrcuXROw5FAcLl4a0cBrbBpGY/8hQs=
github.com/Masterminds/sprig/v3 v3.2.2 h1:17jRggJu518dr3QaafizSXOjKYp94wKfABxUmyxvxX8=
github.com/Masterminds/sprig/v3 v3.2.2/go.mod h1:UoaO7Yp8KlPnJIYWTFkMaqPUYKTfGFPhxNuwnnxkKlk=
github.com/Microsoft/go-winio v0.5.2 h1:a9IhgEQBCUEk6QCdml9CiJGhAws+YwffDHEMp1VMrpA=
github.com/Microsoft/go-winio v0.5.2/go.mod h1:WpS1mjBmmwHBEWmogvA2mj8546UReBk4v8QkMxJ6pZY=
github.com/Netflix/go-expect v0.0.0-20180615182759-c93bf25de8e8 h1:xzYJEypr/85nBpB11F9br+3HUrpgb+fcm5iADzXXYEw=
github.com/Netflix/go-expect v0.0.0-20180615182759-c93bf25de8e8/go.mod h1:oX5x61PbNXchhh0oikYAH+4Pcfw5LKv21+Jnpr6r6Pc=
github.com/PuerkitoBio/purell v1.1.0/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0=
github.com/PuerkitoBio/purell v1.1.1 h1:WEQqlqaGbrPkxLJWfBwQmfEAE1Z7ONdDLqrN38tNFfI=
github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0=
github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 h1:d+Bc7a5rLufV/sSk/8dngufqelfh6jnri85riMAaF/M=
github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE=
github.com/agext/levenshtein v1.2.3 h1:YB2fHEn0UJagG8T1rrWknE3ZQzWM06O8AMAatNn7lmo=
github.com/agext/levenshtein v1.2.3/go.mod h1:JEDfjyjHDjOF/1e4FlBE/PkbqA9OfWu2ki2W0IB5558=
github.com/agnivade/levenshtein v1.0.1/go.mod h1:CURSv5d9Uaml+FovSIICkLbAUZ9S4RqaHDIsdSBg7lM=
github.com/ahmetalpbalkan/dlog v0.0.0-20170105205344-4fb5f8204f26 h1:pzStYMLAXM7CNQjS/Wn+zK9MUxDhSUNfVvnHsyQyjs0=
github.com/ahmetalpbalkan/dlog v0.0.0-20170105205344-4fb5f8204f26/go.mod h1:ilK+u7u1HoqaDk0mjhh27QJB7PyWMreGffEvOCoEKiY=
github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d/go.mod h1:rBZYJk541a8SKzHPHnH3zbiI+7dagKZ0cgpgrD7Fyho=
github.com/alexliesenfeld/health v0.5.1 h1:cohQdtQbJdA6bj0aMD4gdXA9xQyvh9NxWO9XLGYTYcY=
github.com/alexliesenfeld/health v0.5.1/go.mod h1:N4NDIeQtlWumG+6z1ne1v62eQxktz5ylEgGgH9emdMw=
github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883/go.mod h1:rCTlJbsFo29Kk6CurOXKm700vrz8f0KW0JNfpkRJY/8=
github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY=
github.com/antonmedv/expr v1.9.0 h1:j4HI3NHEdgDnN9p6oI6Ndr0G5QryMY0FNxT4ONrFDGU=
github.com/antonmedv/expr v1.9.0/go.mod h1:5qsM3oLGDND7sDmQGDXHkYfkjYMUX14qsgqmHhwGEk8=
github.com/apparentlymart/go-textseg/v13 v13.0.0 h1:Y+KvPE1NYz0xl601PVImeQfFyEy6iT90AvPUL1NNfNw=
github.com/apparentlymart/go-textseg/v13 v13.0.0/go.mod h1:ZK2fH7c4NqDTLtiYLvIkEghdlcqw7yxLeM89kiTRPUo=
github.com/appleboy/gin-jwt/v2 v2.8.0 h1:Glo7cb9eBR+hj8Y7WzgfkOlqCaNLjP+RV4dNO3fpdps=
github.com/appleboy/gin-jwt/v2 v2.8.0/go.mod h1:KsK7E8HTvRg3vOiumTsr/ntNTHbZ3IbHLe4Eto31p7k=
github.com/appleboy/gofight/v2 v2.1.2 h1:VOy3jow4vIK8BRQJoC/I9muxyYlJ2yb9ht2hZoS3rf4=
github.com/appleboy/gofight/v2 v2.1.2/go.mod h1:frW+U1QZEdDgixycTj4CygQ48yLTUhplt43+Wczp3rw=
github.com/aquasecurity/table v1.8.0 h1:9ntpSwrUfjrM6/YviArlx/ZBGd6ix8W+MtojQcM7tv0=
github.com/aquasecurity/table v1.8.0/go.mod h1:eqOmvjjB7AhXFgFqpJUEE/ietg7RrMSJZXyTN8E/wZw=
github.com/asaskevich/govalidator v0.0.0-20180720115003-f9ffefc3facf/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
github.com/asaskevich/govalidator v0.0.0-20200108200545-475eaeb16496/go.mod h1:oGkLhpf+kjZl6xBf758TQhh5XrAeiJv/7FRz/2spLIg=
github.com/asaskevich/govalidator v0.0.0-20200428143746-21a406dcc535/go.mod h1:oGkLhpf+kjZl6xBf758TQhh5XrAeiJv/7FRz/2spLIg=
github.com/asaskevich/govalidator v0.0.0-20200907205600-7a23bdc65eef h1:46PFijGLmAjMPwCCCo7Jf0W6f9slllCkkv7vyc1yOSg=
github.com/asaskevich/govalidator v0.0.0-20200907205600-7a23bdc65eef/go.mod h1:WaHUgvxTVq04UNunO+XhnAqY/wQc+bxr74GqbsZ/Jqw=
github.com/aws/aws-sdk-go v1.34.28/go.mod h1:H7NKnBqNVzoTJpGfLrQkkD+ytBA93eiDYi/+8rV9s48=
github.com/aws/aws-sdk-go v1.42.25 h1:BbdvHAi+t9LRiaYUyd53noq9jcaAcfzOhSVbKfr6Avs=
github.com/aws/aws-sdk-go v1.42.25/go.mod h1:gyRszuZ/icHmHAVE4gc/r+cfCmhA1AD+vqfWbgI+eHs=
github.com/beevik/etree v1.1.0 h1:T0xke/WvNtMoCqgzPhkX2r4rjY3GDZFi+FjpRZY2Jbs=
github.com/beevik/etree v1.1.0/go.mod h1:r8Aw8JqVegEf0w2fDnATrX9VpkMcyFeM0FhwO62wh+A=
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/blackfireio/osinfo v1.0.3 h1:Yk2t2GTPjBcESv6nDSWZKO87bGMQgO+Hi9OoXPpxX8c=
github.com/blackfireio/osinfo v1.0.3/go.mod h1:Pd987poVNmd5Wsx6PRPw4+w7kLlf9iJxoRKPtPAjOrA=
github.com/buger/jsonparser v1.1.1 h1:2PnMjfWD7wBILjqQbt530v576A/cAbQvEW9gGIpYMUs=
github.com/buger/jsonparser v1.1.1/go.mod h1:6RYKKt7H4d4+iWqouImQ9R2FZql3VbhNgx27UK13J/0=
github.com/c-robinson/iplib v1.0.3 h1:NG0UF0GoEsrC1/vyfX1Lx2Ss7CySWl3KqqXh3q4DdPU=
github.com/c-robinson/iplib v1.0.3/go.mod h1:i3LuuFL1hRT5gFpBRnEydzw8R6yhGkF4szNDIbF8pgo=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/cespare/xxhash/v2 v2.1.2 h1:YRXhKfTDauu4ajMg1TPgFO5jnlC2HCbmLXMcTG5cbYE=
github.com/cespare/xxhash/v2 v2.1.2/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
github.com/cncf/udpa/go v0.0.0-20201120205902-5459f2c99403/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
github.com/cncf/udpa/go v0.0.0-20210930031921-04548b0d99d4/go.mod h1:6pvJx4me5XPnfI9Z40ddWsdw2W/uZgQLFXToKeRcDiI=
github.com/cncf/xds/go v0.0.0-20210805033703-aa0b78936158/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
github.com/cncf/xds/go v0.0.0-20210922020428-25de7278fc84/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
github.com/cncf/xds/go v0.0.0-20211011173535-cb28da3451f1/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
github.com/cockroachdb/apd v1.1.0 h1:3LFP3629v+1aKXU5Q37mxmRxX/pIu1nijXydLShEq5I=
github.com/cockroachdb/apd v1.1.0/go.mod h1:8Sl8LxpKi29FqWXR16WEFZRNSz3SoPzUzeMeY4+DwBQ=
github.com/confluentinc/bincover v0.2.0 h1:WSS3MqzwJbosCLMOuF3tJ0pMpALzBfrm80Tb+/3gbQs=
github.com/confluentinc/bincover v0.2.0/go.mod h1:qeI1wx0RxdGTZtrJY0HVlgJ4NqC/X2Z+fHbvy87tgHE=
github.com/containerd/containerd v1.6.2 h1:pcaPUGbYW8kBw6OgIZwIVIeEhdWVrBzsoCfVJ5BjrLU=
github.com/containerd/containerd v1.6.2/go.mod h1:sidY30/InSE1j2vdD1ihtKoJz+lWdaXMdiAeIupaf+s=
github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
github.com/coreos/go-systemd v0.0.0-20190719114852-fd7a80b32e1f/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
github.com/coreos/go-systemd v0.0.0-20191104093116-d3cd4ed1dbcf h1:iW4rZ826su+pqaw19uhpSCzhj44qo35pNgKFGqzDKkU=
github.com/coreos/go-systemd v0.0.0-20191104093116-d3cd4ed1dbcf/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
github.com/cpuguy83/go-md2man/v2 v2.0.1/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
github.com/cpuguy83/go-md2man/v2 v2.0.2 h1:p1EgwI/C7NhT0JmVkwCD2ZBK8j4aeHQX2pMHHBfMQ6w=
github.com/cpuguy83/go-md2man/v2 v2.0.2/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
github.com/creachadair/staticfile v0.1.3/go.mod h1:a3qySzCIXEprDGxk6tSxSI+dBBdLzqeBOMhZ+o2d3pM=
github.com/creack/pty v1.1.7/go.mod h1:lj5s0c3V2DBrqTV7llrYr5NG6My20zk30Fl46Y7DoTY=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/creack/pty v1.1.11 h1:07n33Z8lZxZ2qwegKbObQohDhXDQxiMMz1NOUGYlesw=
github.com/creack/pty v1.1.11/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/crowdsecurity/dlog v0.0.0-20170105205344-4fb5f8204f26 h1:r97WNVC30Uen+7WnLs4xDScS/Ex988+id2k6mDf8psU=
github.com/crowdsecurity/dlog v0.0.0-20170105205344-4fb5f8204f26/go.mod h1:zpv7r+7KXwgVUZnUNjyP22zc/D7LKjyoY02weH2RBbk=
github.com/crowdsecurity/grokky v0.1.0 h1:jLUzZd3vKxYrM4hQ8n5HWLfvs5ag4UP08eT9OTekI4U=
github.com/crowdsecurity/grokky v0.1.0/go.mod h1:fx5UYUYAFIrOUNAkFCUOM2wJcsp9EWSQE9R0/9kaFJg=
github.com/crowdsecurity/machineid v1.0.2 h1:wpkpsUghJF8Khtmn/tg6GxgdhLA1Xflerh5lirI+bdc=
github.com/crowdsecurity/machineid v1.0.2/go.mod h1:XWUSlnS0R0+u/JK5ulidwlbceNT3ZOCKteoVQEn6Luo=
github.com/davecgh/go-spew v0.0.0-20161028175848-04cdfd42973b/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dghubble/sling v1.3.0 h1:pZHjCJq4zJvc6qVQ5wN1jo5oNZlNE0+8T/h0XeXBUKU=
github.com/dghubble/sling v1.3.0/go.mod h1:XXShWaBWKzNLhu2OxikSNFrlsvowtz4kyRuXUG7oQKY=
github.com/docker/distribution v2.7.1+incompatible h1:a5mlkVzth6W5A4fOsS3D2EO5BUmsJpcB+cRlLU7cSug=
github.com/docker/distribution v2.7.1+incompatible/go.mod h1:J2gT2udsDAN96Uj4KfcMRqY0/ypR+oyYUYmja8H+y+w=
github.com/docker/docker v20.10.2+incompatible h1:vFgEHPqWBTp4pTjdLwjAA4bSo3gvIGOYwuJTlEjVBCw=
github.com/docker/docker v20.10.2+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
github.com/docker/go-connections v0.4.0 h1:El9xVISelRB7BuFusrZozjnkIM5YnzCViNKohAFqRJQ=
github.com/docker/go-connections v0.4.0/go.mod h1:Gbd7IOopHjR8Iph03tsViu4nIes5XhDvyHbTtUxmeec=
github.com/docker/go-units v0.3.3/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/docker/go-units v0.4.0 h1:3uh0PgVws3nIA0Q+MwDC8yjEPf9zjRfZZWXZYDct3Tw=
github.com/docker/go-units v0.4.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/enescakir/emoji v1.0.0 h1:W+HsNql8swfCQFtioDGDHCHri8nudlK1n5p2rHCJoog=
github.com/enescakir/emoji v1.0.0/go.mod h1:Bt1EKuLnKDTYpLALApstIkAjdDrS/8IAgTkKp+WKFD0=
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk=
github.com/envoyproxy/go-control-plane v0.9.10-0.20210907150352-cf90f659a021/go.mod h1:AFq3mo9L8Lqqiid3OhADV3RfLJnjiw63cSpi+fDTRC0=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
github.com/fatih/color v1.13.0 h1:8LOYc1KYPPmyKMuN8QV2DNRWNbLo6LZ0iLs8+mlH53w=
github.com/fatih/color v1.13.0/go.mod h1:kLAiJbzzSOZDVNGyDpeOxJ47H46qBXwg5ILebYFFOfk=
github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
github.com/fsnotify/fsnotify v1.5.1 h1:mZcQUHVQUQWoPXXtuf9yuEXKudkV2sx1E06UadKWpgI=
github.com/fsnotify/fsnotify v1.5.1/go.mod h1:T3375wBYaZdLLcVNkcVbzGHY7f1l/uK5T5Ai1i3InKU=
github.com/gdamore/encoding v1.0.0/go.mod h1:alR0ol34c49FCSBLjhosxzcPHQbf2trDkoo5dl+VrEg=
github.com/gdamore/tcell v1.3.0/go.mod h1:Hjvr+Ofd+gLglo7RYKxxnzCBmev3BzsS67MebKS4zMM=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/gin-contrib/sse v0.1.0 h1:Y/yl/+YNO8GZSjAhjMsSuLt29uWRFHdHYUb5lYOV9qE=
github.com/gin-contrib/sse v0.1.0/go.mod h1:RHrZQHXnP2xjPF+u1gW/2HnVO7nvIa9PG3Gm+fLHvGI=
github.com/gin-gonic/gin v1.7.7 h1:3DoBmSbJbZAWqXJC3SLjAPfutPJJRN1U5pALB7EeTTs=
github.com/gin-gonic/gin v1.7.7/go.mod h1:axIBovoeJpVj8S3BwE0uPMTeReE4+AfFtqpqaZ1qq1U=
github.com/globalsign/mgo v0.0.0-20180905125535-1ca0a4f7cbcb/go.mod h1:xkRDCp4j0OGD1HRkm4kmhM+pmpv3AKq5SU7GMg4oO/Q=
github.com/globalsign/mgo v0.0.0-20181015135952-eeefdecb41b8/go.mod h1:xkRDCp4j0OGD1HRkm4kmhM+pmpv3AKq5SU7GMg4oO/Q=
github.com/go-co-op/gocron v1.17.0 h1:IixLXsti+Qo0wMvmn6Kmjp2csk2ykpkcL+EmHmST18w=
github.com/go-co-op/gocron v1.17.0/go.mod h1:IpDBSaJOVfFw7hXZuTag3SCSkqazXBBUkbQ1m1aesBs=
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-kit/log v0.1.0/go.mod h1:zbhenjAZHb184qTLMA9ZjW7ThYL0H2mk7Q6pNt4vbaY=
github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk=
github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG1KdI/P7A=
github.com/go-openapi/analysis v0.0.0-20180825180245-b006789cd277/go.mod h1:k70tL6pCuVxPJOHXQ+wIac1FUrvNkHolPie/cLEU6hI=
github.com/go-openapi/analysis v0.17.0/go.mod h1:IowGgpVeD0vNm45So8nr+IcQ3pxVtpRoBWb8PVZO0ik=
github.com/go-openapi/analysis v0.18.0/go.mod h1:IowGgpVeD0vNm45So8nr+IcQ3pxVtpRoBWb8PVZO0ik=
github.com/go-openapi/analysis v0.19.2/go.mod h1:3P1osvZa9jKjb8ed2TPng3f0i/UY9snX6gxi44djMjk=
github.com/go-openapi/analysis v0.19.4/go.mod h1:3P1osvZa9jKjb8ed2TPng3f0i/UY9snX6gxi44djMjk=
github.com/go-openapi/analysis v0.19.5/go.mod h1:hkEAkxagaIvIP7VTn8ygJNkd4kAYON2rCu0v0ObL0AU=
github.com/go-openapi/analysis v0.19.10/go.mod h1:qmhS3VNFxBlquFJ0RGoDtylO9y4pgTAUNE9AEEMdlJQ=
github.com/go-openapi/analysis v0.19.16 h1:Ub9e++M8sDwtHD+S587TYi+6ANBG1NRYGZDihqk0SaY=
github.com/go-openapi/analysis v0.19.16/go.mod h1:GLInF007N83Ad3m8a/CbQ5TPzdnGT7workfHwuVjNVk=
github.com/go-openapi/errors v0.17.0/go.mod h1:LcZQpmvG4wyF5j4IhA73wkLFQg+QJXOQHVjmcZxhka0=
github.com/go-openapi/errors v0.18.0/go.mod h1:LcZQpmvG4wyF5j4IhA73wkLFQg+QJXOQHVjmcZxhka0=
github.com/go-openapi/errors v0.19.2/go.mod h1:qX0BLWsyaKfvhluLejVpVNwNRdXZhEbTA4kxxpKBC94=
github.com/go-openapi/errors v0.19.3/go.mod h1:qX0BLWsyaKfvhluLejVpVNwNRdXZhEbTA4kxxpKBC94=
github.com/go-openapi/errors v0.19.6/go.mod h1:cM//ZKUKyO06HSwqAelJ5NsEMMcpa6VpXe8DOa1Mi1M=
github.com/go-openapi/errors v0.19.7/go.mod h1:cM//ZKUKyO06HSwqAelJ5NsEMMcpa6VpXe8DOa1Mi1M=
github.com/go-openapi/errors v0.19.8/go.mod h1:cM//ZKUKyO06HSwqAelJ5NsEMMcpa6VpXe8DOa1Mi1M=
github.com/go-openapi/errors v0.19.9/go.mod h1:cM//ZKUKyO06HSwqAelJ5NsEMMcpa6VpXe8DOa1Mi1M=
github.com/go-openapi/errors v0.20.1 h1:j23mMDtRxMwIobkpId7sWh7Ddcx4ivaoqUbfXx5P+a8=
github.com/go-openapi/errors v0.20.1/go.mod h1:cM//ZKUKyO06HSwqAelJ5NsEMMcpa6VpXe8DOa1Mi1M=
github.com/go-openapi/inflect v0.19.0 h1:9jCH9scKIbHeV9m12SmPilScz6krDxKRasNNSNPXu/4=
github.com/go-openapi/inflect v0.19.0/go.mod h1:lHpZVlpIQqLyKwJ4N+YSc9hchQy/i12fJykb83CRBH4=
github.com/go-openapi/jsonpointer v0.17.0/go.mod h1:cOnomiV+CVVwFLk0A/MExoFMjwdsUdVpsRhURCKh+3M=
github.com/go-openapi/jsonpointer v0.18.0/go.mod h1:cOnomiV+CVVwFLk0A/MExoFMjwdsUdVpsRhURCKh+3M=
github.com/go-openapi/jsonpointer v0.19.2/go.mod h1:3akKfEdA7DF1sugOqz1dVQHBcuDBPKZGEoHC/NkiQRg=
github.com/go-openapi/jsonpointer v0.19.3/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg=
github.com/go-openapi/jsonpointer v0.19.5 h1:gZr+CIYByUqjcgeLXnQu2gHYQC9o73G2XUeOFYEICuY=
github.com/go-openapi/jsonpointer v0.19.5/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg=
github.com/go-openapi/jsonreference v0.17.0/go.mod h1:g4xxGn04lDIRh0GJb5QlpE3HfopLOL6uZrK/VgnsK9I=
github.com/go-openapi/jsonreference v0.18.0/go.mod h1:g4xxGn04lDIRh0GJb5QlpE3HfopLOL6uZrK/VgnsK9I=
github.com/go-openapi/jsonreference v0.19.2/go.mod h1:jMjeRr2HHw6nAVajTXJ4eiUwohSTlpa0o73RUL1owJc=
github.com/go-openapi/jsonreference v0.19.3/go.mod h1:rjx6GuL8TTa9VaixXglHmQmIL98+wF9xc8zWvFonSJ8=
github.com/go-openapi/jsonreference v0.19.5 h1:1WJP/wi4OjB4iV8KVbH73rQaoialJrqv8gitZLxGLtM=
github.com/go-openapi/jsonreference v0.19.5/go.mod h1:RdybgQwPxbL4UEjuAruzK1x3nE69AqPYEJeo/TWfEeg=
github.com/go-openapi/loads v0.17.0/go.mod h1:72tmFy5wsWx89uEVddd0RjRWPZm92WRLhf7AC+0+OOU=
github.com/go-openapi/loads v0.18.0/go.mod h1:72tmFy5wsWx89uEVddd0RjRWPZm92WRLhf7AC+0+OOU=
github.com/go-openapi/loads v0.19.0/go.mod h1:72tmFy5wsWx89uEVddd0RjRWPZm92WRLhf7AC+0+OOU=
github.com/go-openapi/loads v0.19.2/go.mod h1:QAskZPMX5V0C2gvfkGZzJlINuP7Hx/4+ix5jWFxsNPs=
github.com/go-openapi/loads v0.19.3/go.mod h1:YVfqhUCdahYwR3f3iiwQLhicVRvLlU/WO5WPaZvcvSI=
github.com/go-openapi/loads v0.19.5/go.mod h1:dswLCAdonkRufe/gSUC3gN8nTSaB9uaS2es0x5/IbjY=
github.com/go-openapi/loads v0.19.6/go.mod h1:brCsvE6j8mnbmGBh103PT/QLHfbyDxA4hsKvYBNEGVc=
github.com/go-openapi/loads v0.19.7/go.mod h1:brCsvE6j8mnbmGBh103PT/QLHfbyDxA4hsKvYBNEGVc=
github.com/go-openapi/loads v0.20.0 h1:Pymw1O8zDmWeNv4kVsHd0W3cvgdp8juRa4U/U/8D/Pk=
github.com/go-openapi/loads v0.20.0/go.mod h1:2LhKquiE513rN5xC6Aan6lYOSddlL8Mp20AW9kpviM4=
github.com/go-openapi/runtime v0.0.0-20180920151709-4f900dc2ade9/go.mod h1:6v9a6LTXWQCdL8k1AO3cvqx5OtZY/Y9wKTgaoP6YRfA=
github.com/go-openapi/runtime v0.19.0/go.mod h1:OwNfisksmmaZse4+gpV3Ne9AyMOlP1lt4sK4FXt0O64=
github.com/go-openapi/runtime v0.19.4/go.mod h1:X277bwSUBxVlCYR3r7xgZZGKVvBd/29gLDlFGtJ8NL4=
github.com/go-openapi/runtime v0.19.15/go.mod h1:dhGWCTKRXlAfGnQG0ONViOZpjfg0m2gUt9nTQPQZuoo=
github.com/go-openapi/runtime v0.19.16/go.mod h1:5P9104EJgYcizotuXhEuUrzVc+j1RiSjahULvYmlv98=
github.com/go-openapi/runtime v0.19.24 h1:TqagMVlRAOTwllE/7hNKx6rQ10O6T8ZzeJdMjSTKaD4=
github.com/go-openapi/runtime v0.19.24/go.mod h1:Lm9YGCeecBnUUkFTxPC4s1+lwrkJ0pthx8YvyjCfkgk=
github.com/go-openapi/spec v0.17.0/go.mod h1:XkF/MOi14NmjsfZ8VtAKf8pIlbZzyoTvZsdfssdxcBI=
github.com/go-openapi/spec v0.18.0/go.mod h1:XkF/MOi14NmjsfZ8VtAKf8pIlbZzyoTvZsdfssdxcBI=
github.com/go-openapi/spec v0.19.2/go.mod h1:sCxk3jxKgioEJikev4fgkNmwS+3kuYdJtcsZsD5zxMY=
github.com/go-openapi/spec v0.19.3/go.mod h1:FpwSN1ksY1eteniUU7X0N/BgJ7a4WvBFVA8Lj9mJglo=
github.com/go-openapi/spec v0.19.6/go.mod h1:Hm2Jr4jv8G1ciIAo+frC/Ft+rR2kQDh8JHKHb3gWUSk=
github.com/go-openapi/spec v0.19.8/go.mod h1:Hm2Jr4jv8G1ciIAo+frC/Ft+rR2kQDh8JHKHb3gWUSk=
github.com/go-openapi/spec v0.19.15/go.mod h1:+81FIL1JwC5P3/Iuuozq3pPE9dXdIEGxFutcFKaVbmU=
github.com/go-openapi/spec v0.20.0 h1:HGLc8AJ7ynOxwv0Lq4TsnwLsWMawHAYiJIFzbcML86I=
github.com/go-openapi/spec v0.20.0/go.mod h1:+81FIL1JwC5P3/Iuuozq3pPE9dXdIEGxFutcFKaVbmU=
github.com/go-openapi/strfmt v0.17.0/go.mod h1:P82hnJI0CXkErkXi8IKjPbNBM6lV6+5pLP5l494TcyU=
github.com/go-openapi/strfmt v0.18.0/go.mod h1:P82hnJI0CXkErkXi8IKjPbNBM6lV6+5pLP5l494TcyU=
github.com/go-openapi/strfmt v0.19.0/go.mod h1:+uW+93UVvGGq2qGaZxdDeJqSAqBqBdl+ZPMF/cC8nDY=
github.com/go-openapi/strfmt v0.19.2/go.mod h1:0yX7dbo8mKIvc3XSKp7MNfxw4JytCfCD6+bY1AVL9LU=
github.com/go-openapi/strfmt v0.19.3/go.mod h1:0yX7dbo8mKIvc3XSKp7MNfxw4JytCfCD6+bY1AVL9LU=
github.com/go-openapi/strfmt v0.19.4/go.mod h1:eftuHTlB/dI8Uq8JJOyRlieZf+WkkxUuk0dgdHXr2Qk=
github.com/go-openapi/strfmt v0.19.5/go.mod h1:eftuHTlB/dI8Uq8JJOyRlieZf+WkkxUuk0dgdHXr2Qk=
github.com/go-openapi/strfmt v0.19.11 h1:0+YvbNh05rmBkgztd6zHp4OCFn7Mtu30bn46NQo2ZRw=
github.com/go-openapi/strfmt v0.19.11/go.mod h1:UukAYgTaQfqJuAFlNxxMWNvMYiwiXtLsF2VwmoFtbtc=
github.com/go-openapi/swag v0.17.0/go.mod h1:AByQ+nYG6gQg71GINrmuDXCPWdL640yX49/kXLo40Tg=
github.com/go-openapi/swag v0.18.0/go.mod h1:AByQ+nYG6gQg71GINrmuDXCPWdL640yX49/kXLo40Tg=
github.com/go-openapi/swag v0.19.2/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk=
github.com/go-openapi/swag v0.19.5/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk=
github.com/go-openapi/swag v0.19.7/go.mod h1:ao+8BpOPyKdpQz3AOJfbeEVpLmWAvlT1IfTe5McPyhY=
github.com/go-openapi/swag v0.19.9/go.mod h1:ao+8BpOPyKdpQz3AOJfbeEVpLmWAvlT1IfTe5McPyhY=
github.com/go-openapi/swag v0.19.12 h1:Bc0bnY2c3AoF7Gc+IMIAQQsD8fLHjHpc19wXvYuayQI=
github.com/go-openapi/swag v0.19.12/go.mod h1:eFdyEBkTdoAf/9RXBvj4cr1nH7GD8Kzo5HTt47gr72M=
github.com/go-openapi/validate v0.18.0/go.mod h1:Uh4HdOzKt19xGIGm1qHf/ofbX1YQ4Y+MYsct2VUrAJ4=
github.com/go-openapi/validate v0.19.2/go.mod h1:1tRCw7m3jtI8eNWEEliiAqUIcBztB2KDnRCRMUi7GTA=
github.com/go-openapi/validate v0.19.3/go.mod h1:90Vh6jjkTn+OT1Eefm0ZixWNFjhtOH7vS9k0lo6zwJo=
github.com/go-openapi/validate v0.19.10/go.mod h1:RKEZTUWDkxKQxN2jDT7ZnZi2bhZlbNMAuKvKB+IaGx8=
github.com/go-openapi/validate v0.19.12/go.mod h1:Rzou8hA/CBw8donlS6WNEUQupNvUZ0waH08tGe6kAQ4=
github.com/go-openapi/validate v0.19.15/go.mod h1:tbn/fdOwYHgrhPBzidZfJC2MIVvs9GA7monOmWBbeCI=
github.com/go-openapi/validate v0.20.0 h1:pzutNCCBZGZlE+u8HD3JZyWdc/TVbtVwlWUp8/vgUKk=
github.com/go-openapi/validate v0.20.0/go.mod h1:b60iJT+xNNLfaQJUqLI7946tYiFEOuE9E4k54HpKcJ0=
github.com/go-playground/assert/v2 v2.0.1 h1:MsBgLAaY856+nPRTKrp3/OZK38U/wa0CcBYNjji3q3A=
github.com/go-playground/assert/v2 v2.0.1/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4=
github.com/go-playground/locales v0.13.0/go.mod h1:taPMhCMXrRLJO55olJkUXHZBHCxTMfnGwq/HNwmWNS8=
github.com/go-playground/locales v0.14.0 h1:u50s323jtVGugKlcYeyzC0etD1HifMjqmJqb8WugfUU=
github.com/go-playground/locales v0.14.0/go.mod h1:sawfccIbzZTqEDETgFXqTho0QybSa7l++s0DH+LDiLs=
github.com/go-playground/universal-translator v0.17.0/go.mod h1:UkSxE5sNxxRwHyU+Scu5vgOQjsIJAF8j9muTVoKLVtA=
github.com/go-playground/universal-translator v0.18.0 h1:82dyy6p4OuJq4/CByFNOn/jYrnRPArHwAcmLoJZxyho=
github.com/go-playground/universal-translator v0.18.0/go.mod h1:UvRDBj+xPUEGrFYl+lu/H90nyDXpg0fqeB/AQUGNTVA=
github.com/go-playground/validator/v10 v10.4.1/go.mod h1:nlOn6nFhuKACm19sB/8EGNn9GlaMV7XkbRSipzJ0Ii4=
github.com/go-playground/validator/v10 v10.10.0 h1:I7mrTYv78z8k8VXa/qJlOlEXn/nBh+BF8dHX5nt/dr0=
github.com/go-playground/validator/v10 v10.10.0/go.mod h1:74x4gJWsvQexRdW8Pn3dXSGrTK4nAUsbPlLADvpJkos=
github.com/go-sql-driver/mysql v1.5.0/go.mod h1:DCzpHaOWr8IXmIStZouvnhqoel9Qv2LBy8hT2VhHyBg=
github.com/go-sql-driver/mysql v1.6.0 h1:BCTh4TKNUYmOmMUcQ3IipzF5prigylS7XXjEkfCHuOE=
github.com/go-sql-driver/mysql v1.6.0/go.mod h1:DCzpHaOWr8IXmIStZouvnhqoel9Qv2LBy8hT2VhHyBg=
github.com/go-stack/stack v1.8.0 h1:5SgMzNM5HxrEjV0ww2lTmX6E2Izsfxas4+YHWRs3Lsk=
github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
github.com/go-test/deep v1.0.3 h1:ZrJSEWsXzPOxaZnFteGEfooLba+ju3FYIbOrS+rQd68=
github.com/gobuffalo/attrs v0.0.0-20190224210810-a9411de4debd/go.mod h1:4duuawTqi2wkkpB4ePgWMaai6/Kc6WEz83bhFwpHzj0=
github.com/gobuffalo/depgen v0.0.0-20190329151759-d478694a28d3/go.mod h1:3STtPUQYuzV0gBVOY3vy6CfMm/ljR4pABfrTeHNLHUY=
github.com/gobuffalo/depgen v0.1.0/go.mod h1:+ifsuy7fhi15RWncXQQKjWS9JPkdah5sZvtHc2RXGlg=
github.com/gobuffalo/envy v1.6.15/go.mod h1:n7DRkBerg/aorDM8kbduw5dN3oXGswK5liaSCx4T5NI=
github.com/gobuffalo/envy v1.7.0/go.mod h1:n7DRkBerg/aorDM8kbduw5dN3oXGswK5liaSCx4T5NI=
github.com/gobuffalo/flect v0.1.0/go.mod h1:d2ehjJqGOH/Kjqcoz+F7jHTBbmDb38yXA598Hb50EGs=
github.com/gobuffalo/flect v0.1.1/go.mod h1:8JCgGVbRjJhVgD6399mQr4fx5rRfGKVzFjbj6RE/9UI=
github.com/gobuffalo/flect v0.1.3/go.mod h1:8JCgGVbRjJhVgD6399mQr4fx5rRfGKVzFjbj6RE/9UI=
github.com/gobuffalo/genny v0.0.0-20190329151137-27723ad26ef9/go.mod h1:rWs4Z12d1Zbf19rlsn0nurr75KqhYp52EAGGxTbBhNk=
github.com/gobuffalo/genny v0.0.0-20190403191548-3ca520ef0d9e/go.mod h1:80lIj3kVJWwOrXWWMRzzdhW3DsrdjILVil/SFKBzF28=
github.com/gobuffalo/genny v0.1.0/go.mod h1:XidbUqzak3lHdS//TPu2OgiFB+51Ur5f7CSnXZ/JDvo=
github.com/gobuffalo/genny v0.1.1/go.mod h1:5TExbEyY48pfunL4QSXxlDOmdsD44RRq4mVZ0Ex28Xk=
github.com/gobuffalo/gitgen v0.0.0-20190315122116-cc086187d211/go.mod h1:vEHJk/E9DmhejeLeNt7UVvlSGv3ziL+djtTr3yyzcOw=
github.com/gobuffalo/gogen v0.0.0-20190315121717-8f38393713f5/go.mod h1:V9QVDIxsgKNZs6L2IYiGR8datgMhB577vzTDqypH360=
github.com/gobuffalo/gogen v0.1.0/go.mod h1:8NTelM5qd8RZ15VjQTFkAW6qOMx5wBbW4dSCS3BY8gg=
github.com/gobuffalo/gogen v0.1.1/go.mod h1:y8iBtmHmGc4qa3urIyo1shvOD8JftTtfcKi+71xfDNE=
github.com/gobuffalo/logger v0.0.0-20190315122211-86e12af44bc2/go.mod h1:QdxcLw541hSGtBnhUc4gaNIXRjiDppFGaDqzbrBd3v8=
github.com/gobuffalo/mapi v1.0.1/go.mod h1:4VAGh89y6rVOvm5A8fKFxYG+wIW6LO1FMTG9hnKStFc=
github.com/gobuffalo/mapi v1.0.2/go.mod h1:4VAGh89y6rVOvm5A8fKFxYG+wIW6LO1FMTG9hnKStFc=
github.com/gobuffalo/packd v0.0.0-20190315124812-a385830c7fc0/go.mod h1:M2Juc+hhDXf/PnmBANFCqx4DM3wRbgDvnVWeG2RIxq4=
github.com/gobuffalo/packd v0.1.0/go.mod h1:M2Juc+hhDXf/PnmBANFCqx4DM3wRbgDvnVWeG2RIxq4=
github.com/gobuffalo/packr/v2 v2.0.9/go.mod h1:emmyGweYTm6Kdper+iywB6YK5YzuKchGtJQZ0Odn4pQ=
github.com/gobuffalo/packr/v2 v2.2.0/go.mod h1:CaAwI0GPIAv+5wKLtv8Afwl+Cm78K/I/VCm/3ptBN+0=
github.com/gobuffalo/syncx v0.0.0-20190224160051-33c29581e754/go.mod h1:HhnNqWY95UYwwW3uSASeV7vtgYkT2t16hJgV3AEPUpw=
github.com/godbus/dbus v4.1.0+incompatible/go.mod h1:/YcGZj5zSblfDWMMoOzV4fas9FZnQYTkDnsGvmh2Grw=
github.com/gofrs/uuid v4.0.0+incompatible h1:1SD/1F5pU8p29ybwgQSwpQk+mwdRrXCYuPhW6m+TnJw=
github.com/gofrs/uuid v4.0.0+incompatible/go.mod h1:b2aQJv3Z4Fp6yNu3cdSllBxTCLRxnplIgP/c0N/04lM=
github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/golang-jwt/jwt/v4 v4.2.0 h1:besgBTC8w8HjP6NzQdxwKH9Z5oQMZ24ThTrHp3cZ8eU=
github.com/golang-jwt/jwt/v4 v4.2.0/go.mod h1:/xlHOz8bRuivTWchD4jCa+NbatV+wEUSzwAxVc6locg=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/glog v0.0.0-20210429001901-424d2337a529 h1:2voWjNECnrZRbfwXxHB1/j8wa6xdKn85B5NzgVL/pTU=
github.com/golang/glog v0.0.0-20210429001901-424d2337a529/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y=
github.com/golang/mock v1.4.0/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
github.com/golang/mock v1.4.1/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
github.com/golang/mock v1.4.3/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
github.com/golang/mock v1.4.4/go.mod h1:l3mdAwkq5BuhzHwde/uurv3sEJeZMXNpwsxVWU71h+4=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw=
github.com/golang/protobuf v1.3.4/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw=
github.com/golang/protobuf v1.3.5/go.mod h1:6O5/vntMXwX2lRkT1hjjk0nAC1IDOTvTlVgjlRvqsdk=
github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8=
github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA=
github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs=
github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8=
github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
github.com/golang/protobuf v1.5.2 h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.4.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.4/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.7 h1:81/ik6ipDQS2aGcBfIN5dHDB36BwrStyeAQquSYCV4o=
github.com/google/go-cmp v0.5.7/go.mod h1:n+brtR0CgQNWTVd5ZUFpTBC8YFBDLK/h/bpaJ8/DtOE=
github.com/google/go-querystring v1.0.0 h1:Xkwi/a1rcvNg1PPYe5vI8GbeBY/jrVuDX5ASuANWTrk=
github.com/google/go-querystring v1.0.0/go.mod h1:odCYkC5MyYFN7vkCjXpyrEuKhc/BUO6wN/zVPAxq5ck=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/logger v1.1.1/go.mod h1:BkeJZ+1FhQ+/d087r4dzojEg1u2ZX+ZqG1jTUrLM+zQ=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
github.com/google/martian/v3 v3.0.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0=
github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20200212024743-f11f1df84d12/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20200430221834-fc25d7d30c6d/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20200708004538-1a94d8640e99/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
github.com/google/uuid v1.0.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/winops v0.0.0-20211216095627-f0e86eb1453b h1:THwEE9J2wPxF3BZm7WjLCASMcM7ctFzqLpTsCGh7gDY=
github.com/google/winops v0.0.0-20211216095627-f0e86eb1453b/go.mod h1:ShbX8v8clPm/3chw9zHVwtW3QhrFpL8mXOwNxClt4pg=
github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
github.com/goombaio/namegenerator v0.0.0-20181006234301-989e774b106e h1:XmA6L9IPRdUr28a+SK/oMchGgQy159wvzXA5tJ7l+40=
github.com/goombaio/namegenerator v0.0.0-20181006234301-989e774b106e/go.mod h1:AFIo+02s+12CEg8Gzz9kzhCbmbq6JcKNrhHffCGA9z4=
github.com/gorilla/mux v1.7.3 h1:gnP5JzjVOuiZD07fKKToCAOjS0yOpj/qPETTXCCS6hw=
github.com/gorilla/mux v1.7.3/go.mod h1:1lud6UwP+6orDFRuTfBEV8e9/aOM/c4fVVCaMa2zaAs=
github.com/groob/plist v0.0.0-20210519001750-9f754062e6d6/go.mod h1:itkABA+w2cw7x5nYUS/pLRef6ludkZKOigbROmCTaFw=
github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFbcEtjT1g+wF4CSlocrBnw=
github.com/hashicorp/go-hclog v0.14.1/go.mod h1:whpDNt7SSdeAju8AWKIWsul05p54N/39EeqMAyrmvFQ=
github.com/hashicorp/go-hclog v1.0.0 h1:bkKf0BeBXcSYa7f5Fyi9gMuQ8gNsxeiNpZjR6VxNZeo=
github.com/hashicorp/go-hclog v1.0.0/go.mod h1:whpDNt7SSdeAju8AWKIWsul05p54N/39EeqMAyrmvFQ=
github.com/hashicorp/go-plugin v1.4.2 h1:yFvG3ufXXpqiMiZx9HLcaK3XbIqQ1WJFR/F1a2CuVw0=
github.com/hashicorp/go-plugin v1.4.2/go.mod h1:5fGEH17QVwTTcR0zV7yhDPLLmFX9YSZ38b18Udy6vYQ=
github.com/hashicorp/go-version v1.2.1 h1:zEfKbn2+PDgroKdiOzqiE8rsmLqU2uwi5PB5pBJ3TkI=
github.com/hashicorp/go-version v1.2.1/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA=
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/hcl/v2 v2.13.0 h1:0Apadu1w6M11dyGFxWnmhhcMjkbAiKCv7G1r/2QgCNc=
github.com/hashicorp/hcl/v2 v2.13.0/go.mod h1:e4z5nxYlWNPdDSNYX+ph14EvWYMFm3eP0zIUqPc2jr0=
github.com/hashicorp/yamux v0.0.0-20180604194846-3520598351bb h1:b5rjCoWHc7eqmAS4/qyk21ZsHyb6Mxv/jykxvNTkU4M=
github.com/hashicorp/yamux v0.0.0-20180604194846-3520598351bb/go.mod h1:+NfK9FKeTrX5uv1uIXGdwYDTeHna2qgaIlx54MXqjAM=
github.com/hinshun/vt10x v0.0.0-20180616224451-1954e6464174 h1:WlZsjVhE8Af9IcZDGgJGQpNflI3+MJSBhsgT5PCtzBQ=
github.com/hinshun/vt10x v0.0.0-20180616224451-1954e6464174/go.mod h1:DqJ97dSdRW1W22yXSB90986pcOyQ7r45iio1KN2ez1A=
github.com/huandu/xstrings v1.3.1/go.mod h1:y5/lhBue+AyNmUVz9RLU9xbLR0o4KIIExikq4ovT0aE=
github.com/huandu/xstrings v1.3.2 h1:L18LIDzqlW6xN2rEkpdV8+oL/IXWJ1APd+vsdYy4Wdw=
github.com/huandu/xstrings v1.3.2/go.mod h1:y5/lhBue+AyNmUVz9RLU9xbLR0o4KIIExikq4ovT0aE=
github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/imdario/mergo v0.3.11/go.mod h1:jmQim1M+e3UYxmgPu/WyfjB3N3VflVyUjjjwH0dnCYA=
github.com/imdario/mergo v0.3.12 h1:b6R2BslTbIEToALKP7LxUvijTsNI9TAe80pLWN2g/HU=
github.com/imdario/mergo v0.3.12/go.mod h1:jmQim1M+e3UYxmgPu/WyfjB3N3VflVyUjjjwH0dnCYA=
github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
github.com/inconshreveable/mousetrap v1.0.1 h1:U3uMjPSQEBMNp1lFxmllqCPM6P5u/Xq7Pgzkat/bFNc=
github.com/inconshreveable/mousetrap v1.0.1/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/ivanpirog/coloredcobra v1.0.1 h1:aURSdEmlR90/tSiWS0dMjdwOvCVUeYLfltLfbgNxrN4=
github.com/ivanpirog/coloredcobra v1.0.1/go.mod h1:iho4nEKcnwZFiniGSdcgdvRgZNjxm+h20acv8vqmN6Q=
github.com/jackc/chunkreader v1.0.0/go.mod h1:RT6O25fNZIuasFJRyZ4R/Y2BbhasbmZXF9QQ7T3kePo=
github.com/jackc/chunkreader/v2 v2.0.0/go.mod h1:odVSm741yZoC3dpHEUXIqA9tQRhFrgOHwnPIn9lDKlk=
github.com/jackc/chunkreader/v2 v2.0.1 h1:i+RDz65UE+mmpjTfyz0MoVTnzeYxroil2G82ki7MGG8=
github.com/jackc/chunkreader/v2 v2.0.1/go.mod h1:odVSm741yZoC3dpHEUXIqA9tQRhFrgOHwnPIn9lDKlk=
github.com/jackc/pgconn v0.0.0-20190420214824-7e0022ef6ba3/go.mod h1:jkELnwuX+w9qN5YIfX0fl88Ehu4XC3keFuOJJk9pcnA=
github.com/jackc/pgconn v0.0.0-20190824142844-760dd75542eb/go.mod h1:lLjNuW/+OfW9/pnVKPazfWOgNfH2aPem8YQ7ilXGvJE=
github.com/jackc/pgconn v0.0.0-20190831204454-2fabfa3c18b7/go.mod h1:ZJKsE/KZfsUgOEh9hBm+xYTstcNHg7UPMVJqRfQxq4s=
github.com/jackc/pgconn v1.8.0/go.mod h1:1C2Pb36bGIP9QHGBYCjnyhqu7Rv3sGshaQUvmfGIB/o=
github.com/jackc/pgconn v1.9.0/go.mod h1:YctiPyvzfU11JFxoXokUOOKQXQmDMoJL9vJzHH8/2JY=
github.com/jackc/pgconn v1.9.1-0.20210724152538-d89c8390a530/go.mod h1:4z2w8XhRbP1hYxkpTuBjTS3ne3J48K83+u0zoyvg2pI=
github.com/jackc/pgconn v1.10.1 h1:DzdIHIjG1AxGwoEEqS+mGsURyjt4enSmqzACXvVzOT8=
github.com/jackc/pgconn v1.10.1/go.mod h1:4z2w8XhRbP1hYxkpTuBjTS3ne3J48K83+u0zoyvg2pI=
github.com/jackc/pgio v1.0.0 h1:g12B9UwVnzGhueNavwioyEEpAmqMe1E/BN9ES+8ovkE=
github.com/jackc/pgio v1.0.0/go.mod h1:oP+2QK2wFfUWgr+gxjoBH9KGBb31Eio69xUb0w5bYf8=
github.com/jackc/pgmock v0.0.0-20190831213851-13a1b77aafa2/go.mod h1:fGZlG77KXmcq05nJLRkk0+p82V8B8Dw8KN2/V9c/OAE=
github.com/jackc/pgmock v0.0.0-20201204152224-4fe30f7445fd/go.mod h1:hrBW0Enj2AZTNpt/7Y5rr2xe/9Mn757Wtb2xeBzPv2c=
github.com/jackc/pgmock v0.0.0-20210724152146-4ad1a8207f65 h1:DadwsjnMwFjfWc9y5Wi/+Zz7xoE5ALHsRQlOctkOiHc=
github.com/jackc/pgmock v0.0.0-20210724152146-4ad1a8207f65/go.mod h1:5R2h2EEX+qri8jOWMbJCtaPWkrrNc7OHwsp2TCqp7ak=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
github.com/jackc/pgproto3 v1.1.0/go.mod h1:eR5FA3leWg7p9aeAqi37XOTgTIbkABlvcPB3E5rlc78=
github.com/jackc/pgproto3/v2 v2.0.0-alpha1.0.20190420180111-c116219b62db/go.mod h1:bhq50y+xrl9n5mRYyCBFKkpRVTLYJVWeCc+mEAI3yXA=
github.com/jackc/pgproto3/v2 v2.0.0-alpha1.0.20190609003834-432c2951c711/go.mod h1:uH0AWtUmuShn0bcesswc4aBTWGvw0cAxIJp+6OB//Wg=
github.com/jackc/pgproto3/v2 v2.0.0-rc3/go.mod h1:ryONWYqW6dqSg1Lw6vXNMXoBJhpzvWKnT95C46ckYeM=
github.com/jackc/pgproto3/v2 v2.0.0-rc3.0.20190831210041-4c03ce451f29/go.mod h1:ryONWYqW6dqSg1Lw6vXNMXoBJhpzvWKnT95C46ckYeM=
github.com/jackc/pgproto3/v2 v2.0.6/go.mod h1:WfJCnwN3HIg9Ish/j3sgWXnAfK8A9Y0bwXYU5xKaEdA=
github.com/jackc/pgproto3/v2 v2.1.1/go.mod h1:WfJCnwN3HIg9Ish/j3sgWXnAfK8A9Y0bwXYU5xKaEdA=
github.com/jackc/pgproto3/v2 v2.2.0 h1:r7JypeP2D3onoQTCxWdTpCtJ4D+qpKr0TxvoyMhZ5ns=
github.com/jackc/pgproto3/v2 v2.2.0/go.mod h1:WfJCnwN3HIg9Ish/j3sgWXnAfK8A9Y0bwXYU5xKaEdA=
github.com/jackc/pgservicefile v0.0.0-20200714003250-2b9c44734f2b h1:C8S2+VttkHFdOOCXJe+YGfa4vHYwlt4Zx+IVXQ97jYg=
github.com/jackc/pgservicefile v0.0.0-20200714003250-2b9c44734f2b/go.mod h1:vsD4gTJCa9TptPL8sPkXrLZ+hDuNrZCnj29CQpr4X1E=
github.com/jackc/pgtype v0.0.0-20190421001408-4ed0de4755e0/go.mod h1:hdSHsc1V01CGwFsrv11mJRHWJ6aifDLfdV3aVjFF0zg=
github.com/jackc/pgtype v0.0.0-20190824184912-ab885b375b90/go.mod h1:KcahbBH1nCMSo2DXpzsoWOAfFkdEtEJpPbVLq8eE+mc=
github.com/jackc/pgtype v0.0.0-20190828014616-a8802b16cc59/go.mod h1:MWlu30kVJrUS8lot6TQqcg7mtthZ9T0EoIBFiJcmcyw=
github.com/jackc/pgtype v1.8.1-0.20210724151600-32e20a603178/go.mod h1:C516IlIV9NKqfsMCXTdChteoXmwgUceqaLfjg2e3NlM=
github.com/jackc/pgtype v1.9.1 h1:MJc2s0MFS8C3ok1wQTdQxWuXQcB6+HwAm5x1CzW7mf0=
github.com/jackc/pgtype v1.9.1/go.mod h1:LUMuVrfsFfdKGLw+AFFVv6KtHOFMwRgDDzBt76IqCA4=
github.com/jackc/pgx/v4 v4.0.0-20190420224344-cc3461e65d96/go.mod h1:mdxmSJJuR08CZQyj1PVQBHy9XOp5p8/SHH6a0psbY9Y=
github.com/jackc/pgx/v4 v4.0.0-20190421002000-1b8f0016e912/go.mod h1:no/Y67Jkk/9WuGR0JG/JseM9irFbnEPbuWV2EELPNuM=
github.com/jackc/pgx/v4 v4.0.0-pre1.0.20190824185557-6972a5742186/go.mod h1:X+GQnOEnf1dqHGpw7JmHqHc1NxDoalibchSk9/RWuDc=
github.com/jackc/pgx/v4 v4.12.1-0.20210724153913-640aa07df17c/go.mod h1:1QD0+tgSXP7iUjYm9C1NxKhny7lq6ee99u/z+IHFcgs=
github.com/jackc/pgx/v4 v4.14.1 h1:71oo1KAGI6mXhLiTMn6iDFcp3e7+zon/capWjl2OEFU=
github.com/jackc/pgx/v4 v4.14.1/go.mod h1:RgDuE4Z34o7XE92RpLsvFiOEfrAUT0Xt2KxvX73W06M=
github.com/jackc/puddle v0.0.0-20190413234325-e4ced69a3a2b/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jackc/puddle v0.0.0-20190608224051-11cab39313c9/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jackc/puddle v1.1.3/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jackc/puddle v1.2.0/go.mod h1:m4B5Dj62Y0fbyuIc15OsIqK0+JU8nkqQjsgx7dvjSWk=
github.com/jarcoal/httpmock v1.1.0 h1:F47ChZj1Y2zFsCXxNkBPwNNKnAyOATcdQibk0qEdVCE=
github.com/jarcoal/httpmock v1.1.0/go.mod h1:ATjnClrvW/3tijVmpL/va5Z3aAyGvqU3gCT8nX0Txik=
github.com/jhump/protoreflect v1.6.0 h1:h5jfMVslIg6l29nsMs0D8Wj17RDVdNYti0vDN/PZZoE=
github.com/jhump/protoreflect v1.6.0/go.mod h1:eaTn3RZAmMBcV0fifFvlm6VHNz3wSkYyXYWUh7ymB74=
github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg=
github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo=
github.com/jmespath/go-jmespath/internal/testify v1.5.1 h1:shLQSRRSCCPj3f2gpwzGwWFoC7ycTf1rcQZHOlsJ6N8=
github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U=
github.com/joho/godotenv v1.3.0/go.mod h1:7hK45KPybAkOC6peb+G5yklZfMxEjkZhHbwpqxOKXbg=
github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY=
github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
github.com/jpillora/backoff v1.0.0/go.mod h1:J/6gKK9jxlEcS3zixgDgUAsiuZ7yrSoa/FX5e0EB2j4=
github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
github.com/json-iterator/go v1.1.9/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/json-iterator/go v1.1.11/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
github.com/jszwec/csvutil v1.5.1 h1:c3GFBhj6DFMUl4dMK3+B6rz2+LWWS/e9VJiVJ9t9kfQ=
github.com/jszwec/csvutil v1.5.1/go.mod h1:Rpu7Uu9giO9subDyMCIQfHVDuLrcaC36UA4YcJjGBkg=
github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
github.com/julienschmidt/httprouter v1.3.0/go.mod h1:JR6WtHb+2LUe8TCKY3cZOxFyyO8IZAc4RVcycCCAKdM=
github.com/karrick/godirwalk v1.8.0/go.mod h1:H5KPZjojv4lE+QYImBI8xVtrBRgYrIVsaRPx4tDPEn4=
github.com/karrick/godirwalk v1.10.3/go.mod h1:RoGL9dQei4vP9ilrpETWE8CLOZ1kiN0LhBygSwrAsHA=
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 h1:Z9n2FFNUXsshfwJMBgNA0RU6/i7WVaAegv3PtuIHPMs=
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51/go.mod h1:CzGEWj7cYgsdH8dAjBGEr58BoE7ScuLd+fwFZ44+/x8=
github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/klauspost/compress v1.9.5/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A=
github.com/klauspost/compress v1.13.6/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk=
github.com/klauspost/compress v1.15.7 h1:7cgTQxJCU/vy+oP/E3B9RGbQTgbiVzIJWIKOLoAsPok=
github.com/klauspost/compress v1.15.7/go.mod h1:PhcZ0MbTNciWF3rruxRgKxI5NkcHHrHUDtV4Yw2GlzU=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/konsorten/go-windows-terminal-sequences v1.0.2/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
github.com/kr/pretty v0.3.0 h1:WgNl7dwNpEZ6jJ9k1snq4pZsg7DOEN8hP9Xw0Tsjwk0=
github.com/kr/pretty v0.3.0/go.mod h1:640gp4NfQd8pI5XOwp5fnNeVWj67G7CFk/SaSQn7NBk=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/pty v1.1.4/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/pty v1.1.5/go.mod h1:9r2w37qlBe7rQ6e1fg1S/9xpWHSnaqNdHD3WcMdbPDA=
github.com/kr/pty v1.1.8 h1:AkaSdXYQOWeaO3neb8EM634ahkXXe3jYbVh/F9lq+GI=
github.com/kr/pty v1.1.8/go.mod h1:O1sed60cT9XZ5uDucP5qwvh+TE3NnUj51EiZO/lmSfw=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/kylelemons/godebug v0.0.0-20170820004349-d65d576e9348 h1:MtvEpTB6LX3vkb4ax0b5D2DHbNAUsen0Gx5wZoq3lV4=
github.com/leodido/go-urn v1.2.0/go.mod h1:+8+nEpDfqqsY+g338gtMEUOtuK+4dEMhiQEgxpxOKII=
github.com/leodido/go-urn v1.2.1 h1:BqpAaACuzVSgi/VLzGZIobT2z4v53pjosyNd9Yv6n/w=
github.com/leodido/go-urn v1.2.1/go.mod h1:zt4jvISO2HfUBqxjfIshjdMTYS56ZS/qv49ictyFfxY=
github.com/lib/pq v1.0.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/lib/pq v1.1.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/lib/pq v1.2.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/lib/pq v1.10.2/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
github.com/lib/pq v1.10.7 h1:p7ZhMD+KsSRozJr34udlUrhboJwWAgCg34+/ZZNvZZw=
github.com/lib/pq v1.10.7/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
github.com/lucasb-eyer/go-colorful v1.0.2/go.mod h1:0MS4r+7BZKSJ5mw4/S5MPN+qHFF1fYclkSPilDOKW0s=
github.com/lucasb-eyer/go-colorful v1.0.3/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=
github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
github.com/mailru/easyjson v0.0.0-20190312143242-1de009706dbe/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
github.com/mailru/easyjson v0.7.1/go.mod h1:KAzv3t3aY1NaHWoQz1+4F1ccyAH66Jk7yos7ldAVICs=
github.com/mailru/easyjson v0.7.6 h1:8yTIVnZgCoiM1TgqoeTl+LfU5Jg6/xL3QhGQnimLYnA=
github.com/mailru/easyjson v0.7.6/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
github.com/markbates/oncer v0.0.0-20181203154359-bf2de49a0be2/go.mod h1:Ld9puTsIW75CHf65OeIOkyKbteujpZVXDpWK6YGZbxE=
github.com/markbates/safe v1.0.1/go.mod h1:nAqgmRi7cY2nqMc92/bSEeQA+R4OheNU2T1kNSCBdG0=
github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ=
github.com/mattn/go-colorable v0.1.2/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE=
github.com/mattn/go-colorable v0.1.4/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE=
github.com/mattn/go-colorable v0.1.6/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=
github.com/mattn/go-colorable v0.1.9/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=
github.com/mattn/go-colorable v0.1.12 h1:jF+Du6AlPIjs2BiUiQlKOX0rt3SujHxPnksPKZbaA40=
github.com/mattn/go-colorable v0.1.12/go.mod h1:u5H1YNBxpqRaxsYJYSkiCWKzEfiAb1Gb520KVy5xxl4=
github.com/mattn/go-isatty v0.0.5/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
github.com/mattn/go-isatty v0.0.7/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
github.com/mattn/go-isatty v0.0.10/go.mod h1:qgIWMr58cqv1PHHyhnkY9lrL7etaEgOFcMEpPG5Rm84=
github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU=
github.com/mattn/go-isatty v0.0.14 h1:yVuAays6BHfxijgZPzw+3Zlu5yQgKGP2/hcQbHb7S9Y=
github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94=
github.com/mattn/go-runewidth v0.0.4/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU=
github.com/mattn/go-runewidth v0.0.8/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
github.com/mattn/go-runewidth v0.0.13 h1:lTGmDsbAYt5DmK6OnoV7EuIF1wEIFAcxld6ypU4OSgU=
github.com/mattn/go-runewidth v0.0.13/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
github.com/mattn/go-sqlite3 v1.14.15 h1:vfoHhTN1af61xCRSWzFIWzx2YskyMTwHLrExkBOjvxI=
github.com/mattn/go-sqlite3 v1.14.15/go.mod h1:2eHXhiwb8IkHr+BDWZGa96P6+rkvnG63S2DGjv9HUNg=
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369 h1:I0XW9+e1XWDxdcEniV4rQAIOPUGDq67JSCiRCgGCZLI=
github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369/go.mod h1:BSXmuO+STAnVfrANrmjBb36TMTDstsz7MSK+HVaYKv4=
github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b/go.mod h1:01TrycV0kFyexm33Z7vhZRXopbI8J3TDReVlkTgMUxE=
github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d h1:5PJl274Y63IEHC+7izoQE9x6ikvDFZS2mDVS3drnohI=
github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d/go.mod h1:01TrycV0kFyexm33Z7vhZRXopbI8J3TDReVlkTgMUxE=
github.com/mitchellh/copystructure v1.0.0/go.mod h1:SNtv71yrdKgLRyLFxmLdkAbkKEFWgYaq1OVrnRcwhnw=
github.com/mitchellh/copystructure v1.2.0 h1:vpKXTN4ewci03Vljg/q9QvCGUDttBOGBIa15WveJJGw=
github.com/mitchellh/copystructure v1.2.0/go.mod h1:qLl+cE2AmVv+CoeAwDPye/v+N2HKCj9FbZEVFJRxO9s=
github.com/mitchellh/go-testing-interface v0.0.0-20171004221916-a61a99592b77/go.mod h1:kRemZodwjscx+RGhAo8eIhFbs2+BFgRtFPeD/KE+zxI=
github.com/mitchellh/go-testing-interface v1.0.0 h1:fzU/JVNcaqHQEcVFAKeR41fkiLdIPrefOvVG1VZ96U0=
github.com/mitchellh/go-testing-interface v1.0.0/go.mod h1:kRemZodwjscx+RGhAo8eIhFbs2+BFgRtFPeD/KE+zxI=
github.com/mitchellh/go-wordwrap v1.0.1 h1:TLuKupo69TCn6TQSyGxwI1EblZZEsQ0vMlAFQflz0v0=
github.com/mitchellh/go-wordwrap v1.0.1/go.mod h1:R62XHJLzvMFRBbcrT7m7WgmE1eOyTSsCt+hzestvNj0=
github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
github.com/mitchellh/mapstructure v1.3.2/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
github.com/mitchellh/mapstructure v1.3.3/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
github.com/mitchellh/mapstructure v1.4.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyuac5Z2hdY=
github.com/mitchellh/mapstructure v1.5.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
github.com/mitchellh/reflectwalk v1.0.0/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw=
github.com/mitchellh/reflectwalk v1.0.2 h1:G2LzWKi524PWgd3mLHV8Y5k7s6XUvT0Gef6zxSIeXaQ=
github.com/mitchellh/reflectwalk v1.0.2/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw=
github.com/moby/term v0.0.0-20201216013528-df9cb8a40635 h1:rzf0wL0CHVc8CEsgyygG0Mn9CNCCPZqOPaz8RiiHYQk=
github.com/moby/term v0.0.0-20201216013528-df9cb8a40635/go.mod h1:FBS0z0QWA44HXygs7VXDUOGoN/1TV3RuWkLO04am3wc=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 h1:RWengNIwukTxcDr9M+97sNutRR1RKhG96O6jWumTTnw=
github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826/go.mod h1:TaXosZuwdSHYgviHp1DAtfrULt5eUgsSMsZf+YrPgl8=
github.com/montanaflynn/stats v0.0.0-20171201202039-1bf9dbcd8cbe/go.mod h1:wL8QJuTMNUDYhXwkmfOly8iTdp5TEcJFWZD2D7SIkUc=
github.com/morikuni/aec v1.0.0 h1:nP9CBfwrvYnBRgY6qfDQkygYDmYwOilePFkwzv4dU8A=
github.com/morikuni/aec v1.0.0/go.mod h1:BbKIizmSmc5MMPqRYbxO4ZU0S0+P200+tUnFx7PXmsc=
github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno=
github.com/nxadm/tail v1.4.6 h1:11TGpSHY7Esh/i/qnq02Jo5oVrI1Gue8Slbq0ujPZFQ=
github.com/nxadm/tail v1.4.6/go.mod h1:+ncqLTQzXmGhMZNUePPaPqPvBxHAIsmXswZKocGu+AU=
github.com/oklog/run v1.0.0 h1:Ru7dDtJNOyC66gQ5dQmaCa0qIsAUFY3sFpK1Xk8igrw=
github.com/oklog/run v1.0.0/go.mod h1:dlhp/R75TPv97u0XWUtDeV/lRKWPKSdTuV0TZvrmrQA=
github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
github.com/opencontainers/image-spec v1.0.2-0.20211117181255-693428a734f5 h1:q37d91F6BO4Jp1UqWiun0dUFYaqv6WsKTLTCaWv+8LY=
github.com/opencontainers/image-spec v1.0.2-0.20211117181255-693428a734f5/go.mod h1:BtxoFyWECRxE4U/7sNtV5W15zMzWCbyJoFRP3s7yZA0=
github.com/oschwald/geoip2-golang v1.4.0 h1:5RlrjCgRyIGDz/mBmPfnAF4h8k0IAcRv9PvrpOfz+Ug=
github.com/oschwald/geoip2-golang v1.4.0/go.mod h1:8QwxJvRImBH+Zl6Aa6MaIcs5YdlZSTKtzmPGzQqi9ng=
github.com/oschwald/maxminddb-golang v1.6.0/go.mod h1:DUJFucBg2cvqx42YmDa/+xHvb0elJtOm3o4aFQ/nb/w=
github.com/oschwald/maxminddb-golang v1.8.0 h1:Uh/DSnGoxsyp/KYbY1AuP0tYEwfs0sCph9p/UMXK/Hk=
github.com/oschwald/maxminddb-golang v1.8.0/go.mod h1:RXZtst0N6+FY/3qCNmZMBApR19cdQj43/NM9VkrNAis=
github.com/pbnjay/memory v0.0.0-20210728143218-7b4eea64cf58 h1:onHthvaw9LFnH4t2DcNVpwGmV9E1BkGknEliJkfwQj0=
github.com/pbnjay/memory v0.0.0-20210728143218-7b4eea64cf58/go.mod h1:DXv8WO4yhMYhSNPKjeNKa5WY9YCIEBRbNzFFPJbWO6Y=
github.com/pborman/uuid v1.2.0/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtPdI/k=
github.com/pelletier/go-toml v1.4.0/go.mod h1:PN7xzY2wHTK0K9p34ErDQMlFxa51Fk0OUruD3k1mMwo=
github.com/pelletier/go-toml v1.7.0/go.mod h1:vwGMzjaWMwyfHwgIBhI2YUM4fB6nL6lVAvS1LBMMhTE=
github.com/pierrec/lz4/v4 v4.1.15 h1:MO0/ucJhngq7299dKLwIMtgTfbkoSPF6AoMYDd8Q4q0=
github.com/pierrec/lz4/v4 v4.1.15/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v0.0.0-20151028094244-d8ed2627bdf0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
github.com/prometheus/client_golang v1.0.0/go.mod h1:db9x61etRT2tGnBNRi70OPL5FsnadC4Ky3P0J6CfImo=
github.com/prometheus/client_golang v1.7.1/go.mod h1:PY5Wy2awLA44sXw4AOSfFBetzPP4j5+D6mVACh+pe2M=
github.com/prometheus/client_golang v1.11.0 h1:HNkLOAEQMIDv/K+04rukrLx6ch7msSRwf3/SASFAGtQ=
github.com/prometheus/client_golang v1.11.0/go.mod h1:Z6t4BnS23TR94PD6BsDNk8yVqroYurpAkEiz0P2BEV0=
github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/client_model v0.1.0/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/client_model v0.2.0 h1:uq5h0d+GuxiXLJLNABMgp2qUWDPiLvgCzz2dUR+/W/M=
github.com/prometheus/client_model v0.2.0/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/common v0.4.1/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
github.com/prometheus/common v0.7.0/go.mod h1:DjGbpBbp5NYNiECxcL/VnbXCCaQpKd3tt26CguLLsqA=
github.com/prometheus/common v0.10.0/go.mod h1:Tlit/dnDKsSWFlCLTWaA1cyBgKHSMdTB80sz/V91rCo=
github.com/prometheus/common v0.26.0/go.mod h1:M7rCNAaPfAosfx8veZJCuw84e35h3Cfd9VFqTh1DIvc=
github.com/prometheus/common v0.30.0 h1:JEkYlQnpzrzQFxi6gnukFPdQ+ac82oRhzMcIduJu/Ug=
github.com/prometheus/common v0.30.0/go.mod h1:vu+V0TpY+O6vW9J44gczi3Ap/oXXR10b+M/gUGO4Hls=
github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
github.com/prometheus/procfs v0.1.3/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4OA4YeYWdaU=
github.com/prometheus/procfs v0.6.0/go.mod h1:cz+aTbrPOrUb4q7XlbU9ygM+/jj0fzG6c1xBZuNvfVA=
github.com/prometheus/procfs v0.7.3 h1:4jVXhlkAyzOScmCkXBTOLRLTz8EeU+eyjrwB/EPq0VU=
github.com/prometheus/procfs v0.7.3/go.mod h1:cz+aTbrPOrUb4q7XlbU9ygM+/jj0fzG6c1xBZuNvfVA=
github.com/prometheus/prom2json v1.3.0 h1:BlqrtbT9lLH3ZsOVhXPsHzFrApCTKRifB7gjJuypu6Y=
github.com/prometheus/prom2json v1.3.0/go.mod h1:rMN7m0ApCowcoDlypBHlkNbp5eJQf/+1isKykIP5ZnM=
github.com/r3labs/diff/v2 v2.14.1 h1:wRZ3jB44Ny50DSXsoIcFQ27l2x+n5P31K/Pk+b9B0Ic=
github.com/r3labs/diff/v2 v2.14.1/go.mod h1:I8noH9Fc2fjSaMxqF3G2lhDdC0b+JXCfyx85tWFM9kc=
github.com/rivo/tview v0.0.0-20200219210816-cd38d7432498/go.mod h1:6lkG1x+13OShEf0EaOCaTQYyB7d5nSbb181KtjlS+84=
github.com/rivo/uniseg v0.1.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/rivo/uniseg v0.2.0 h1:S1pD9weZBuJdFmowNwbpi7BJ8TNftyUImj/0WQi72jY=
github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs=
github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro=
github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ=
github.com/rogpeppe/go-internal v1.1.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/rogpeppe/go-internal v1.2.2/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/rogpeppe/go-internal v1.6.1/go.mod h1:xXDCJY+GAPziupqXw64V24skbSoqbTEfhy4qGm1nDQc=
github.com/rogpeppe/go-internal v1.8.0 h1:FCbCCtXNOY3UtUuHUYaghJg4y7Fd14rXifAYUAtL9R8=
github.com/rogpeppe/go-internal v1.8.0/go.mod h1:WmiCO8CzOY8rg0OYDC4/i/2WRWAB6poM+XZ2dLUbcbE=
github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ=
github.com/rs/zerolog v1.13.0/go.mod h1:YbFCdg8HfsridGWAh22vktObvhZbQsZXe4/zB0OKkWU=
github.com/rs/zerolog v1.15.0/go.mod h1:xYTKnLHcpfU2225ny5qZjxnj9NvkumZYjJHlAThCjNc=
github.com/russross/blackfriday/v2 v2.1.0 h1:JIOH55/0cWyOuilr9/qlrm0BSXldqnqwMsf35Ld67mk=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/sanity-io/litter v1.2.0/go.mod h1:JF6pZUFgu2Q0sBZ+HSV35P8TVPI1TTzEwyu9FXAw2W4=
github.com/satori/go.uuid v1.2.0/go.mod h1:dA0hQrYB0VpLJoorglMZABFdXlWrHn1NEOzdhQKdks0=
github.com/segmentio/kafka-go v0.4.34 h1:Dm6YlLMiVSiwwav20KY0AoY63s661FXevwJ3CVHUERo=
github.com/segmentio/kafka-go v0.4.34/go.mod h1:GAjxBQJdQMB5zfNA21AhpaqOB2Mu+w3De4ni3Gbm8y0=
github.com/sergi/go-diff v1.0.0 h1:Kpca3qRNrduNnOQeazBd0ysaKrUJiIuISHxogkT9RPQ=
github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
github.com/shopspring/decimal v0.0.0-20180709203117-cd690d0c9e24/go.mod h1:M+9NzErvs504Cn4c5DxATwIqPbtswREoFCre64PpcG4=
github.com/shopspring/decimal v1.2.0 h1:abSATXmQEYyShuxI4/vyW3tV1MrKAJzCZ/0zLUXYbsQ=
github.com/shopspring/decimal v1.2.0/go.mod h1:DKyhrW/HYNuLGql+MJL6WCR6knT2jwCFRcu2hWCYk4o=
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.4.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.4.1/go.mod h1:ni0Sbl8bgC9z8RoU9G6nDWqqs/fq4eDPysMBDgk/93Q=
github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88=
github.com/sirupsen/logrus v1.7.0/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
github.com/sirupsen/logrus v1.8.1 h1:dJKuHgqk1NNQlqoA6BTlM1Wf9DOH3NBjQyu0h9+AZZE=
github.com/sirupsen/logrus v1.8.1/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
github.com/spf13/cast v1.3.1 h1:nFm6S0SMdyzrzcmThSipiEubIDy8WEXKNZ0UOgiRpng=
github.com/spf13/cast v1.3.1/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
github.com/spf13/cobra v0.0.3/go.mod h1:1l0Ry5zgKvJasoi3XT1TypsSe7PqH0Sj9dhYf7v3XqQ=
github.com/spf13/cobra v1.4.0/go.mod h1:Wo4iy3BUC+X2Fybo0PDqwJIv3dNRiZLHQymsfxlB84g=
github.com/spf13/cobra v1.5.0 h1:X+jTBEBqF0bHN+9cSMgmfuvv2VHJ9ezmFNf9Y/XstYU=
github.com/spf13/cobra v1.5.0/go.mod h1:dWXEIy2H428czQCjInthrTRUg7yKbok+2Qi/yBIJoUM=
github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.2.0/go.mod h1:qt09Ya8vawLte6SNmTgCsAVtYtaKzEcn8ATUoHMkEqE=
github.com/stretchr/objx v0.4.0 h1:M2gUjqZET1qApGOWNSnZ49BAIMX4F/1plDv3+l31EJ4=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/testify v0.0.0-20161117074351-18a02ba4a312/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.2.1/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0 h1:pSgiaMZlXftHpm5L7V1+rVB+AZJydKsMxsQBIJw4PKk=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/texttheater/golang-levenshtein/levenshtein v0.0.0-20200805054039-cae8b0eaed6c h1:HelZ2kAFadG0La9d+4htN4HzQ68Bm2iM9qKMSMES6xg=
github.com/texttheater/golang-levenshtein/levenshtein v0.0.0-20200805054039-cae8b0eaed6c/go.mod h1:JlzghshsemAMDGZLytTFY8C1JQxQPhnatWqNwUXjggo=
github.com/tidwall/gjson v1.12.1/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
github.com/tidwall/gjson v1.13.0 h1:3TFY9yxOQShrvmjdM76K+jc66zJeT6D3/VFFYCGQf7M=
github.com/tidwall/gjson v1.13.0/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=
github.com/tidwall/match v1.1.1/go.mod h1:eRSPERbgtNPcGhD8UCthc6PmLEQXEWd3PRB5JTxsfmM=
github.com/tidwall/pretty v1.0.0/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk=
github.com/tidwall/pretty v1.2.0 h1:RWIZEg2iJ8/g6fDDYzMpobmaoGh5OLl4AXtGUGPcqCs=
github.com/tidwall/pretty v1.2.0/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
github.com/ugorji/go v1.1.7/go.mod h1:kZn38zHttfInRq0xu/PH0az30d+z6vm202qpg1oXVMw=
github.com/ugorji/go v1.2.6/go.mod h1:anCg0y61KIhDlPZmnH+so+RQbysYVyDko0IMgJv0Nn0=
github.com/ugorji/go/codec v1.1.7/go.mod h1:Ax+UKWsSmolVDwsd+7N3ZtXu+yMGCf907BLYF3GoBXY=
github.com/ugorji/go/codec v1.2.6 h1:7kbGefxLoDBuYXOms4yD7223OpNMMPNPZxXk5TvFcyQ=
github.com/ugorji/go/codec v1.2.6/go.mod h1:V6TCNZ4PHqoHGFZuSG1W8nrCzzdgA2DozYxWFFpvxTw=
github.com/vektah/gqlparser v1.1.2/go.mod h1:1ycwN7Ij5njmMkPPAOaRFY4rET2Enx7IkVv3vaXspKw=
github.com/vjeantet/grok v1.0.1 h1:2rhIR7J4gThTgcZ1m2JY4TrJZNgjn985U28kT2wQrJ4=
github.com/vjeantet/grok v1.0.1/go.mod h1:ax1aAchzC6/QMXMcyzHQGZWaW1l195+uMYIkCWPCNIo=
github.com/vmihailenco/msgpack v4.0.4+incompatible h1:dSLoQfGFAo3F6OoNhwUmLwVgaUXK79GlxNBwueZn0xI=
github.com/vmihailenco/msgpack v4.0.4+incompatible/go.mod h1:fy3FlTQTDXWkZ7Bh6AcGMlsjHatGryHQYUTf1ShIgkk=
github.com/vmihailenco/msgpack/v4 v4.3.12/go.mod h1:gborTTJjAo/GWTqqRjrLCn9pgNN+NXzzngzBKDPIqw4=
github.com/vmihailenco/tagparser v0.1.1/go.mod h1:OeAg3pn3UbLjkWt+rN9oFYB6u/cQgqMEUPoW2WPyhdI=
github.com/xdg-go/pbkdf2 v1.0.0/go.mod h1:jrpuAogTd400dnrH08LKmI/xc1MbPOebTwRqcT5RDeI=
github.com/xdg-go/scram v1.0.2/go.mod h1:1WAq6h33pAW+iRreB34OORO2Nf7qel3VV3fjBj+hCSs=
github.com/xdg-go/stringprep v1.0.2/go.mod h1:8F9zXuvzgwmyT5DUm4GUfZGDdT3W+LCvS6+da4O5kxM=
github.com/xdg/scram v0.0.0-20180814205039-7eeb5667e42c/go.mod h1:lB8K/P019DLNhemzwFU4jHLhdvlE6uDZjXFejJXr49I=
github.com/xdg/scram v1.0.5 h1:TuS0RFmt5Is5qm9Tm2SoD89OPqe4IRiFtyFY4iwWXsw=
github.com/xdg/scram v1.0.5/go.mod h1:lB8K/P019DLNhemzwFU4jHLhdvlE6uDZjXFejJXr49I=
github.com/xdg/stringprep v0.0.0-20180714160509-73f8eece6fdc/go.mod h1:Jhud4/sHMO4oL310DaZAKk9ZaJ08SJfe+sJh0HrGL1Y=
github.com/xdg/stringprep v1.0.3 h1:cmL5Enob4W83ti/ZHuZLuKD/xqJfus4fVPwE+/BDm+4=
github.com/xdg/stringprep v1.0.3/go.mod h1:Jhud4/sHMO4oL310DaZAKk9ZaJ08SJfe+sJh0HrGL1Y=
github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d/go.mod h1:rHwXgn7JulP+udvsHwJoVG1YGAP6VLg4y9I5dyZdqmA=
github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.1.32/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/zclconf/go-cty v1.10.0 h1:mp9ZXQeIcN8kAwuqorjH+Q+njbJKjLrvB2yIh4q7U+0=
github.com/zclconf/go-cty v1.10.0/go.mod h1:vVKLxnk3puL4qRAv72AO+W99LUD4da90g3uUAzyuvAk=
github.com/zenazn/goji v0.9.0/go.mod h1:7S9M489iMyHBNxwZnk9/EHS098H4/F6TATF2mIxtB1Q=
go.mongodb.org/mongo-driver v1.0.3/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qLUO4lqsUM=
go.mongodb.org/mongo-driver v1.1.1/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qLUO4lqsUM=
go.mongodb.org/mongo-driver v1.3.0/go.mod h1:MSWZXKOynuguX+JSvwP8i+58jYCXxbia8HS3gZBapIE=
go.mongodb.org/mongo-driver v1.3.4/go.mod h1:MSWZXKOynuguX+JSvwP8i+58jYCXxbia8HS3gZBapIE=
go.mongodb.org/mongo-driver v1.4.3/go.mod h1:WcMNYLx/IlOxLe6JRJiv2uXuCz6zBLndR4SoGjYphSc=
go.mongodb.org/mongo-driver v1.4.4/go.mod h1:WcMNYLx/IlOxLe6JRJiv2uXuCz6zBLndR4SoGjYphSc=
go.mongodb.org/mongo-driver v1.9.0 h1:f3aLGJvQmBl8d9S40IL+jEyBC6hfLPbJjv9t5hEM9ck=
go.mongodb.org/mongo-driver v1.9.0/go.mod h1:0sQWfOeY63QTntERDJJ/0SuKK0T1uVSgKCuAROlKEPY=
go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
go.opencensus.io v0.22.4/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
go.opentelemetry.io/proto/otlp v0.7.0/go.mod h1:PqfVotwruBrMGOCsRd/89rSnXhoiJIqeYNgFYFoEGnI=
go.uber.org/atomic v1.3.2/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
go.uber.org/atomic v1.5.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ=
go.uber.org/atomic v1.6.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ=
go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
go.uber.org/multierr v1.3.0/go.mod h1:VgVr7evmIr6uPjLBxg28wmKNXyqE9akIJ5XnfpiKl+4=
go.uber.org/multierr v1.5.0/go.mod h1:FeouvMocqHpRaaGuG9EjoKcStLC43Zu/fmqdUMPcKYU=
go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9Ejo0C68/HhF8uaILCdgjnY+goOA=
go.uber.org/zap v1.9.1/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
go.uber.org/zap v1.13.0/go.mod h1:zwrFLgMcdUuIBviXEYEH1YKNaOBnKXsx2IPda5bBwHM=
golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190320223903-b7391e95e576/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190411191339-88737f569e3a/go.mod h1:WFFai1msRO1wXaEeE5yQxYXgSfI8pQAWXbQop6sCtWE=
golang.org/x/crypto v0.0.0-20190422162423-af44ce270edf/go.mod h1:WFFai1msRO1wXaEeE5yQxYXgSfI8pQAWXbQop6sCtWE=
golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190530122614-20be4c3c3ed5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190617133340-57b3e21c3d56/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190820162420-60c769a6c586/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200414173820-0848c9571904/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20201203163018-be400aefbc4c/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
golang.org/x/crypto v0.0.0-20201216223049-8b5274cf687f/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
golang.org/x/crypto v0.0.0-20210616213533-5ff15b29337e/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d h1:sK3txAijHtOK88l68nt020reeT1ZdKLIYetKl95FzVY=
golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
golang.org/x/exp v0.0.0-20190829153037-c13cbed26979/go.mod h1:86+5VVa7VpoJ4kLfm080zCjGlMRFzhUhsZKEZO7MGek=
golang.org/x/exp v0.0.0-20191030013958-a1ab85dbe136/go.mod h1:JXzH8nQsPlswgeRAPE3MuO9GYsAcnJvJ4vnMwN/5qkY=
golang.org/x/exp v0.0.0-20191129062945-2f5052295587/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM=
golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU=
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20190409202823-959b441ac422/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20190909230951-414d861bb4ac/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f/go.mod h1:5qLYkcX4OjUUV8bRuDixDT3tpyyb+LUpUlRWLxfhWrs=
golang.org/x/lint v0.0.0-20200130185559-910be7a94367/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE=
golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o=
golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc=
golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY=
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4 h1:6zppjxzCulZykYSLyVDYbneBfbaBIQPYMevg0bEwv2s=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/net v0.0.0-20180530234432-1e491301e022/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181005035420-146acd28ed58/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190320064053-1272bf9dcd53/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190628185345-da137c7871d7/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190724013045-ca1201d0de80/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190827160401-ba9fcec4b297/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200222125558-5a598a2470a0/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200301022130-244492dfa37a/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200501053045-e0ff5e5a1de5/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200506145744-7e3656a0809f/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200513185701-a91f0712d120/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200520182314-0ba52f642ac2/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200602114024-627f9648deb9/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201202161906-c7110b5ffcbb/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM=
golang.org/x/net v0.0.0-20210525063256-abc453219eb5/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211209124913-491a49abca63/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20220706163947-c90051bbdb60 h1:8NSylCMxLW4JvserAndSgFL7aPli6A68yf0bYFTcWCM=
golang.org/x/net v0.0.0-20220706163947-c90051bbdb60/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20210514164344-f6687ab2804c/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190412183630-56d357773e84/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4 h1:uVc8UZUe6tr40fFVnUP5Oj+veunVezqYl9z7DYw9xzw=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190321052220-f7bb7a8bee54/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190403152447-81d4e9dc473e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190419153524-e8e3143a4f4a/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190530182044-ad28b68e88f1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190531175056-4c3a928424d2/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190616124812-15dcb6c0061f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190626150813-e07cf5db2756/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190813064441-fde4db37ae7a/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191001151750-bb3f8db39f24/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191008105621-543471e840be/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191220142924-d4481acd189f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191224085550-c709ea063b76/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200106162015-b016eb3dc98e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200113162924-86b910548bc1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200122134326-e047566fdf82/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200212091648-12a6c2dcc1e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200331124033-c3d80250170d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200501052902-10377860bb8e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200511232937-7e40ca221e25/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200515095857-1151b9dac4a9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200523222454-059865788121/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200615200032-f1bc736245b1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200803210538-64077c9b5642/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200831180312-196b9ba8737a/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210426230700-d19ff857e887/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210601080250-7ecdf8ef093b/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210806184541-e5e7981a1069/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f h1:v4INt8xihDGvnrfjMDVXGxw9wrfxYyCjk0KbXjhR55s=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.0.0-20220526004731-065cf7ba2467 h1:CBpWXWQpIRjzmkkA+M7q9Fqnwd2mZr3AFqexg8YTfoM=
golang.org/x/term v0.0.0-20220526004731-065cf7ba2467/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20210723032227-1f47c861a9ac h1:7zkz7BUtwNFFqcowJ+RIgu2MaV/MapERkDIy+mwPyjs=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190125232054-d66bd3c5d5a6/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190329151228-23e29df326fe/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190416151739-9c9e1878f421/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190420181800-aa740d480789/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190425163242-31fd60d6bfdc/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190506145303-2d16b83fe98c/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190531172133-b3315ee88b7d/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190606124116-d0a3d012864b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190614205625-5aca471b1d59/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190617190820-da514acc4774/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190624222133-a101b041ded4/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190628153133-6cdbf07be9d0/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20190823170909-c4a336ef6a2f/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20190911174233-4f2ddba30aff/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191012152004-8de300cfc20a/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191029041327-9cc4af7d6b2c/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191029190741-b9c20aec41a5/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191113191852-77e3bb0ad9e7/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191115202509-3a792d9c32b2/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191125144606-a911d9008d1f/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191130070609-6e064ea0cf2d/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191216173652-a0e659d51361/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200103221440-774c71fcf114/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200117161641-43d50277825c/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200122220014-bf1340f18c4a/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200204074204-1cc6d1ef6c74/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200207183749-b753a1ba74fa/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200212150539-ea181f53ac56/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200224181240-023911ca70b2/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200227222343-706bc42d1f0d/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200304193943-95d2e580d8eb/go.mod h1:o4KQGtdN14AW+yjsvvwRTJJuXz8XRtIHtEnmAXLyFUw=
golang.org/x/tools v0.0.0-20200312045724-11d5b4c81c7d/go.mod h1:o4KQGtdN14AW+yjsvvwRTJJuXz8XRtIHtEnmAXLyFUw=
golang.org/x/tools v0.0.0-20200331025713-a30bf2db82d4/go.mod h1:Sl4aGygMT6LrqrWclx+PTx3U+LnKx/seiNR+3G19Ar8=
golang.org/x/tools v0.0.0-20200501065659-ab2804fb9c9d/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200512131952-2bc93b1c0c88/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200515010526-7d3b6ebf133d/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200618134242-20370b0cb4b2/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200729194436-6467de6f59a7/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
golang.org/x/tools v0.0.0-20200804011535-6c149bb5ef0d/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
golang.org/x/tools v0.0.0-20200825202427-b303f430e36d/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/xerrors v0.0.0-20190410155217-1f06c39b4373/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20190513163551-3ee3066db522/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE=
google.golang.org/api v0.7.0/go.mod h1:WtwebWUNSVBH/HAw79HIFXZNqEvBhG+Ra+ax0hx3E3M=
google.golang.org/api v0.8.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=
google.golang.org/api v0.9.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=
google.golang.org/api v0.13.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/api v0.14.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/api v0.15.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/api v0.17.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/api v0.18.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/api v0.19.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/api v0.20.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/api v0.22.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/api v0.24.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0MncE=
google.golang.org/api v0.28.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0MncE=
google.golang.org/api v0.29.0/go.mod h1:Lcubydp8VUV7KeIHD9z2Bys/sm/vGKnG1UHuDBSrHWM=
google.golang.org/api v0.30.0/go.mod h1:QGmEvQ87FHZNiUVJkT14jQNYJ4ZJjdRF23ZXz5138Fc=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww/cMBSeb0=
google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
google.golang.org/appengine v1.6.6/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
google.golang.org/appengine v1.6.7 h1:FZR1q0exgwxzPzp/aF+VccGrSfxfPpkBqjIIEq3ru6c=
google.golang.org/appengine v1.6.7/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
google.golang.org/genproto v0.0.0-20170818010345-ee236bd376b0/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190502173448-54afdca5d873/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190801165951-fa694d86fc64/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBrqXvoKpeg0TB2l7cyZUmoaFKYIwrEpbDKLA8=
google.golang.org/genproto v0.0.0-20191108220845-16a3f7862a1a/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20191115194625-c23dd37a84c9/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20191216164720-4f79533eabd1/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20191230161307-f3c370f40bfb/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200115191322-ca5a22157cba/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200122232147-0452cf42e150/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200204135345-fa8e72b47b90/go.mod h1:GmwEX6Z4W5gMy59cAlVYjN9JhxgbQH6Gn+gFDQe2lzA=
google.golang.org/genproto v0.0.0-20200212174721-66ed5ce911ce/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200224152610-e50cd9704f63/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200228133532-8c2c7df3a383/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200305110556-506484158171/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200312145019-da6875a35672/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200331122359-1ee6d9798940/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200430143042-b979b6f78d84/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200511104702-f5ebc3bea380/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200513103714-09dca8ec2884/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200515170657-fc4c6c6a6587/go.mod h1:YsZOwe1myG/8QRHRsmBRE1LrgQY60beZKjly0O1fX9U=
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
google.golang.org/genproto v0.0.0-20200618031413-b414f8b61790/go.mod h1:jDfRM7FcilCzHH/e9qn6dsT145K34l5v+OpcnNgKAAA=
google.golang.org/genproto v0.0.0-20200729003335-053ba62fc06f/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20200804131852-c06518451d9c/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20200825200019-8632dd797987/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20220414192740-2d67ff6cf2b4 h1:myaecH64R0bIEDjNORIel4iXubqzaHU1K2z8ajBwWcM=
google.golang.org/genproto v0.0.0-20220414192740-2d67ff6cf2b4/go.mod h1:8w6bsBMX6yCPbAVTeqQHvzxW0EIFigd5lZyahWgyfDo=
google.golang.org/grpc v1.8.0/go.mod h1:yo6s7OP7yaDglbqo1J04qKzAhqBH6lvTonzMVmEdcZw=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
google.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.27.1/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.28.0/go.mod h1:rpkK4SK4GF4Ach/+MFLZUBavHOvF2JJB5uozKKal+60=
google.golang.org/grpc v1.29.1/go.mod h1:itym6AZVZYACWQqET3MqgPpjcuV5QH3BxFS3IjizoKk=
google.golang.org/grpc v1.30.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.31.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.33.1/go.mod h1:fr5YgcSWrqhRRxogOsw7RzIpsmvOZ6IcH4kBYTpR3n0=
google.golang.org/grpc v1.36.0/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU=
google.golang.org/grpc v1.45.0 h1:NEpgUqV3Z+ZjkqMsxMg11IaDrXY4RY6CQukSGK0uI1M=
google.golang.org/grpc v1.45.0/go.mod h1:lN7owxKUQEqMfSyQikvvk5tf/6zMPsrK+ONuO11+0rQ=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE=
google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo=
google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGjtUeSXeh4=
google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c=
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
google.golang.org/protobuf v1.28.0 h1:w43yiav+6bVFTBQFZX0r7ipe9JQ1QsbMgHwbBziscLw=
google.golang.org/protobuf v1.28.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
gopkg.in/inconshreveable/log15.v2 v2.0.0-20180818164646-67afb5ed74ec/go.mod h1:aPpfJ7XW+gOuirDoZ8gHhLh3kZ1B08FtV2bbmy7Jv3s=
gopkg.in/natefinch/lumberjack.v2 v2.0.0 h1:1Lc07Kr7qY4U2YPouBjpCLxpiyxIVoxqXgkXLknAOE8=
gopkg.in/natefinch/lumberjack.v2 v2.0.0/go.mod h1:l0ndWWf7gzL7RNwBG7wST/UCcT4T24xpD6X8LsfU/+k=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
gopkg.in/tomb.v2 v2.0.0-20161208151619-d5d1b5820637 h1:yiW+nvdHb9LVqSHQBXfZCieqV4fzYhNBql77zY0ykqs=
gopkg.in/tomb.v2 v2.0.0-20161208151619-d5d1b5820637/go.mod h1:BHsqpu/nsuzkT5BpiH1EMZPLyqSMM8JbIavyFACoFNk=
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.3/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.5/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.7/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.0-20200605160147-a5ece683394c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gotest.tools/v3 v3.0.2/go.mod h1:3SzNCllyD9/Y+b5r9JIKQ474KzkZyqLqEfYqMsX94Bk=
gotest.tools/v3 v3.0.3 h1:4AuOwCGf4lLR9u3YOe2awrHygurzhO/HeQ6laiA6Sx0=
gotest.tools/v3 v3.0.3/go.mod h1:Z7Lb0S5l+klDB31fvDQX8ss/FlKDxtlFlw3Oa8Ymbl8=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg=
honnef.co/go/tools v0.0.1-2020.1.3/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k=
honnef.co/go/tools v0.0.1-2020.1.4/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k=
rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=
rsc.io/quote/v3 v3.1.0/go.mod h1:yEA65RcK8LyAZtP9Kv3t0HmxON59tX3rD+tICJqUlj0=
rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA=
crowdsec-1.4.6/make_chocolatey.ps1 0000664 0000000 0000000 00000000675 14371172257 0017130 0 ustar 00root root 0000000 0000000 param (
$version
)
if ($version.StartsWith("v"))
{
$version = $version.Substring(1)
}
#Pre-releases will be like 1.4.0-rc1, remove everything after the dash as it does not conform to the MSI versioning scheme
if ($version.Contains("-"))
{
$version = $version.Substring(0, $version.IndexOf("-"))
}
Set-Location .\windows\Chocolatey\crowdsec
Copy-Item ..\..\..\crowdsec_$version.msi tools\crowdsec.msi
choco pack --version $version crowdsec-1.4.6/make_installer.ps1 0000664 0000000 0000000 00000002066 14371172257 0016767 0 ustar 00root root 0000000 0000000 param (
$version
)
$env:Path += ";C:\Program Files (x86)\WiX Toolset v3.11\bin"
if ($version.StartsWith("v"))
{
$version = $version.Substring(1)
}
#Pre-releases will be like 1.4.0-rc1, remove everything after the dash as it does not conform to the MSI versioning scheme
if ($version.Contains("-"))
{
$version = $version.Substring(0, $version.IndexOf("-"))
}
Remove-Item -Force -Recurse -Path .\msi -ErrorAction SilentlyContinue
#we only harvest the patterns dir, as we want to handle differently some yaml files in the config directory, and I really don't want to write xlst filters to exclude the files :(
heat.exe dir config\patterns -nologo -cg CrowdsecPatterns -dr PatternsDir -g1 -ag -sf -srd -scom -sreg -out "msi\fragment.wxs"
candle.exe -arch x64 -dSourceDir=config\patterns -dVersion="$version" -out msi\ msi\fragment.wxs windows\installer\WixUI_HK.wxs windows\installer\product.wxs
light.exe -b .\config\patterns -ext WixUIExtension -ext WixUtilExtension -sacl -spdb -out crowdsec_$version.msi msi\fragment.wixobj msi\WixUI_HK.wixobj msi\product.wixobj crowdsec-1.4.6/pkg/ 0000775 0000000 0000000 00000000000 14371172257 0014125 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/pkg/acquisition/ 0000775 0000000 0000000 00000000000 14371172257 0016455 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/pkg/acquisition/acquisition.go 0000664 0000000 0000000 00000022014 14371172257 0021333 0 ustar 00root root 0000000 0000000 package acquisition
import (
"fmt"
"io"
"os"
"strings"
"github.com/crowdsecurity/crowdsec/pkg/acquisition/configuration"
cloudwatchacquisition "github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/cloudwatch"
dockeracquisition "github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/docker"
fileacquisition "github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/file"
journalctlacquisition "github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/journalctl"
kafkaacquisition "github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/kafka"
kinesisacquisition "github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/kinesis"
syslogacquisition "github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/syslog"
wineventlogacquisition "github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/wineventlog"
"github.com/crowdsecurity/crowdsec/pkg/csconfig"
"github.com/crowdsecurity/crowdsec/pkg/types"
"github.com/pkg/errors"
"github.com/prometheus/client_golang/prometheus"
log "github.com/sirupsen/logrus"
"gopkg.in/yaml.v2"
tomb "gopkg.in/tomb.v2"
)
// The interface each datasource must implement
type DataSource interface {
GetMetrics() []prometheus.Collector // Returns pointers to metrics that are managed by the module
GetAggregMetrics() []prometheus.Collector // Returns pointers to metrics that are managed by the module (aggregated mode, limits cardinality)
Configure([]byte, *log.Entry) error // Configure the datasource
ConfigureByDSN(string, map[string]string, *log.Entry) error // Configure the datasource
GetMode() string // Get the mode (TAIL, CAT or SERVER)
GetName() string // Get the name of the module
OneShotAcquisition(chan types.Event, *tomb.Tomb) error // Start one shot acquisition(eg, cat a file)
StreamingAcquisition(chan types.Event, *tomb.Tomb) error // Start live acquisition (eg, tail a file)
CanRun() error // Whether the datasource can run or not (eg, journalctl on BSD is a non-sense)
Dump() interface{}
}
var AcquisitionSources = []struct {
name string
iface func() DataSource
}{
{
name: "file",
iface: func() DataSource { return &fileacquisition.FileSource{} },
},
{
name: "journalctl",
iface: func() DataSource { return &journalctlacquisition.JournalCtlSource{} },
},
{
name: "cloudwatch",
iface: func() DataSource { return &cloudwatchacquisition.CloudwatchSource{} },
},
{
name: "syslog",
iface: func() DataSource { return &syslogacquisition.SyslogSource{} },
},
{
name: "docker",
iface: func() DataSource { return &dockeracquisition.DockerSource{} },
},
{
name: "kinesis",
iface: func() DataSource { return &kinesisacquisition.KinesisSource{} },
},
{
name: "wineventlog",
iface: func() DataSource { return &wineventlogacquisition.WinEventLogSource{} },
},
{
name: "kafka",
iface: func() DataSource { return &kafkaacquisition.KafkaSource{} },
},
}
func GetDataSourceIface(dataSourceType string) DataSource {
for _, source := range AcquisitionSources {
if source.name == dataSourceType {
return source.iface()
}
}
return nil
}
func DataSourceConfigure(commonConfig configuration.DataSourceCommonCfg) (*DataSource, error) {
//we dump it back to []byte, because we want to decode the yaml blob twice :
//once to DataSourceCommonCfg, and then later to the dedicated type of the datasource
yamlConfig, err := yaml.Marshal(commonConfig)
if err != nil {
return nil, errors.Wrap(err, "unable to marshal back interface")
}
if dataSrc := GetDataSourceIface(commonConfig.Source); dataSrc != nil {
/* this logger will then be used by the datasource at runtime */
clog := log.New()
if err := types.ConfigureLogger(clog); err != nil {
return nil, errors.Wrap(err, "while configuring datasource logger")
}
if commonConfig.LogLevel != nil {
clog.SetLevel(*commonConfig.LogLevel)
}
customLog := log.Fields{
"type": commonConfig.Source,
}
if commonConfig.Name != "" {
customLog["name"] = commonConfig.Name
}
subLogger := clog.WithFields(customLog)
/* check eventual dependencies are satisfied (ie. journald will check journalctl availability) */
if err := dataSrc.CanRun(); err != nil {
return nil, errors.Wrapf(err, "datasource %s cannot be run", commonConfig.Source)
}
/* configure the actual datasource */
if err := dataSrc.Configure(yamlConfig, subLogger); err != nil {
return nil, errors.Wrapf(err, "failed to configure datasource %s", commonConfig.Source)
}
return &dataSrc, nil
}
return nil, fmt.Errorf("cannot find source %s", commonConfig.Source)
}
//detectBackwardCompatAcquis : try to magically detect the type for backward compat (type was not mandatory then)
func detectBackwardCompatAcquis(sub configuration.DataSourceCommonCfg) string {
if _, ok := sub.Config["filename"]; ok {
return "file"
}
if _, ok := sub.Config["filenames"]; ok {
return "file"
}
if _, ok := sub.Config["journalctl_filter"]; ok {
return "journalctl"
}
return ""
}
func LoadAcquisitionFromDSN(dsn string, labels map[string]string) ([]DataSource, error) {
var sources []DataSource
frags := strings.Split(dsn, ":")
if len(frags) == 1 {
return nil, fmt.Errorf("%s isn't valid dsn (no protocol)", dsn)
}
dataSrc := GetDataSourceIface(frags[0])
if dataSrc == nil {
return nil, fmt.Errorf("no acquisition for protocol %s://", frags[0])
}
/* this logger will then be used by the datasource at runtime */
clog := log.New()
if err := types.ConfigureLogger(clog); err != nil {
return nil, errors.Wrap(err, "while configuring datasource logger")
}
subLogger := clog.WithFields(log.Fields{
"type": dsn,
})
err := dataSrc.ConfigureByDSN(dsn, labels, subLogger)
if err != nil {
return nil, errors.Wrapf(err, "while configuration datasource for %s", dsn)
}
sources = append(sources, dataSrc)
return sources, nil
}
// LoadAcquisitionFromFile unmarshals the configuration item and checks its availability
func LoadAcquisitionFromFile(config *csconfig.CrowdsecServiceCfg) ([]DataSource, error) {
var sources []DataSource
for _, acquisFile := range config.AcquisitionFiles {
log.Infof("loading acquisition file : %s", acquisFile)
yamlFile, err := os.Open(acquisFile)
if err != nil {
return nil, err
}
dec := yaml.NewDecoder(yamlFile)
dec.SetStrict(true)
for {
var sub configuration.DataSourceCommonCfg
var idx int
err = dec.Decode(&sub)
if err != nil {
if ! errors.Is(err, io.EOF) {
return nil, errors.Wrapf(err, "failed to yaml decode %s", acquisFile)
}
log.Tracef("End of yaml file")
break
}
//for backward compat ('type' was not mandatory, detect it)
if guessType := detectBackwardCompatAcquis(sub); guessType != "" {
sub.Source = guessType
}
//it's an empty item, skip it
if len(sub.Labels) == 0 {
if sub.Source == "" {
log.Debugf("skipping empty item in %s", acquisFile)
idx += 1
continue
}
return nil, fmt.Errorf("missing labels in %s (position: %d)", acquisFile, idx)
}
if sub.Source == "" {
return nil, fmt.Errorf("data source type is empty ('source') in %s (position: %d)", acquisFile, idx)
}
if GetDataSourceIface(sub.Source) == nil {
return nil, fmt.Errorf("unknown data source %s in %s (position: %d)", sub.Source, acquisFile, idx)
}
src, err := DataSourceConfigure(sub)
if err != nil {
return nil, errors.Wrapf(err, "while configuring datasource of type %s from %s (position: %d)", sub.Source, acquisFile, idx)
}
sources = append(sources, *src)
idx += 1
}
}
return sources, nil
}
func GetMetrics(sources []DataSource, aggregated bool) error {
var metrics []prometheus.Collector
for i := 0; i < len(sources); i++ {
if aggregated {
metrics = sources[i].GetMetrics()
} else {
metrics = sources[i].GetAggregMetrics()
}
for _, metric := range metrics {
if err := prometheus.Register(metric); err != nil {
if _, ok := err.(prometheus.AlreadyRegisteredError); !ok {
return errors.Wrapf(err, "could not register metrics for datasource %s", sources[i].GetName())
}
//ignore the error
}
}
}
return nil
}
func StartAcquisition(sources []DataSource, output chan types.Event, AcquisTomb *tomb.Tomb) error {
for i := 0; i < len(sources); i++ {
subsrc := sources[i] //ensure its a copy
log.Debugf("starting one source %d/%d ->> %T", i, len(sources), subsrc)
AcquisTomb.Go(func() error {
defer types.CatchPanic("crowdsec/acquis")
var err error
if subsrc.GetMode() == configuration.TAIL_MODE {
err = subsrc.StreamingAcquisition(output, AcquisTomb)
} else {
err = subsrc.OneShotAcquisition(output, AcquisTomb)
}
if err != nil {
//if one of the acqusition returns an error, we kill the others to properly shutdown
AcquisTomb.Kill(err)
}
return nil
})
}
// Don't wait if we have no sources, as it will hang forever
if len(sources) > 0 {
/*return only when acquisition is over (cat) or never (tail)*/
err := AcquisTomb.Wait()
return err
}
return nil
}
crowdsec-1.4.6/pkg/acquisition/acquisition_test.go 0000664 0000000 0000000 00000034722 14371172257 0022403 0 ustar 00root root 0000000 0000000 package acquisition
import (
"fmt"
"strings"
"testing"
"time"
"github.com/pkg/errors"
"github.com/prometheus/client_golang/prometheus"
log "github.com/sirupsen/logrus"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
tomb "gopkg.in/tomb.v2"
"gopkg.in/yaml.v2"
"github.com/crowdsecurity/crowdsec/pkg/acquisition/configuration"
"github.com/crowdsecurity/crowdsec/pkg/csconfig"
"github.com/crowdsecurity/crowdsec/pkg/cstest"
"github.com/crowdsecurity/crowdsec/pkg/types"
)
type MockSource struct {
configuration.DataSourceCommonCfg `yaml:",inline"`
Toto string `yaml:"toto"`
logger *log.Entry
}
func (f *MockSource) Configure(cfg []byte, logger *log.Entry) error {
f.logger = logger
if err := yaml.UnmarshalStrict(cfg, &f); err != nil {
return errors.Wrap(err, "while unmarshaling to reader specific config")
}
if f.Mode == "" {
f.Mode = configuration.CAT_MODE
}
if f.Mode != configuration.CAT_MODE && f.Mode != configuration.TAIL_MODE {
return fmt.Errorf("mode %s is not supported", f.Mode)
}
if f.Toto == "" {
return fmt.Errorf("expect non-empty toto")
}
return nil
}
func (f *MockSource) GetMode() string { return f.Mode }
func (f *MockSource) OneShotAcquisition(chan types.Event, *tomb.Tomb) error { return nil }
func (f *MockSource) StreamingAcquisition(chan types.Event, *tomb.Tomb) error { return nil }
func (f *MockSource) CanRun() error { return nil }
func (f *MockSource) GetMetrics() []prometheus.Collector { return nil }
func (f *MockSource) GetAggregMetrics() []prometheus.Collector { return nil }
func (f *MockSource) Dump() interface{} { return f }
func (f *MockSource) GetName() string { return "mock" }
func (f *MockSource) ConfigureByDSN(string, map[string]string, *log.Entry) error {
return fmt.Errorf("not supported")
}
// copy the mocksource, but this one can't run
type MockSourceCantRun struct {
MockSource
}
func (f *MockSourceCantRun) CanRun() error { return fmt.Errorf("can't run bro") }
func (f *MockSourceCantRun) GetName() string { return "mock_cant_run" }
// appendMockSource is only used to add mock source for tests
func appendMockSource() {
if GetDataSourceIface("mock") == nil {
mock := struct {
name string
iface func() DataSource
}{
name: "mock",
iface: func() DataSource { return &MockSource{} },
}
AcquisitionSources = append(AcquisitionSources, mock)
}
if GetDataSourceIface("mock_cant_run") == nil {
mock := struct {
name string
iface func() DataSource
}{
name: "mock_cant_run",
iface: func() DataSource { return &MockSourceCantRun{} },
}
AcquisitionSources = append(AcquisitionSources, mock)
}
}
func TestDataSourceConfigure(t *testing.T) {
appendMockSource()
tests := []struct {
TestName string
String string
ExpectedError string
}{
{
TestName: "basic_valid_config",
String: `
mode: cat
labels:
test: foobar
log_level: info
source: mock
toto: test_value1
`,
},
{
TestName: "basic_debug_config",
String: `
mode: cat
labels:
test: foobar
log_level: debug
source: mock
toto: test_value1
`,
},
{
TestName: "basic_tailmode_config",
String: `
mode: tail
labels:
test: foobar
log_level: debug
source: mock
toto: test_value1
`,
},
{
TestName: "bad_mode_config",
String: `
mode: ratata
labels:
test: foobar
log_level: debug
source: mock
toto: test_value1
`,
ExpectedError: "failed to configure datasource mock: mode ratata is not supported",
},
{
TestName: "bad_type_config",
String: `
mode: cat
labels:
test: foobar
log_level: debug
source: tutu
`,
ExpectedError: "cannot find source tutu",
},
{
TestName: "mismatch_config",
String: `
mode: cat
labels:
test: foobar
log_level: debug
source: mock
wowo: ajsajasjas
`,
ExpectedError: "field wowo not found in type acquisition.MockSource",
},
{
TestName: "cant_run_error",
String: `
mode: cat
labels:
test: foobar
log_level: debug
source: mock_cant_run
wowo: ajsajasjas
`,
ExpectedError: "datasource mock_cant_run cannot be run: can't run bro",
},
}
for _, tc := range tests {
tc := tc
t.Run(tc.TestName, func(t *testing.T) {
common := configuration.DataSourceCommonCfg{}
yaml.Unmarshal([]byte(tc.String), &common)
ds, err := DataSourceConfigure(common)
cstest.RequireErrorContains(t, err, tc.ExpectedError)
if tc.ExpectedError == "" {
return
}
switch tc.TestName {
case "basic_valid_config":
mock := (*ds).Dump().(*MockSource)
assert.Equal(t, mock.Toto, "test_value1")
assert.Equal(t, mock.Mode, "cat")
assert.Equal(t, mock.logger.Logger.Level, log.InfoLevel)
assert.Equal(t, mock.Labels, map[string]string{"test": "foobar"})
case "basic_debug_config":
mock := (*ds).Dump().(*MockSource)
assert.Equal(t, mock.Toto, "test_value1")
assert.Equal(t, mock.Mode, "cat")
assert.Equal(t, mock.logger.Logger.Level, log.DebugLevel)
assert.Equal(t, mock.Labels, map[string]string{"test": "foobar"})
case "basic_tailmode_config":
mock := (*ds).Dump().(*MockSource)
assert.Equal(t, mock.Toto, "test_value1")
assert.Equal(t, mock.Mode, "tail")
assert.Equal(t, mock.logger.Logger.Level, log.DebugLevel)
assert.Equal(t, mock.Labels, map[string]string{"test": "foobar"})
}
})
}
}
func TestLoadAcquisitionFromFile(t *testing.T) {
appendMockSource()
tests := []struct {
TestName string
Config csconfig.CrowdsecServiceCfg
ExpectedError string
ExpectedLen int
}{
{
TestName: "non_existent_file",
Config: csconfig.CrowdsecServiceCfg{
AcquisitionFiles: []string{"does_not_exist"},
},
ExpectedError: "open does_not_exist: " + cstest.FileNotFoundMessage,
ExpectedLen: 0,
},
{
TestName: "invalid_yaml_file",
Config: csconfig.CrowdsecServiceCfg{
AcquisitionFiles: []string{"test_files/badyaml.yaml"},
},
ExpectedError: "failed to yaml decode test_files/badyaml.yaml: yaml: unmarshal errors",
ExpectedLen: 0,
},
{
TestName: "invalid_empty_yaml",
Config: csconfig.CrowdsecServiceCfg{
AcquisitionFiles: []string{"test_files/emptyitem.yaml"},
},
ExpectedLen: 0,
},
{
TestName: "basic_valid",
Config: csconfig.CrowdsecServiceCfg{
AcquisitionFiles: []string{"test_files/basic_filemode.yaml"},
},
ExpectedLen: 2,
},
{
TestName: "missing_labels",
Config: csconfig.CrowdsecServiceCfg{
AcquisitionFiles: []string{"test_files/missing_labels.yaml"},
},
ExpectedError: "missing labels in test_files/missing_labels.yaml",
},
{
TestName: "backward_compat",
Config: csconfig.CrowdsecServiceCfg{
AcquisitionFiles: []string{"test_files/backward_compat.yaml"},
},
ExpectedLen: 2,
},
{
TestName: "bad_type",
Config: csconfig.CrowdsecServiceCfg{
AcquisitionFiles: []string{"test_files/bad_source.yaml"},
},
ExpectedError: "unknown data source does_not_exist in test_files/bad_source.yaml",
},
{
TestName: "invalid_filetype_config",
Config: csconfig.CrowdsecServiceCfg{
AcquisitionFiles: []string{"test_files/bad_filetype.yaml"},
},
ExpectedError: "while configuring datasource of type file from test_files/bad_filetype.yaml",
},
}
for _, tc := range tests {
tc := tc
t.Run(tc.TestName, func(t *testing.T) {
dss, err := LoadAcquisitionFromFile(&tc.Config)
cstest.RequireErrorContains(t, err, tc.ExpectedError)
if tc.ExpectedError != "" {
return
}
assert.Len(t, dss, tc.ExpectedLen)
})
}
}
/*
test start acquisition :
- create mock parser in cat mode : start acquisition, check it returns, count items in chan
- create mock parser in tail mode : start acquisition, sleep, check item count, tomb kill it, wait for it to return
*/
type MockCat struct {
configuration.DataSourceCommonCfg `yaml:",inline"`
logger *log.Entry
}
func (f *MockCat) Configure(cfg []byte, logger *log.Entry) error {
f.logger = logger
if f.Mode == "" {
f.Mode = configuration.CAT_MODE
}
if f.Mode != configuration.CAT_MODE {
return fmt.Errorf("mode %s is not supported", f.Mode)
}
return nil
}
func (f *MockCat) GetName() string { return "mock_cat" }
func (f *MockCat) GetMode() string { return "cat" }
func (f *MockCat) OneShotAcquisition(out chan types.Event, tomb *tomb.Tomb) error {
for i := 0; i < 10; i++ {
evt := types.Event{}
evt.Line.Src = "test"
out <- evt
}
return nil
}
func (f *MockCat) StreamingAcquisition(chan types.Event, *tomb.Tomb) error {
return fmt.Errorf("can't run in tail")
}
func (f *MockCat) CanRun() error { return nil }
func (f *MockCat) GetMetrics() []prometheus.Collector { return nil }
func (f *MockCat) GetAggregMetrics() []prometheus.Collector { return nil }
func (f *MockCat) Dump() interface{} { return f }
func (f *MockCat) ConfigureByDSN(string, map[string]string, *log.Entry) error {
return fmt.Errorf("not supported")
}
//----
type MockTail struct {
configuration.DataSourceCommonCfg `yaml:",inline"`
logger *log.Entry
}
func (f *MockTail) Configure(cfg []byte, logger *log.Entry) error {
f.logger = logger
if f.Mode == "" {
f.Mode = configuration.TAIL_MODE
}
if f.Mode != configuration.TAIL_MODE {
return fmt.Errorf("mode %s is not supported", f.Mode)
}
return nil
}
func (f *MockTail) GetName() string { return "mock_tail" }
func (f *MockTail) GetMode() string { return "tail" }
func (f *MockTail) OneShotAcquisition(out chan types.Event, tomb *tomb.Tomb) error {
return fmt.Errorf("can't run in cat mode")
}
func (f *MockTail) StreamingAcquisition(out chan types.Event, t *tomb.Tomb) error {
for i := 0; i < 10; i++ {
evt := types.Event{}
evt.Line.Src = "test"
out <- evt
}
<-t.Dying()
return nil
}
func (f *MockTail) CanRun() error { return nil }
func (f *MockTail) GetMetrics() []prometheus.Collector { return nil }
func (f *MockTail) GetAggregMetrics() []prometheus.Collector { return nil }
func (f *MockTail) Dump() interface{} { return f }
func (f *MockTail) ConfigureByDSN(string, map[string]string, *log.Entry) error {
return fmt.Errorf("not supported")
}
//func StartAcquisition(sources []DataSource, output chan types.Event, AcquisTomb *tomb.Tomb) error {
func TestStartAcquisitionCat(t *testing.T) {
sources := []DataSource{
&MockCat{},
}
out := make(chan types.Event)
acquisTomb := tomb.Tomb{}
go func() {
if err := StartAcquisition(sources, out, &acquisTomb); err != nil {
t.Errorf("unexpected error")
}
}()
count := 0
READLOOP:
for {
select {
case <-out:
count++
case <-time.After(1 * time.Second):
break READLOOP
}
}
assert.Equal(t, 10, count)
}
func TestStartAcquisitionTail(t *testing.T) {
sources := []DataSource{
&MockTail{},
}
out := make(chan types.Event)
acquisTomb := tomb.Tomb{}
go func() {
if err := StartAcquisition(sources, out, &acquisTomb); err != nil {
t.Errorf("unexpected error")
}
}()
count := 0
READLOOP:
for {
select {
case <-out:
count++
case <-time.After(1 * time.Second):
break READLOOP
}
}
assert.Equal(t, 10, count)
acquisTomb.Kill(nil)
time.Sleep(1 * time.Second)
require.NoError(t, acquisTomb.Err(), "tomb is not dead")
}
type MockTailError struct {
MockTail
}
func (f *MockTailError) StreamingAcquisition(out chan types.Event, t *tomb.Tomb) error {
for i := 0; i < 10; i++ {
evt := types.Event{}
evt.Line.Src = "test"
out <- evt
}
t.Kill(fmt.Errorf("got error (tomb)"))
return fmt.Errorf("got error")
}
func TestStartAcquisitionTailError(t *testing.T) {
sources := []DataSource{
&MockTailError{},
}
out := make(chan types.Event)
acquisTomb := tomb.Tomb{}
go func() {
if err := StartAcquisition(sources, out, &acquisTomb); err != nil && err.Error() != "got error (tomb)" {
t.Errorf("expected error, got '%s'", err)
}
}()
count := 0
READLOOP:
for {
select {
case <-out:
count++
case <-time.After(1 * time.Second):
break READLOOP
}
}
assert.Equal(t, 10, count)
//acquisTomb.Kill(nil)
time.Sleep(1 * time.Second)
cstest.RequireErrorContains(t, acquisTomb.Err(), "got error (tomb)")
}
type MockSourceByDSN struct {
configuration.DataSourceCommonCfg `yaml:",inline"`
Toto string `yaml:"toto"`
logger *log.Entry //nolint: unused
}
func (f *MockSourceByDSN) Configure(cfg []byte, logger *log.Entry) error { return nil }
func (f *MockSourceByDSN) GetMode() string { return f.Mode }
func (f *MockSourceByDSN) OneShotAcquisition(chan types.Event, *tomb.Tomb) error { return nil }
func (f *MockSourceByDSN) StreamingAcquisition(chan types.Event, *tomb.Tomb) error { return nil }
func (f *MockSourceByDSN) CanRun() error { return nil }
func (f *MockSourceByDSN) GetMetrics() []prometheus.Collector { return nil }
func (f *MockSourceByDSN) GetAggregMetrics() []prometheus.Collector { return nil }
func (f *MockSourceByDSN) Dump() interface{} { return f }
func (f *MockSourceByDSN) GetName() string { return "mockdsn" }
func (f *MockSourceByDSN) ConfigureByDSN(dsn string, labels map[string]string, logger *log.Entry) error {
dsn = strings.TrimPrefix(dsn, "mockdsn://")
if dsn != "test_expect" {
return fmt.Errorf("unexpected value")
}
return nil
}
func TestConfigureByDSN(t *testing.T) {
tests := []struct {
dsn string
ExpectedError string
ExpectedResLen int
}{
{
dsn: "baddsn",
ExpectedError: "baddsn isn't valid dsn (no protocol)",
},
{
dsn: "foobar://toto",
ExpectedError: "no acquisition for protocol foobar://",
},
{
dsn: "mockdsn://test_expect",
ExpectedResLen: 1,
},
{
dsn: "mockdsn://bad",
ExpectedError: "unexpected value",
},
}
if GetDataSourceIface("mockdsn") == nil {
mock := struct {
name string
iface func() DataSource
}{
name: "mockdsn",
iface: func() DataSource { return &MockSourceByDSN{} },
}
AcquisitionSources = append(AcquisitionSources, mock)
}
for _, tc := range tests {
tc := tc
t.Run(tc.dsn, func(t *testing.T) {
srcs, err := LoadAcquisitionFromDSN(tc.dsn, map[string]string{"type": "test_label"})
cstest.RequireErrorContains(t, err, tc.ExpectedError)
assert.Len(t, srcs, tc.ExpectedResLen)
})
}
}
crowdsec-1.4.6/pkg/acquisition/configuration/ 0000775 0000000 0000000 00000000000 14371172257 0021324 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/pkg/acquisition/configuration/configuration.go 0000664 0000000 0000000 00000001347 14371172257 0024527 0 ustar 00root root 0000000 0000000 package configuration
import (
log "github.com/sirupsen/logrus"
)
type DataSourceCommonCfg struct {
Mode string `yaml:"mode,omitempty"`
Labels map[string]string `yaml:"labels,omitempty"`
LogLevel *log.Level `yaml:"log_level,omitempty"`
Source string `yaml:"source,omitempty"`
Name string `yaml:"name,omitempty"`
UseTimeMachine bool `yaml:"use_time_machine,omitempty"`
Config map[string]interface{} `yaml:",inline"` //to keep the datasource-specific configuration directives
}
var TAIL_MODE = "tail"
var CAT_MODE = "cat"
var SERVER_MODE = "server" // No difference with tail, just a bit more verbose
crowdsec-1.4.6/pkg/acquisition/modules/ 0000775 0000000 0000000 00000000000 14371172257 0020125 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/pkg/acquisition/modules/cloudwatch/ 0000775 0000000 0000000 00000000000 14371172257 0022262 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/pkg/acquisition/modules/cloudwatch/cloudwatch.go 0000664 0000000 0000000 00000061723 14371172257 0024757 0 ustar 00root root 0000000 0000000 package cloudwatchacquisition
import (
"context"
"fmt"
"net/url"
"os"
"regexp"
"strings"
"sync"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/crowdsecurity/crowdsec/pkg/acquisition/configuration"
leaky "github.com/crowdsecurity/crowdsec/pkg/leakybucket"
"github.com/crowdsecurity/crowdsec/pkg/parser"
"github.com/crowdsecurity/crowdsec/pkg/types"
"github.com/pkg/errors"
"github.com/prometheus/client_golang/prometheus"
log "github.com/sirupsen/logrus"
"gopkg.in/tomb.v2"
"gopkg.in/yaml.v2"
"github.com/aws/aws-sdk-go/service/cloudwatchlogs"
)
var openedStreams = prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Name: "cs_cloudwatch_openstreams_total",
Help: "Number of opened stream within group.",
},
[]string{"group"},
)
var streamIndexMutex = sync.Mutex{}
var linesRead = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "cs_cloudwatch_stream_hits_total",
Help: "Number of event read from stream.",
},
[]string{"group", "stream"},
)
//CloudwatchSource is the runtime instance keeping track of N streams within 1 cloudwatch group
type CloudwatchSource struct {
Config CloudwatchSourceConfiguration
/*runtime stuff*/
logger *log.Entry
t *tomb.Tomb
cwClient *cloudwatchlogs.CloudWatchLogs
monitoredStreams []*LogStreamTailConfig
streamIndexes map[string]string
}
//CloudwatchSourceConfiguration allows user to define one or more streams to monitor within a cloudwatch log group
type CloudwatchSourceConfiguration struct {
configuration.DataSourceCommonCfg `yaml:",inline"`
GroupName string `yaml:"group_name"` //the group name to be monitored
StreamRegexp *string `yaml:"stream_regexp,omitempty"` //allow to filter specific streams
StreamName *string `yaml:"stream_name,omitempty"`
StartTime, EndTime *time.Time `yaml:"-"`
DescribeLogStreamsLimit *int64 `yaml:"describelogstreams_limit,omitempty"` //batch size for DescribeLogStreamsPagesWithContext
GetLogEventsPagesLimit *int64 `yaml:"getlogeventspages_limit,omitempty"`
PollNewStreamInterval *time.Duration `yaml:"poll_new_stream_interval,omitempty"` //frequency at which we poll for new streams within the log group
MaxStreamAge *time.Duration `yaml:"max_stream_age,omitempty"` //monitor only streams that have been updated within $duration
PollStreamInterval *time.Duration `yaml:"poll_stream_interval,omitempty"` //frequency at which we poll each stream
StreamReadTimeout *time.Duration `yaml:"stream_read_timeout,omitempty"` //stop monitoring streams that haven't been updated within $duration, might be reopened later tho
AwsApiCallTimeout *time.Duration `yaml:"aws_api_timeout,omitempty"`
AwsProfile *string `yaml:"aws_profile,omitempty"`
PrependCloudwatchTimestamp *bool `yaml:"prepend_cloudwatch_timestamp,omitempty"`
AwsConfigDir *string `yaml:"aws_config_dir,omitempty"`
AwsRegion *string `yaml:"aws_region,omitempty"`
}
//LogStreamTailConfig is the configuration for one given stream within one group
type LogStreamTailConfig struct {
GroupName string
StreamName string
GetLogEventsPagesLimit int64
PollStreamInterval time.Duration
StreamReadTimeout time.Duration
PrependCloudwatchTimestamp *bool
Labels map[string]string
logger *log.Entry
ExpectMode int
t tomb.Tomb
StartTime, EndTime time.Time //only used for CatMode
}
var (
def_DescribeLogStreamsLimit = int64(50)
def_PollNewStreamInterval = 10 * time.Second
def_MaxStreamAge = 5 * time.Minute
def_PollStreamInterval = 10 * time.Second
def_AwsApiCallTimeout = 10 * time.Second
def_StreamReadTimeout = 10 * time.Minute
def_PollDeadStreamInterval = 10 * time.Second
def_GetLogEventsPagesLimit = int64(1000)
def_AwsConfigDir = ""
)
func (cw *CloudwatchSource) Configure(cfg []byte, logger *log.Entry) error {
cwConfig := CloudwatchSourceConfiguration{}
targetStream := "*"
if err := yaml.UnmarshalStrict(cfg, &cwConfig); err != nil {
return errors.Wrap(err, "Cannot parse CloudwatchSource configuration")
}
cw.Config = cwConfig
if len(cw.Config.GroupName) == 0 {
return fmt.Errorf("group_name is mandatory for CloudwatchSource")
}
cw.logger = logger.WithField("group", cw.Config.GroupName)
if cw.Config.Mode == "" {
cw.Config.Mode = configuration.TAIL_MODE
}
logger.Debugf("Starting configuration for Cloudwatch group %s", cw.Config.GroupName)
if cw.Config.DescribeLogStreamsLimit == nil {
cw.Config.DescribeLogStreamsLimit = &def_DescribeLogStreamsLimit
}
logger.Tracef("describelogstreams_limit set to %d", *cw.Config.DescribeLogStreamsLimit)
if cw.Config.PollNewStreamInterval == nil {
cw.Config.PollNewStreamInterval = &def_PollNewStreamInterval
}
logger.Tracef("poll_new_stream_interval set to %v", *cw.Config.PollNewStreamInterval)
if cw.Config.MaxStreamAge == nil {
cw.Config.MaxStreamAge = &def_MaxStreamAge
}
logger.Tracef("max_stream_age set to %v", *cw.Config.MaxStreamAge)
if cw.Config.PollStreamInterval == nil {
cw.Config.PollStreamInterval = &def_PollStreamInterval
}
logger.Tracef("poll_stream_interval set to %v", *cw.Config.PollStreamInterval)
if cw.Config.StreamReadTimeout == nil {
cw.Config.StreamReadTimeout = &def_StreamReadTimeout
}
logger.Tracef("stream_read_timeout set to %v", *cw.Config.StreamReadTimeout)
if cw.Config.GetLogEventsPagesLimit == nil {
cw.Config.GetLogEventsPagesLimit = &def_GetLogEventsPagesLimit
}
logger.Tracef("getlogeventspages_limit set to %v", *cw.Config.GetLogEventsPagesLimit)
if cw.Config.AwsApiCallTimeout == nil {
cw.Config.AwsApiCallTimeout = &def_AwsApiCallTimeout
}
logger.Tracef("aws_api_timeout set to %v", *cw.Config.AwsApiCallTimeout)
if *cw.Config.MaxStreamAge > *cw.Config.StreamReadTimeout {
logger.Warningf("max_stream_age > stream_read_timeout, stream might keep being opened/closed")
}
if cw.Config.AwsConfigDir == nil {
cw.Config.AwsConfigDir = &def_AwsConfigDir
}
logger.Tracef("aws_config_dir set to %s", *cw.Config.AwsConfigDir)
if *cw.Config.AwsConfigDir != "" {
_, err := os.Stat(*cw.Config.AwsConfigDir)
if err != nil {
logger.Errorf("can't read aws_config_dir '%s' got err %s", *cw.Config.AwsConfigDir, err)
return fmt.Errorf("can't read aws_config_dir %s got err %s ", *cw.Config.AwsConfigDir, err)
}
os.Setenv("AWS_SDK_LOAD_CONFIG", "1")
//as aws sdk relies on $HOME, let's allow the user to override it :)
os.Setenv("AWS_CONFIG_FILE", fmt.Sprintf("%s/config", *cw.Config.AwsConfigDir))
os.Setenv("AWS_SHARED_CREDENTIALS_FILE", fmt.Sprintf("%s/credentials", *cw.Config.AwsConfigDir))
} else {
if cw.Config.AwsRegion == nil {
logger.Errorf("aws_region is not specified, specify it or aws_config_dir")
return fmt.Errorf("aws_region is not specified, specify it or aws_config_dir")
}
os.Setenv("AWS_REGION", *cw.Config.AwsRegion)
}
if err := cw.newClient(); err != nil {
return err
}
cw.streamIndexes = make(map[string]string)
if cw.Config.StreamRegexp != nil {
if _, err := regexp.Compile(*cw.Config.StreamRegexp); err != nil {
return errors.Wrapf(err, "error while compiling regexp '%s'", *cw.Config.StreamRegexp)
}
targetStream = *cw.Config.StreamRegexp
} else if cw.Config.StreamName != nil {
targetStream = *cw.Config.StreamName
}
logger.Infof("Adding cloudwatch group '%s' (stream:%s) to datasources", cw.Config.GroupName, targetStream)
return nil
}
func (cw *CloudwatchSource) newClient() error {
var sess *session.Session
if cw.Config.AwsProfile != nil {
sess = session.Must(session.NewSessionWithOptions(session.Options{
SharedConfigState: session.SharedConfigEnable,
Profile: *cw.Config.AwsProfile,
}))
} else {
sess = session.Must(session.NewSessionWithOptions(session.Options{
SharedConfigState: session.SharedConfigEnable,
}))
}
if sess == nil {
return fmt.Errorf("failed to create aws session")
}
if v := os.Getenv("AWS_ENDPOINT_FORCE"); v != "" {
cw.logger.Debugf("[testing] overloading endpoint with %s", v)
cw.cwClient = cloudwatchlogs.New(sess, aws.NewConfig().WithEndpoint(v))
} else {
cw.cwClient = cloudwatchlogs.New(sess)
}
if cw.cwClient == nil {
return fmt.Errorf("failed to create cloudwatch client")
}
return nil
}
func (cw *CloudwatchSource) StreamingAcquisition(out chan types.Event, t *tomb.Tomb) error {
cw.t = t
monitChan := make(chan LogStreamTailConfig)
t.Go(func() error {
return cw.LogStreamManager(monitChan, out)
})
return cw.WatchLogGroupForStreams(monitChan)
}
func (cw *CloudwatchSource) GetMetrics() []prometheus.Collector {
return []prometheus.Collector{linesRead, openedStreams}
}
func (cw *CloudwatchSource) GetAggregMetrics() []prometheus.Collector {
return []prometheus.Collector{linesRead, openedStreams}
}
func (cw *CloudwatchSource) GetMode() string {
return cw.Config.Mode
}
func (cw *CloudwatchSource) GetName() string {
return "cloudwatch"
}
func (cw *CloudwatchSource) CanRun() error {
return nil
}
func (cw *CloudwatchSource) Dump() interface{} {
return cw
}
func (cw *CloudwatchSource) WatchLogGroupForStreams(out chan LogStreamTailConfig) error {
cw.logger.Debugf("Starting to watch group (interval:%s)", cw.Config.PollNewStreamInterval)
ticker := time.NewTicker(*cw.Config.PollNewStreamInterval)
var startFrom *string
for {
select {
case <-cw.t.Dying():
cw.logger.Infof("stopping group watch")
return nil
case <-ticker.C:
hasMoreStreams := true
startFrom = nil
for hasMoreStreams {
cw.logger.Tracef("doing the call to DescribeLogStreamsPagesWithContext")
ctx := context.Background()
//there can be a lot of streams in a group, and we're only interested in those recently written to, so we sort by LastEventTime
err := cw.cwClient.DescribeLogStreamsPagesWithContext(
ctx,
&cloudwatchlogs.DescribeLogStreamsInput{
LogGroupName: aws.String(cw.Config.GroupName),
Descending: aws.Bool(true),
NextToken: startFrom,
OrderBy: aws.String(cloudwatchlogs.OrderByLastEventTime),
Limit: cw.Config.DescribeLogStreamsLimit,
},
func(page *cloudwatchlogs.DescribeLogStreamsOutput, lastPage bool) bool {
cw.logger.Tracef("in helper of DescribeLogStreamsPagesWithContext")
for _, event := range page.LogStreams {
startFrom = page.NextToken
//we check if the stream has been written to recently enough to be monitored
if event.LastIngestionTime != nil {
//aws uses millisecond since the epoch
oldest := time.Now().UTC().Add(-*cw.Config.MaxStreamAge)
//TBD : verify that this is correct : Unix 2nd arg expects Nanoseconds, and have a code that is more explicit.
LastIngestionTime := time.Unix(0, *event.LastIngestionTime*int64(time.Millisecond))
if LastIngestionTime.Before(oldest) {
cw.logger.Tracef("stop iteration, %s reached oldest age, stop (%s < %s)", *event.LogStreamName, LastIngestionTime, time.Now().UTC().Add(-*cw.Config.MaxStreamAge))
hasMoreStreams = false
return false
}
cw.logger.Tracef("stream %s is elligible for monitoring", *event.LogStreamName)
//the stream has been updated recently, check if we should monitor it
var expectMode int
if !cw.Config.UseTimeMachine {
expectMode = leaky.LIVE
} else {
expectMode = leaky.TIMEMACHINE
}
monitorStream := LogStreamTailConfig{
GroupName: cw.Config.GroupName,
StreamName: *event.LogStreamName,
GetLogEventsPagesLimit: *cw.Config.GetLogEventsPagesLimit,
PollStreamInterval: *cw.Config.PollStreamInterval,
StreamReadTimeout: *cw.Config.StreamReadTimeout,
PrependCloudwatchTimestamp: cw.Config.PrependCloudwatchTimestamp,
ExpectMode: expectMode,
Labels: cw.Config.Labels,
}
out <- monitorStream
}
}
if lastPage {
cw.logger.Tracef("reached last page")
hasMoreStreams = false
}
return true
},
)
if err != nil {
newerr := errors.Wrapf(err, "while describing group %s", cw.Config.GroupName)
return newerr
}
cw.logger.Tracef("after DescribeLogStreamsPagesWithContext")
}
}
}
}
//LogStreamManager receives the potential streams to monitor, and starts a go routine when needed
func (cw *CloudwatchSource) LogStreamManager(in chan LogStreamTailConfig, outChan chan types.Event) error {
cw.logger.Debugf("starting to monitor streams for %s", cw.Config.GroupName)
pollDeadStreamInterval := time.NewTicker(def_PollDeadStreamInterval)
for {
select {
case newStream := <-in: //nolint:govet // copylocks won't matter if the tomb is not initialized
shouldCreate := true
cw.logger.Tracef("received new streams to monitor : %s/%s", newStream.GroupName, newStream.StreamName)
if cw.Config.StreamName != nil && newStream.StreamName != *cw.Config.StreamName {
cw.logger.Tracef("stream %s != %s", newStream.StreamName, *cw.Config.StreamName)
continue
}
if cw.Config.StreamRegexp != nil {
match, err := regexp.Match(*cw.Config.StreamRegexp, []byte(newStream.StreamName))
if err != nil {
cw.logger.Warningf("invalid regexp : %s", err)
} else if !match {
cw.logger.Tracef("stream %s doesn't match %s", newStream.StreamName, *cw.Config.StreamRegexp)
continue
}
}
for idx, stream := range cw.monitoredStreams {
if newStream.GroupName == stream.GroupName && newStream.StreamName == stream.StreamName {
//stream exists, but is dead, remove it from list
if !stream.t.Alive() {
cw.logger.Debugf("stream %s already exists, but is dead", newStream.StreamName)
cw.monitoredStreams = append(cw.monitoredStreams[:idx], cw.monitoredStreams[idx+1:]...)
openedStreams.With(prometheus.Labels{"group": newStream.GroupName}).Dec()
break
}
shouldCreate = false
break
}
}
//let's start watching this stream
if shouldCreate {
openedStreams.With(prometheus.Labels{"group": newStream.GroupName}).Inc()
newStream.t = tomb.Tomb{}
newStream.logger = cw.logger.WithFields(log.Fields{"stream": newStream.StreamName})
cw.logger.Debugf("starting tail of stream %s", newStream.StreamName)
newStream.t.Go(func() error {
return cw.TailLogStream(&newStream, outChan)
})
cw.monitoredStreams = append(cw.monitoredStreams, &newStream)
}
case <-pollDeadStreamInterval.C:
newMonitoredStreams := cw.monitoredStreams[:0]
for idx, stream := range cw.monitoredStreams {
if !cw.monitoredStreams[idx].t.Alive() {
cw.logger.Debugf("remove dead stream %s", stream.StreamName)
openedStreams.With(prometheus.Labels{"group": cw.monitoredStreams[idx].GroupName}).Dec()
} else {
newMonitoredStreams = append(newMonitoredStreams, stream)
}
}
cw.monitoredStreams = newMonitoredStreams
case <-cw.t.Dying():
cw.logger.Infof("LogStreamManager for %s is dying, %d alive streams", cw.Config.GroupName, len(cw.monitoredStreams))
for idx, stream := range cw.monitoredStreams {
if cw.monitoredStreams[idx].t.Alive() {
cw.logger.Debugf("killing stream %s", stream.StreamName)
cw.monitoredStreams[idx].t.Kill(nil)
if err := cw.monitoredStreams[idx].t.Wait(); err != nil {
cw.logger.Debugf("error while waiting for death of %s : %s", stream.StreamName, err)
}
}
}
cw.monitoredStreams = nil
cw.logger.Debugf("routine cleanup done, return")
return nil
}
}
}
func (cw *CloudwatchSource) TailLogStream(cfg *LogStreamTailConfig, outChan chan types.Event) error {
var startFrom *string
lastReadMessage := time.Now().UTC()
ticker := time.NewTicker(cfg.PollStreamInterval)
//resume at existing index if we already had
streamIndexMutex.Lock()
v := cw.streamIndexes[cfg.GroupName+"+"+cfg.StreamName]
streamIndexMutex.Unlock()
if v != "" {
cfg.logger.Debugf("restarting on index %s", v)
startFrom = &v
}
/*during first run, we want to avoid reading any message, but just get a token.
if we don't, we might end up sending the same item several times. hence the 'startup' hack */
for {
select {
case <-ticker.C:
cfg.logger.Tracef("entering loop")
hasMorePages := true
for hasMorePages {
/*for the first call, we only consume the last item*/
cfg.logger.Tracef("calling GetLogEventsPagesWithContext")
ctx := context.Background()
err := cw.cwClient.GetLogEventsPagesWithContext(ctx,
&cloudwatchlogs.GetLogEventsInput{
Limit: aws.Int64(cfg.GetLogEventsPagesLimit),
LogGroupName: aws.String(cfg.GroupName),
LogStreamName: aws.String(cfg.StreamName),
NextToken: startFrom,
StartFromHead: aws.Bool(true),
},
func(page *cloudwatchlogs.GetLogEventsOutput, lastPage bool) bool {
cfg.logger.Tracef("%d results, last:%t", len(page.Events), lastPage)
startFrom = page.NextForwardToken
if page.NextForwardToken != nil {
streamIndexMutex.Lock()
cw.streamIndexes[cfg.GroupName+"+"+cfg.StreamName] = *page.NextForwardToken
streamIndexMutex.Unlock()
}
if lastPage { /*wait another ticker to check on new log availability*/
cfg.logger.Tracef("last page")
hasMorePages = false
}
if len(page.Events) > 0 {
lastReadMessage = time.Now().UTC()
}
for _, event := range page.Events {
evt, err := cwLogToEvent(event, cfg)
if err != nil {
cfg.logger.Warningf("cwLogToEvent error, discarded event : %s", err)
} else {
cfg.logger.Debugf("pushing message : %s", evt.Line.Raw)
linesRead.With(prometheus.Labels{"group": cfg.GroupName, "stream": cfg.StreamName}).Inc()
outChan <- evt
}
}
return true
},
)
if err != nil {
newerr := errors.Wrapf(err, "while reading %s/%s", cfg.GroupName, cfg.StreamName)
cfg.logger.Warningf("err : %s", newerr)
return newerr
}
cfg.logger.Tracef("done reading GetLogEventsPagesWithContext")
if time.Since(lastReadMessage) > cfg.StreamReadTimeout {
cfg.logger.Infof("%s/%s reached timeout (%s) (last message was %s)", cfg.GroupName, cfg.StreamName, time.Since(lastReadMessage),
lastReadMessage)
return nil
}
}
case <-cfg.t.Dying():
cfg.logger.Infof("logstream tail stopping")
return fmt.Errorf("killed")
}
}
}
func (cw *CloudwatchSource) ConfigureByDSN(dsn string, labels map[string]string, logger *log.Entry) error {
cw.logger = logger
dsn = strings.TrimPrefix(dsn, cw.GetName()+"://")
args := strings.Split(dsn, "?")
if len(args) != 2 {
return fmt.Errorf("query is mandatory (at least start_date and end_date or backlog)")
}
frags := strings.Split(args[0], ":")
if len(frags) != 2 {
return fmt.Errorf("cloudwatch path must contain group and stream : /my/group/name:stream/name")
}
cw.Config.GroupName = frags[0]
cw.Config.StreamName = &frags[1]
cw.Config.Labels = labels
u, err := url.ParseQuery(args[1])
if err != nil {
return errors.Wrapf(err, "while parsing %s", dsn)
}
for k, v := range u {
switch k {
case "log_level":
if len(v) != 1 {
return fmt.Errorf("expected zero or one value for 'log_level'")
}
lvl, err := log.ParseLevel(v[0])
if err != nil {
return errors.Wrapf(err, "unknown level %s", v[0])
}
cw.logger.Logger.SetLevel(lvl)
case "profile":
if len(v) != 1 {
return fmt.Errorf("expected zero or one value for 'profile'")
}
awsprof := v[0]
cw.Config.AwsProfile = &awsprof
cw.logger.Debugf("profile set to '%s'", *cw.Config.AwsProfile)
case "start_date":
if len(v) != 1 {
return fmt.Errorf("expected zero or one argument for 'start_date'")
}
//let's reuse our parser helper so that a ton of date formats are supported
strdate, startDate := parser.GenDateParse(v[0])
cw.logger.Debugf("parsed '%s' as '%s'", v[0], strdate)
cw.Config.StartTime = &startDate
case "end_date":
if len(v) != 1 {
return fmt.Errorf("expected zero or one argument for 'end_date'")
}
//let's reuse our parser helper so that a ton of date formats are supported
strdate, endDate := parser.GenDateParse(v[0])
cw.logger.Debugf("parsed '%s' as '%s'", v[0], strdate)
cw.Config.EndTime = &endDate
case "backlog":
if len(v) != 1 {
return fmt.Errorf("expected zero or one argument for 'backlog'")
}
//let's reuse our parser helper so that a ton of date formats are supported
duration, err := time.ParseDuration(v[0])
if err != nil {
return errors.Wrapf(err, "unable to parse '%s' as duration", v[0])
}
cw.logger.Debugf("parsed '%s' as '%s'", v[0], duration)
start := time.Now().UTC().Add(-duration)
cw.Config.StartTime = &start
end := time.Now().UTC()
cw.Config.EndTime = &end
default:
return fmt.Errorf("unexpected argument %s", k)
}
}
cw.logger.Tracef("host=%s", cw.Config.GroupName)
cw.logger.Tracef("stream=%s", *cw.Config.StreamName)
cw.Config.GetLogEventsPagesLimit = &def_GetLogEventsPagesLimit
if err := cw.newClient(); err != nil {
return err
}
if cw.Config.StreamName == nil || cw.Config.GroupName == "" {
return fmt.Errorf("missing stream or group name")
}
if cw.Config.StartTime == nil || cw.Config.EndTime == nil {
return fmt.Errorf("start_date and end_date or backlog are mandatory in one-shot mode")
}
cw.Config.Mode = configuration.CAT_MODE
cw.streamIndexes = make(map[string]string)
cw.t = &tomb.Tomb{}
return nil
}
func (cw *CloudwatchSource) OneShotAcquisition(out chan types.Event, t *tomb.Tomb) error {
//StreamName string, Start time.Time, End time.Time
config := LogStreamTailConfig{
GroupName: cw.Config.GroupName,
StreamName: *cw.Config.StreamName,
StartTime: *cw.Config.StartTime,
EndTime: *cw.Config.EndTime,
GetLogEventsPagesLimit: *cw.Config.GetLogEventsPagesLimit,
logger: cw.logger.WithFields(log.Fields{
"group": cw.Config.GroupName,
"stream": *cw.Config.StreamName,
}),
Labels: cw.Config.Labels,
ExpectMode: leaky.TIMEMACHINE,
}
return cw.CatLogStream(&config, out)
}
func (cw *CloudwatchSource) CatLogStream(cfg *LogStreamTailConfig, outChan chan types.Event) error {
var startFrom *string
var head = true
/*convert the times*/
startTime := cfg.StartTime.UTC().Unix() * 1000
endTime := cfg.EndTime.UTC().Unix() * 1000
hasMoreEvents := true
for hasMoreEvents {
select {
default:
cfg.logger.Tracef("Calling GetLogEventsPagesWithContext(%s, %s), startTime:%d / endTime:%d",
cfg.GroupName, cfg.StreamName, startTime, endTime)
cfg.logger.Tracef("startTime:%s / endTime:%s", cfg.StartTime, cfg.EndTime)
if startFrom != nil {
cfg.logger.Tracef("next_token: %s", *startFrom)
}
ctx := context.Background()
err := cw.cwClient.GetLogEventsPagesWithContext(ctx,
&cloudwatchlogs.GetLogEventsInput{
Limit: aws.Int64(10),
LogGroupName: aws.String(cfg.GroupName),
LogStreamName: aws.String(cfg.StreamName),
StartTime: aws.Int64(startTime),
EndTime: aws.Int64(endTime),
StartFromHead: &head,
NextToken: startFrom,
},
func(page *cloudwatchlogs.GetLogEventsOutput, lastPage bool) bool {
cfg.logger.Tracef("in GetLogEventsPagesWithContext handker (%d events) (last:%t)", len(page.Events), lastPage)
for _, event := range page.Events {
evt, err := cwLogToEvent(event, cfg)
if err != nil {
cfg.logger.Warningf("discard event : %s", err)
}
cfg.logger.Debugf("pushing message : %s", evt.Line.Raw)
outChan <- evt
}
if startFrom != nil && *page.NextForwardToken == *startFrom {
cfg.logger.Debugf("reached end of available events")
hasMoreEvents = false
return false
}
startFrom = page.NextForwardToken
return true
},
)
if err != nil {
return errors.Wrapf(err, "while reading logs from %s/%s", cfg.GroupName, cfg.StreamName)
}
cfg.logger.Tracef("after GetLogEventsPagesWithContext")
case <-cw.t.Dying():
cfg.logger.Warningf("cat stream killed")
return nil
}
}
cfg.logger.Tracef("CatLogStream out")
return nil
}
func cwLogToEvent(log *cloudwatchlogs.OutputLogEvent, cfg *LogStreamTailConfig) (types.Event, error) {
l := types.Line{}
evt := types.Event{}
if log.Message == nil {
return evt, fmt.Errorf("nil message")
}
msg := *log.Message
if cfg.PrependCloudwatchTimestamp != nil && *cfg.PrependCloudwatchTimestamp {
eventTimestamp := time.Unix(0, *log.Timestamp*int64(time.Millisecond))
msg = eventTimestamp.String() + " " + msg
}
l.Raw = msg
l.Labels = cfg.Labels
l.Time = time.Now().UTC()
l.Src = fmt.Sprintf("%s/%s", cfg.GroupName, cfg.StreamName)
l.Process = true
l.Module = "cloudwatch"
evt.Line = l
evt.Process = true
evt.Type = types.LOG
evt.ExpectMode = cfg.ExpectMode
cfg.logger.Debugf("returned event labels : %+v", evt.Line.Labels)
return evt, nil
}
crowdsec-1.4.6/pkg/acquisition/modules/cloudwatch/cloudwatch_test.go 0000664 0000000 0000000 00000061272 14371172257 0026015 0 ustar 00root root 0000000 0000000 package cloudwatchacquisition
import (
"fmt"
"net"
"os"
"runtime"
"strings"
"testing"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/cloudwatchlogs"
"github.com/crowdsecurity/crowdsec/pkg/cstest"
"github.com/crowdsecurity/crowdsec/pkg/types"
log "github.com/sirupsen/logrus"
"github.com/stretchr/testify/require"
"gopkg.in/tomb.v2"
)
/*
test plan :
- start on bad group/bad stream
- start on good settings (oneshot) -> check expected messages
- start on good settings (stream) -> check expected messages within given time
- check shutdown/restart
*/
func deleteAllLogGroups(t *testing.T, cw *CloudwatchSource) {
input := &cloudwatchlogs.DescribeLogGroupsInput{}
result, err := cw.cwClient.DescribeLogGroups(input)
require.NoError(t, err)
for _, group := range result.LogGroups {
_, err := cw.cwClient.DeleteLogGroup(&cloudwatchlogs.DeleteLogGroupInput{
LogGroupName: group.LogGroupName,
})
require.NoError(t, err)
}
}
func checkForLocalStackAvailability() error {
v := os.Getenv("AWS_ENDPOINT_FORCE")
if v == "" {
return fmt.Errorf("missing aws endpoint for tests : AWS_ENDPOINT_FORCE")
}
v = strings.TrimPrefix(v, "http://")
_, err := net.Dial("tcp", v)
if err != nil {
return fmt.Errorf("while dialing %s : %s : aws endpoint isn't available", v, err)
}
return nil
}
func TestMain(m *testing.M) {
if runtime.GOOS == "windows" {
os.Exit(0)
}
if err := checkForLocalStackAvailability(); err != nil {
log.Fatalf("local stack error : %s", err)
}
def_PollNewStreamInterval = 1 * time.Second
def_PollStreamInterval = 1 * time.Second
def_StreamReadTimeout = 10 * time.Second
def_MaxStreamAge = 5 * time.Second
def_PollDeadStreamInterval = 5 * time.Second
os.Exit(m.Run())
}
func TestWatchLogGroupForStreams(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("Skipping test on windows")
}
log.SetLevel(log.DebugLevel)
tests := []struct {
config []byte
expectedCfgErr string
expectedStartErr string
name string
setup func(*testing.T, *CloudwatchSource)
run func(*testing.T, *CloudwatchSource)
teardown func(*testing.T, *CloudwatchSource)
expectedResLen int
expectedResMessages []string
}{
// require a group name that doesn't exist
{
name: "group_does_not_exists",
config: []byte(`
source: cloudwatch
aws_region: us-east-1
labels:
type: test_source
group_name: b
stream_name: test_stream`),
expectedStartErr: "The specified log group does not exist",
setup: func(t *testing.T, cw *CloudwatchSource) {
deleteAllLogGroups(t, cw)
_, err := cw.cwClient.CreateLogGroup(&cloudwatchlogs.CreateLogGroupInput{
LogGroupName: aws.String("test_group_not_used_1"),
})
require.NoError(t, err)
},
teardown: func(t *testing.T, cw *CloudwatchSource) {
_, err := cw.cwClient.DeleteLogGroup(&cloudwatchlogs.DeleteLogGroupInput{
LogGroupName: aws.String("test_group_not_used_1"),
})
require.NoError(t, err)
},
},
// test stream mismatch
{
name: "group_exists_bad_stream_name",
config: []byte(`
source: cloudwatch
aws_region: us-east-1
labels:
type: test_source
group_name: test_group1
stream_name: test_stream_bad`),
setup: func(t *testing.T, cw *CloudwatchSource) {
deleteAllLogGroups(t, cw)
_, err := cw.cwClient.CreateLogGroup(&cloudwatchlogs.CreateLogGroupInput{
LogGroupName: aws.String("test_group1"),
})
require.NoError(t, err)
_, err = cw.cwClient.CreateLogStream(&cloudwatchlogs.CreateLogStreamInput{
LogGroupName: aws.String("test_group1"),
LogStreamName: aws.String("test_stream"),
})
require.NoError(t, err)
// have a message before we start - won't be popped, but will trigger stream monitoring
_, err = cw.cwClient.PutLogEvents(&cloudwatchlogs.PutLogEventsInput{
LogGroupName: aws.String("test_group1"),
LogStreamName: aws.String("test_stream"),
LogEvents: []*cloudwatchlogs.InputLogEvent{
{
Message: aws.String("test_message_1"),
Timestamp: aws.Int64(time.Now().UTC().Unix() * 1000),
},
},
})
require.NoError(t, err)
},
teardown: func(t *testing.T, cw *CloudwatchSource) {
_, err := cw.cwClient.DeleteLogGroup(&cloudwatchlogs.DeleteLogGroupInput{
LogGroupName: aws.String("test_group1"),
})
require.NoError(t, err)
},
expectedResLen: 0,
},
// test stream mismatch
{
name: "group_exists_bad_stream_regexp",
config: []byte(`
source: cloudwatch
aws_region: us-east-1
labels:
type: test_source
group_name: test_group1
stream_regexp: test_bad[0-9]+`),
setup: func(t *testing.T, cw *CloudwatchSource) {
deleteAllLogGroups(t, cw)
_, err := cw.cwClient.CreateLogGroup(&cloudwatchlogs.CreateLogGroupInput{
LogGroupName: aws.String("test_group1"),
})
require.NoError(t, err)
_, err = cw.cwClient.CreateLogStream(&cloudwatchlogs.CreateLogStreamInput{
LogGroupName: aws.String("test_group1"),
LogStreamName: aws.String("test_stream"),
})
require.NoError(t, err)
// have a message before we start - won't be popped, but will trigger stream monitoring
_, err = cw.cwClient.PutLogEvents(&cloudwatchlogs.PutLogEventsInput{
LogGroupName: aws.String("test_group1"),
LogStreamName: aws.String("test_stream"),
LogEvents: []*cloudwatchlogs.InputLogEvent{
{
Message: aws.String("test_message_1"),
Timestamp: aws.Int64(time.Now().UTC().Unix() * 1000),
},
},
})
require.NoError(t, err)
},
teardown: func(t *testing.T, cw *CloudwatchSource) {
_, err := cw.cwClient.DeleteLogGroup(&cloudwatchlogs.DeleteLogGroupInput{
LogGroupName: aws.String("test_group1"),
})
require.NoError(t, err)
},
expectedResLen: 0,
},
// require a group name that does exist and contains a stream in which we gonna put events
{
name: "group_exists_stream_exists_has_events",
config: []byte(`
source: cloudwatch
aws_region: us-east-1
labels:
type: test_source
group_name: test_log_group1
log_level: trace
stream_name: test_stream`),
// expectedStartErr: "The specified log group does not exist",
setup: func(t *testing.T, cw *CloudwatchSource) {
deleteAllLogGroups(t, cw)
_, err := cw.cwClient.CreateLogGroup(&cloudwatchlogs.CreateLogGroupInput{
LogGroupName: aws.String("test_log_group1"),
})
require.NoError(t, err)
_, err = cw.cwClient.CreateLogStream(&cloudwatchlogs.CreateLogStreamInput{
LogGroupName: aws.String("test_log_group1"),
LogStreamName: aws.String("test_stream"),
})
require.NoError(t, err)
// have a message before we start - won't be popped, but will trigger stream monitoring
_, err = cw.cwClient.PutLogEvents(&cloudwatchlogs.PutLogEventsInput{
LogGroupName: aws.String("test_log_group1"),
LogStreamName: aws.String("test_stream"),
LogEvents: []*cloudwatchlogs.InputLogEvent{
{
Message: aws.String("test_message_1"),
Timestamp: aws.Int64(time.Now().UTC().Unix() * 1000),
},
},
})
require.NoError(t, err)
},
run: func(t *testing.T, cw *CloudwatchSource) {
// wait for new stream pickup + stream poll interval
time.Sleep(def_PollNewStreamInterval + (1 * time.Second))
time.Sleep(def_PollStreamInterval + (1 * time.Second))
_, err := cw.cwClient.PutLogEvents(&cloudwatchlogs.PutLogEventsInput{
LogGroupName: aws.String("test_log_group1"),
LogStreamName: aws.String("test_stream"),
LogEvents: []*cloudwatchlogs.InputLogEvent{
{
Message: aws.String("test_message_4"),
Timestamp: aws.Int64(time.Now().UTC().Unix() * 1000),
},
// and add an event in the future that will be popped
{
Message: aws.String("test_message_5"),
Timestamp: aws.Int64(time.Now().UTC().Unix() * 1000),
},
},
})
require.NoError(t, err)
},
teardown: func(t *testing.T, cw *CloudwatchSource) {
_, err := cw.cwClient.DeleteLogStream(&cloudwatchlogs.DeleteLogStreamInput{
LogGroupName: aws.String("test_log_group1"),
LogStreamName: aws.String("test_stream"),
})
require.NoError(t, err)
_, err = cw.cwClient.DeleteLogGroup(&cloudwatchlogs.DeleteLogGroupInput{
LogGroupName: aws.String("test_log_group1"),
})
require.NoError(t, err)
},
expectedResLen: 3,
expectedResMessages: []string{"test_message_1", "test_message_4", "test_message_5"},
},
// have a stream generate events, reach time-out and gets polled again
{
name: "group_exists_stream_exists_has_events+timeout",
config: []byte(`
source: cloudwatch
aws_region: us-east-1
labels:
type: test_source
group_name: test_log_group1
log_level: trace
stream_name: test_stream`),
// expectedStartErr: "The specified log group does not exist",
setup: func(t *testing.T, cw *CloudwatchSource) {
deleteAllLogGroups(t, cw)
_, err := cw.cwClient.CreateLogGroup(&cloudwatchlogs.CreateLogGroupInput{
LogGroupName: aws.String("test_log_group1"),
})
require.NoError(t, err)
_, err = cw.cwClient.CreateLogStream(&cloudwatchlogs.CreateLogStreamInput{
LogGroupName: aws.String("test_log_group1"),
LogStreamName: aws.String("test_stream"),
})
require.NoError(t, err)
// have a message before we start - won't be popped, but will trigger stream monitoring
_, err = cw.cwClient.PutLogEvents(&cloudwatchlogs.PutLogEventsInput{
LogGroupName: aws.String("test_log_group1"),
LogStreamName: aws.String("test_stream"),
LogEvents: []*cloudwatchlogs.InputLogEvent{
{
Message: aws.String("test_message_1"),
Timestamp: aws.Int64(time.Now().UTC().Unix() * 1000),
},
},
})
require.NoError(t, err)
},
run: func(t *testing.T, cw *CloudwatchSource) {
// wait for new stream pickup + stream poll interval
time.Sleep(def_PollNewStreamInterval + (1 * time.Second))
time.Sleep(def_PollStreamInterval + (1 * time.Second))
// send some events
_, err := cw.cwClient.PutLogEvents(&cloudwatchlogs.PutLogEventsInput{
LogGroupName: aws.String("test_log_group1"),
LogStreamName: aws.String("test_stream"),
LogEvents: []*cloudwatchlogs.InputLogEvent{
{
Message: aws.String("test_message_41"),
Timestamp: aws.Int64(time.Now().UTC().Unix() * 1000),
},
},
})
require.NoError(t, err)
// wait for the stream to time-out
time.Sleep(def_StreamReadTimeout + (1 * time.Second))
// and send events again
_, err = cw.cwClient.PutLogEvents(&cloudwatchlogs.PutLogEventsInput{
LogGroupName: aws.String("test_log_group1"),
LogStreamName: aws.String("test_stream"),
LogEvents: []*cloudwatchlogs.InputLogEvent{
{
Message: aws.String("test_message_51"),
Timestamp: aws.Int64(time.Now().UTC().Unix() * 1000),
},
},
})
require.NoError(t, err)
// wait for new stream pickup + stream poll interval
time.Sleep(def_PollNewStreamInterval + (1 * time.Second))
time.Sleep(def_PollStreamInterval + (1 * time.Second))
},
teardown: func(t *testing.T, cw *CloudwatchSource) {
_, err := cw.cwClient.DeleteLogStream(&cloudwatchlogs.DeleteLogStreamInput{
LogGroupName: aws.String("test_log_group1"),
LogStreamName: aws.String("test_stream"),
})
require.NoError(t, err)
_, err = cw.cwClient.DeleteLogGroup(&cloudwatchlogs.DeleteLogGroupInput{
LogGroupName: aws.String("test_log_group1"),
})
require.NoError(t, err)
},
expectedResLen: 3,
expectedResMessages: []string{"test_message_1", "test_message_41", "test_message_51"},
},
// have a stream generate events, reach time-out and dead body collection
{
name: "group_exists_stream_exists_has_events+timeout+GC",
config: []byte(`
source: cloudwatch
aws_region: us-east-1
labels:
type: test_source
group_name: test_log_group1
log_level: trace
stream_name: test_stream`),
// expectedStartErr: "The specified log group does not exist",
setup: func(t *testing.T, cw *CloudwatchSource) {
deleteAllLogGroups(t, cw)
_, err := cw.cwClient.CreateLogGroup(&cloudwatchlogs.CreateLogGroupInput{
LogGroupName: aws.String("test_log_group1"),
})
require.NoError(t, err)
_, err = cw.cwClient.CreateLogStream(&cloudwatchlogs.CreateLogStreamInput{
LogGroupName: aws.String("test_log_group1"),
LogStreamName: aws.String("test_stream"),
})
require.NoError(t, err)
// have a message before we start - won't be popped, but will trigger stream monitoring
_, err = cw.cwClient.PutLogEvents(&cloudwatchlogs.PutLogEventsInput{
LogGroupName: aws.String("test_log_group1"),
LogStreamName: aws.String("test_stream"),
LogEvents: []*cloudwatchlogs.InputLogEvent{
{
Message: aws.String("test_message_1"),
Timestamp: aws.Int64(time.Now().UTC().Unix() * 1000),
},
},
})
require.NoError(t, err)
},
run: func(t *testing.T, cw *CloudwatchSource) {
// wait for new stream pickup + stream poll interval
time.Sleep(def_PollNewStreamInterval + (1 * time.Second))
time.Sleep(def_PollStreamInterval + (1 * time.Second))
time.Sleep(def_PollDeadStreamInterval + (1 * time.Second))
},
teardown: func(t *testing.T, cw *CloudwatchSource) {
_, err := cw.cwClient.DeleteLogStream(&cloudwatchlogs.DeleteLogStreamInput{
LogGroupName: aws.String("test_log_group1"),
LogStreamName: aws.String("test_stream"),
})
require.NoError(t, err)
_, err = cw.cwClient.DeleteLogGroup(&cloudwatchlogs.DeleteLogGroupInput{
LogGroupName: aws.String("test_log_group1"),
})
require.NoError(t, err)
},
expectedResLen: 1,
},
}
for _, tc := range tests {
tc := tc
t.Run(tc.name, func(t *testing.T) {
dbgLogger := log.New().WithField("test", tc.name)
dbgLogger.Logger.SetLevel(log.DebugLevel)
dbgLogger.Infof("starting test")
cw := CloudwatchSource{}
err := cw.Configure(tc.config, dbgLogger)
cstest.RequireErrorContains(t, err, tc.expectedCfgErr)
if tc.expectedCfgErr != "" {
return
}
// run pre-routine : tests use it to set group & streams etc.
if tc.setup != nil {
tc.setup(t, &cw)
}
out := make(chan types.Event)
tmb := tomb.Tomb{}
var rcvdEvts []types.Event
dbgLogger.Infof("running StreamingAcquisition")
actmb := tomb.Tomb{}
actmb.Go(func() error {
err := cw.StreamingAcquisition(out, &actmb)
dbgLogger.Infof("acquis done")
cstest.RequireErrorContains(t, err, tc.expectedStartErr)
return nil
})
// let's empty output chan
tmb.Go(func() error {
for {
select {
case in := <-out:
log.Debugf("received event %+v", in)
rcvdEvts = append(rcvdEvts, in)
case <-tmb.Dying():
log.Debugf("pumper died")
return nil
}
}
})
if tc.run != nil {
tc.run(t, &cw)
} else {
dbgLogger.Warning("no code to run")
}
time.Sleep(5 * time.Second)
dbgLogger.Infof("killing collector")
tmb.Kill(nil)
<-tmb.Dead()
dbgLogger.Infof("killing datasource")
actmb.Kill(nil)
<-actmb.Dead()
// dbgLogger.Infof("collected events : %d -> %+v", len(rcvd_evts), rcvd_evts)
// check results
if tc.expectedResLen != -1 {
if tc.expectedResLen != len(rcvdEvts) {
t.Fatalf("%s : expected %d results got %d -> %v", tc.name, tc.expectedResLen, len(rcvdEvts), rcvdEvts)
}
dbgLogger.Debugf("got %d expected messages", len(rcvdEvts))
}
if len(tc.expectedResMessages) != 0 {
res := tc.expectedResMessages
for idx, v := range rcvdEvts {
if len(res) == 0 {
t.Fatalf("result %d/%d : received '%s', didn't expect anything (recvd:%d, expected:%d)", idx, len(rcvdEvts), v.Line.Raw, len(rcvdEvts), len(tc.expectedResMessages))
}
if res[0] != v.Line.Raw {
t.Fatalf("result %d/%d : expected '%s', received '%s' (recvd:%d, expected:%d)", idx, len(rcvdEvts), res[0], v.Line.Raw, len(rcvdEvts), len(tc.expectedResMessages))
}
dbgLogger.Debugf("got message '%s'", res[0])
res = res[1:]
}
if len(res) != 0 {
t.Fatalf("leftover unmatched results : %v", res)
}
}
if tc.teardown != nil {
tc.teardown(t, &cw)
}
})
}
}
func TestConfiguration(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("Skipping test on windows")
}
log.SetLevel(log.DebugLevel)
tests := []struct {
config []byte
expectedCfgErr string
expectedStartErr string
name string
}{
{
name: "group_does_not_exists",
config: []byte(`
source: cloudwatch
aws_region: us-east-1
labels:
type: test_source
group_name: test_group
stream_name: test_stream`),
expectedStartErr: "The specified log group does not exist",
},
{
config: []byte(`
xxx: cloudwatch
labels:
type: test_source
group_name: test_group
stream_name: test_stream`),
expectedCfgErr: "field xxx not found in type",
},
{
name: "missing_group_name",
config: []byte(`
source: cloudwatch
aws_region: us-east-1
labels:
type: test_source
stream_name: test_stream`),
expectedCfgErr: "group_name is mandatory for CloudwatchSource",
},
}
for _, tc := range tests {
tc := tc
t.Run(tc.name, func(t *testing.T) {
dbgLogger := log.New().WithField("test", tc.name)
dbgLogger.Logger.SetLevel(log.DebugLevel)
cw := CloudwatchSource{}
err := cw.Configure(tc.config, dbgLogger)
cstest.RequireErrorContains(t, err, tc.expectedCfgErr)
if tc.expectedCfgErr != "" {
return
}
out := make(chan types.Event)
tmb := tomb.Tomb{}
switch cw.GetMode() {
case "tail":
err = cw.StreamingAcquisition(out, &tmb)
case "cat":
err = cw.OneShotAcquisition(out, &tmb)
}
cstest.RequireErrorContains(t, err, tc.expectedStartErr)
log.Debugf("killing ...")
tmb.Kill(nil)
<-tmb.Dead()
log.Debugf("dead :)")
})
}
}
func TestConfigureByDSN(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("Skipping test on windows")
}
log.SetLevel(log.DebugLevel)
tests := []struct {
dsn string
labels map[string]string
expectedCfgErr string
name string
}{
{
name: "missing_query",
dsn: "cloudwatch://bad_log_group:bad_stream_name",
expectedCfgErr: "query is mandatory (at least start_date and end_date or backlog)",
},
{
name: "backlog",
dsn: "cloudwatch://bad_log_group:bad_stream_name?backlog=30m&log_level=info&profile=test",
// expectedCfgErr: "query is mandatory (at least start_date and end_date or backlog)",
},
{
name: "start_date/end_date",
dsn: "cloudwatch://bad_log_group:bad_stream_name?start_date=2021/05/15 14:04&end_date=2021/05/15 15:04",
// expectedCfgErr: "query is mandatory (at least start_date and end_date or backlog)",
},
{
name: "bad_log_level",
dsn: "cloudwatch://bad_log_group:bad_stream_name?backlog=4h&log_level=",
expectedCfgErr: "unknown level : not a valid logrus Level: ",
},
}
for _, tc := range tests {
tc := tc
t.Run(tc.name, func(t *testing.T) {
dbgLogger := log.New().WithField("test", tc.name)
dbgLogger.Logger.SetLevel(log.DebugLevel)
cw := CloudwatchSource{}
err := cw.ConfigureByDSN(tc.dsn, tc.labels, dbgLogger)
cstest.RequireErrorContains(t, err, tc.expectedCfgErr)
})
}
}
func TestOneShotAcquisition(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("Skipping test on windows")
}
log.SetLevel(log.DebugLevel)
tests := []struct {
dsn string
expectedCfgErr string
expectedStartErr string
name string
setup func(*testing.T, *CloudwatchSource)
run func(*testing.T, *CloudwatchSource)
teardown func(*testing.T, *CloudwatchSource)
expectedResLen int
expectedResMessages []string
}{
// stream with no data
{
name: "empty_stream",
dsn: "cloudwatch://test_log_group1:test_stream?backlog=1h",
// expectedStartErr: "The specified log group does not exist",
setup: func(t *testing.T, cw *CloudwatchSource) {
deleteAllLogGroups(t, cw)
_, err := cw.cwClient.CreateLogGroup(&cloudwatchlogs.CreateLogGroupInput{
LogGroupName: aws.String("test_log_group1"),
})
require.NoError(t, err)
_, err = cw.cwClient.CreateLogStream(&cloudwatchlogs.CreateLogStreamInput{
LogGroupName: aws.String("test_log_group1"),
LogStreamName: aws.String("test_stream"),
})
require.NoError(t, err)
},
teardown: func(t *testing.T, cw *CloudwatchSource) {
_, err := cw.cwClient.DeleteLogGroup(&cloudwatchlogs.DeleteLogGroupInput{
LogGroupName: aws.String("test_log_group1"),
})
require.NoError(t, err)
},
expectedResLen: 0,
},
// stream with one event
{
name: "get_one_event",
dsn: "cloudwatch://test_log_group1:test_stream?backlog=1h",
// expectedStartErr: "The specified log group does not exist",
setup: func(t *testing.T, cw *CloudwatchSource) {
deleteAllLogGroups(t, cw)
_, err := cw.cwClient.CreateLogGroup(&cloudwatchlogs.CreateLogGroupInput{
LogGroupName: aws.String("test_log_group1"),
})
require.NoError(t, err)
_, err = cw.cwClient.CreateLogStream(&cloudwatchlogs.CreateLogStreamInput{
LogGroupName: aws.String("test_log_group1"),
LogStreamName: aws.String("test_stream"),
})
require.NoError(t, err)
// this one is too much in the back
_, err = cw.cwClient.PutLogEvents(&cloudwatchlogs.PutLogEventsInput{
LogGroupName: aws.String("test_log_group1"),
LogStreamName: aws.String("test_stream"),
LogEvents: []*cloudwatchlogs.InputLogEvent{
{
Message: aws.String("test_message_1"),
Timestamp: aws.Int64(time.Now().UTC().Add(-(2 * time.Hour)).UTC().Unix() * 1000),
},
},
})
require.NoError(t, err)
// this one can be read
_, err = cw.cwClient.PutLogEvents(&cloudwatchlogs.PutLogEventsInput{
LogGroupName: aws.String("test_log_group1"),
LogStreamName: aws.String("test_stream"),
LogEvents: []*cloudwatchlogs.InputLogEvent{
{
Message: aws.String("test_message_2"),
Timestamp: aws.Int64(time.Now().UTC().Unix() * 1000),
},
},
})
require.NoError(t, err)
// this one is in the past
_, err = cw.cwClient.PutLogEvents(&cloudwatchlogs.PutLogEventsInput{
LogGroupName: aws.String("test_log_group1"),
LogStreamName: aws.String("test_stream"),
LogEvents: []*cloudwatchlogs.InputLogEvent{
{
Message: aws.String("test_message_3"),
Timestamp: aws.Int64(time.Now().UTC().Add(-(3 * time.Hour)).UTC().Unix() * 1000),
},
},
})
require.NoError(t, err)
},
teardown: func(t *testing.T, cw *CloudwatchSource) {
_, err := cw.cwClient.DeleteLogGroup(&cloudwatchlogs.DeleteLogGroupInput{
LogGroupName: aws.String("test_log_group1"),
})
require.NoError(t, err)
},
expectedResLen: 1,
expectedResMessages: []string{"test_message_2"},
},
}
for _, tc := range tests {
tc := tc
t.Run(tc.name, func(t *testing.T) {
dbgLogger := log.New().WithField("test", tc.name)
dbgLogger.Logger.SetLevel(log.DebugLevel)
dbgLogger.Infof("starting test")
cw := CloudwatchSource{}
err := cw.ConfigureByDSN(tc.dsn, map[string]string{"type": "test"}, dbgLogger)
cstest.RequireErrorContains(t, err, tc.expectedCfgErr)
if tc.expectedCfgErr != "" {
return
}
dbgLogger.Infof("config done test")
// run pre-routine : tests use it to set group & streams etc.
if tc.setup != nil {
tc.setup(t, &cw)
}
out := make(chan types.Event, 100)
tmb := tomb.Tomb{}
var rcvdEvts []types.Event
dbgLogger.Infof("running StreamingAcquisition")
err = cw.OneShotAcquisition(out, &tmb)
dbgLogger.Infof("acquis done")
cstest.RequireErrorContains(t, err, tc.expectedStartErr)
close(out)
// let's empty output chan
for evt := range out {
rcvdEvts = append(rcvdEvts, evt)
}
if tc.run != nil {
tc.run(t, &cw)
} else {
dbgLogger.Warning("no code to run")
}
if tc.expectedResLen != -1 {
if tc.expectedResLen != len(rcvdEvts) {
t.Fatalf("%s : expected %d results got %d -> %v", tc.name, tc.expectedResLen, len(rcvdEvts), rcvdEvts)
} else {
dbgLogger.Debugf("got %d expected messages", len(rcvdEvts))
}
}
if len(tc.expectedResMessages) != 0 {
res := tc.expectedResMessages
for idx, v := range rcvdEvts {
if len(res) == 0 {
t.Fatalf("result %d/%d : received '%s', didn't expect anything (recvd:%d, expected:%d)", idx, len(rcvdEvts), v.Line.Raw, len(rcvdEvts), len(tc.expectedResMessages))
}
if res[0] != v.Line.Raw {
t.Fatalf("result %d/%d : expected '%s', received '%s' (recvd:%d, expected:%d)", idx, len(rcvdEvts), res[0], v.Line.Raw, len(rcvdEvts), len(tc.expectedResMessages))
}
dbgLogger.Debugf("got message '%s'", res[0])
res = res[1:]
}
if len(res) != 0 {
t.Fatalf("leftover unmatched results : %v", res)
}
}
if tc.teardown != nil {
tc.teardown(t, &cw)
}
})
}
}
crowdsec-1.4.6/pkg/acquisition/modules/docker/ 0000775 0000000 0000000 00000000000 14371172257 0021374 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/pkg/acquisition/modules/docker/docker.go 0000664 0000000 0000000 00000043067 14371172257 0023204 0 ustar 00root root 0000000 0000000 package dockeracquisition
import (
"bufio"
"context"
"fmt"
"net/url"
"regexp"
"strconv"
"strings"
"time"
"github.com/crowdsecurity/crowdsec/pkg/acquisition/configuration"
leaky "github.com/crowdsecurity/crowdsec/pkg/leakybucket"
"github.com/crowdsecurity/crowdsec/pkg/types"
"github.com/crowdsecurity/dlog"
dockerTypes "github.com/docker/docker/api/types"
"github.com/docker/docker/client"
"github.com/pkg/errors"
"github.com/prometheus/client_golang/prometheus"
log "github.com/sirupsen/logrus"
"gopkg.in/tomb.v2"
"gopkg.in/yaml.v2"
)
var linesRead = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "cs_dockersource_hits_total",
Help: "Total lines that were read.",
},
[]string{"source"})
type DockerConfiguration struct {
CheckInterval string `yaml:"check_interval"`
FollowStdout bool `yaml:"follow_stdout"`
FollowStdErr bool `yaml:"follow_stderr"`
Until string `yaml:"until"`
Since string `yaml:"since"`
DockerHost string `yaml:"docker_host"`
ContainerName []string `yaml:"container_name"`
ContainerID []string `yaml:"container_id"`
ContainerNameRegexp []string `yaml:"container_name_regexp"`
ContainerIDRegexp []string `yaml:"container_id_regexp"`
ForceInotify bool `yaml:"force_inotify"`
configuration.DataSourceCommonCfg `yaml:",inline"`
}
type DockerSource struct {
Config DockerConfiguration
runningContainerState map[string]*ContainerConfig
compiledContainerName []*regexp.Regexp
compiledContainerID []*regexp.Regexp
CheckIntervalDuration time.Duration
logger *log.Entry
Client client.CommonAPIClient
t *tomb.Tomb
containerLogsOptions *dockerTypes.ContainerLogsOptions
}
type ContainerConfig struct {
Name string
ID string
t *tomb.Tomb
logger *log.Entry
Labels map[string]string
Tty bool
}
func (d *DockerSource) Configure(Config []byte, logger *log.Entry) error {
var err error
d.Config = DockerConfiguration{
FollowStdout: true, // default
FollowStdErr: true, // default
CheckInterval: "1s", // default
}
d.logger = logger
d.runningContainerState = make(map[string]*ContainerConfig)
err = yaml.UnmarshalStrict(Config, &d.Config)
if err != nil {
return errors.Wrap(err, "Cannot parse DockerAcquisition configuration")
}
d.logger.Tracef("DockerAcquisition configuration: %+v", d.Config)
if len(d.Config.ContainerName) == 0 && len(d.Config.ContainerID) == 0 && len(d.Config.ContainerIDRegexp) == 0 && len(d.Config.ContainerNameRegexp) == 0 {
return fmt.Errorf("no containers names or containers ID configuration provided")
}
d.CheckIntervalDuration, err = time.ParseDuration(d.Config.CheckInterval)
if err != nil {
return fmt.Errorf("parsing 'check_interval' parameters: %s", d.CheckIntervalDuration)
}
if d.Config.Mode == "" {
d.Config.Mode = configuration.TAIL_MODE
}
if d.Config.Mode != configuration.CAT_MODE && d.Config.Mode != configuration.TAIL_MODE {
return fmt.Errorf("unsupported mode %s for docker datasource", d.Config.Mode)
}
d.logger.Tracef("Actual DockerAcquisition configuration %+v", d.Config)
for _, cont := range d.Config.ContainerNameRegexp {
d.compiledContainerName = append(d.compiledContainerName, regexp.MustCompile(cont))
}
for _, cont := range d.Config.ContainerIDRegexp {
d.compiledContainerID = append(d.compiledContainerID, regexp.MustCompile(cont))
}
dockerClient, err := client.NewClientWithOpts(client.FromEnv, client.WithAPIVersionNegotiation())
if err != nil {
return err
}
if d.Config.Since == "" {
d.Config.Since = time.Now().UTC().Format(time.RFC3339)
}
d.containerLogsOptions = &dockerTypes.ContainerLogsOptions{
ShowStdout: d.Config.FollowStdout,
ShowStderr: d.Config.FollowStdErr,
Follow: true,
Since: d.Config.Since,
}
if d.Config.Until != "" {
d.containerLogsOptions.Until = d.Config.Until
}
if d.Config.DockerHost != "" {
if err := client.WithHost(d.Config.DockerHost)(dockerClient); err != nil {
return err
}
}
d.Client = dockerClient
_, err = d.Client.Info(context.Background())
if err != nil {
return errors.Wrapf(err, "failed to configure docker datasource %s", d.Config.DockerHost)
}
return nil
}
func (d *DockerSource) ConfigureByDSN(dsn string, labels map[string]string, logger *log.Entry) error {
var err error
if !strings.HasPrefix(dsn, d.GetName()+"://") {
return fmt.Errorf("invalid DSN %s for docker source, must start with %s://", dsn, d.GetName())
}
d.Config = DockerConfiguration{
FollowStdout: true,
FollowStdErr: true,
CheckInterval: "1s",
}
d.Config.ContainerName = make([]string, 0)
d.Config.ContainerID = make([]string, 0)
d.runningContainerState = make(map[string]*ContainerConfig)
d.Config.Mode = configuration.CAT_MODE
d.logger = logger
d.Config.Labels = labels
dockerClient, err := client.NewClientWithOpts(client.FromEnv, client.WithAPIVersionNegotiation())
if err != nil {
return err
}
d.containerLogsOptions = &dockerTypes.ContainerLogsOptions{
ShowStdout: d.Config.FollowStdout,
ShowStderr: d.Config.FollowStdErr,
Follow: false,
}
dsn = strings.TrimPrefix(dsn, d.GetName()+"://")
args := strings.Split(dsn, "?")
if len(args) == 0 {
return fmt.Errorf("invalid dsn: %s", dsn)
}
if len(args) == 1 && args[0] == "" {
return fmt.Errorf("empty %s DSN", d.GetName()+"://")
}
d.Config.ContainerName = append(d.Config.ContainerName, args[0])
// we add it as an ID also so user can provide docker name or docker ID
d.Config.ContainerID = append(d.Config.ContainerID, args[0])
// no parameters
if len(args) == 1 {
d.Client = dockerClient
return nil
}
parameters, err := url.ParseQuery(args[1])
if err != nil {
return errors.Wrapf(err, "while parsing parameters %s: %s", dsn, err)
}
for k, v := range parameters {
switch k {
case "log_level":
if len(v) != 1 {
return fmt.Errorf("only one 'log_level' parameters is required, not many")
}
lvl, err := log.ParseLevel(v[0])
if err != nil {
return errors.Wrapf(err, "unknown level %s", v[0])
}
d.logger.Logger.SetLevel(lvl)
case "until":
if len(v) != 1 {
return fmt.Errorf("only one 'until' parameters is required, not many")
}
d.containerLogsOptions.Until = v[0]
case "since":
if len(v) != 1 {
return fmt.Errorf("only one 'since' parameters is required, not many")
}
d.containerLogsOptions.Since = v[0]
case "follow_stdout":
if len(v) != 1 {
return fmt.Errorf("only one 'follow_stdout' parameters is required, not many")
}
followStdout, err := strconv.ParseBool(v[0])
if err != nil {
return fmt.Errorf("parsing 'follow_stdout' parameters: %s", err)
}
d.Config.FollowStdout = followStdout
d.containerLogsOptions.ShowStdout = followStdout
case "follow_stderr":
if len(v) != 1 {
return fmt.Errorf("only one 'follow_stderr' parameters is required, not many")
}
followStdErr, err := strconv.ParseBool(v[0])
if err != nil {
return fmt.Errorf("parsing 'follow_stderr' parameters: %s", err)
}
d.Config.FollowStdErr = followStdErr
d.containerLogsOptions.ShowStderr = followStdErr
case "docker_host":
if len(v) != 1 {
return fmt.Errorf("only one 'docker_host' parameters is required, not many")
}
if err := client.WithHost(v[0])(dockerClient); err != nil {
return err
}
}
}
d.Client = dockerClient
return nil
}
func (d *DockerSource) GetMode() string {
return d.Config.Mode
}
//SupportedModes returns the supported modes by the acquisition module
func (d *DockerSource) SupportedModes() []string {
return []string{configuration.TAIL_MODE, configuration.CAT_MODE}
}
//OneShotAcquisition reads a set of file and returns when done
func (d *DockerSource) OneShotAcquisition(out chan types.Event, t *tomb.Tomb) error {
d.logger.Debug("In oneshot")
runningContainer, err := d.Client.ContainerList(context.Background(), dockerTypes.ContainerListOptions{})
if err != nil {
return err
}
foundOne := false
for _, container := range runningContainer {
if _, ok := d.runningContainerState[container.ID]; ok {
d.logger.Debugf("container with id %s is already being read from", container.ID)
continue
}
if containerConfig, ok := d.EvalContainer(container); ok {
d.logger.Infof("reading logs from container %s", containerConfig.Name)
d.logger.Debugf("logs options: %+v", *d.containerLogsOptions)
dockerReader, err := d.Client.ContainerLogs(context.Background(), containerConfig.ID, *d.containerLogsOptions)
if err != nil {
d.logger.Errorf("unable to read logs from container: %+v", err)
return err
}
// we use this library to normalize docker API logs (cf. https://ahmet.im/blog/docker-logs-api-binary-format-explained/)
foundOne = true
var scanner *bufio.Scanner
if containerConfig.Tty {
scanner = bufio.NewScanner(dockerReader)
} else {
reader := dlog.NewReader(dockerReader)
scanner = bufio.NewScanner(reader)
}
for scanner.Scan() {
line := scanner.Text()
if line == "" {
continue
}
l := types.Line{}
l.Raw = line
l.Labels = d.Config.Labels
l.Time = time.Now().UTC()
l.Src = containerConfig.Name
l.Process = true
l.Module = d.GetName()
linesRead.With(prometheus.Labels{"source": containerConfig.Name}).Inc()
evt := types.Event{Line: l, Process: true, Type: types.LOG, ExpectMode: leaky.TIMEMACHINE}
out <- evt
d.logger.Debugf("Sent line to parsing: %+v", evt.Line.Raw)
}
err = scanner.Err()
if err != nil {
d.logger.Errorf("Got error from docker read: %s", err)
}
d.runningContainerState[container.ID] = containerConfig
}
}
t.Kill(nil)
if !foundOne {
return fmt.Errorf("no container found named: %s, can't run one shot acquisition", d.Config.ContainerName[0])
}
return nil
}
func (d *DockerSource) GetMetrics() []prometheus.Collector {
return []prometheus.Collector{linesRead}
}
func (d *DockerSource) GetAggregMetrics() []prometheus.Collector {
return []prometheus.Collector{linesRead}
}
func (d *DockerSource) GetName() string {
return "docker"
}
func (d *DockerSource) CanRun() error {
return nil
}
func (d *DockerSource) getContainerTTY(containerId string) bool {
containerDetails, err := d.Client.ContainerInspect(context.Background(), containerId)
if err != nil {
return false
}
return containerDetails.Config.Tty
}
func (d *DockerSource) EvalContainer(container dockerTypes.Container) (*ContainerConfig, bool) {
for _, containerID := range d.Config.ContainerID {
if containerID == container.ID {
return &ContainerConfig{ID: container.ID, Name: container.Names[0], Labels: d.Config.Labels, Tty: d.getContainerTTY(container.ID)}, true
}
}
for _, containerName := range d.Config.ContainerName {
for _, name := range container.Names {
if strings.HasPrefix(name, "/") && len(name) > 0 {
name = name[1:]
}
if name == containerName {
return &ContainerConfig{ID: container.ID, Name: name, Labels: d.Config.Labels, Tty: d.getContainerTTY(container.ID)}, true
}
}
}
for _, cont := range d.compiledContainerID {
if matched := cont.Match([]byte(container.ID)); matched {
return &ContainerConfig{ID: container.ID, Name: container.Names[0], Labels: d.Config.Labels, Tty: d.getContainerTTY(container.ID)}, true
}
}
for _, cont := range d.compiledContainerName {
for _, name := range container.Names {
if matched := cont.Match([]byte(name)); matched {
return &ContainerConfig{ID: container.ID, Name: name, Labels: d.Config.Labels, Tty: d.getContainerTTY(container.ID)}, true
}
}
}
return &ContainerConfig{}, false
}
func (d *DockerSource) WatchContainer(monitChan chan *ContainerConfig, deleteChan chan *ContainerConfig) error {
ticker := time.NewTicker(d.CheckIntervalDuration)
d.logger.Infof("Container watcher started, interval: %s", d.CheckIntervalDuration.String())
for {
select {
case <-d.t.Dying():
d.logger.Infof("stopping container watcher")
return nil
case <-ticker.C:
// to track for garbage collection
runningContainersID := make(map[string]bool)
runningContainer, err := d.Client.ContainerList(context.Background(), dockerTypes.ContainerListOptions{})
if err != nil {
if strings.Contains(strings.ToLower(err.Error()), "cannot connect to the docker daemon at") {
for idx, container := range d.runningContainerState {
if d.runningContainerState[idx].t.Alive() {
d.logger.Infof("killing tail for container %s", container.Name)
d.runningContainerState[idx].t.Kill(nil)
if err := d.runningContainerState[idx].t.Wait(); err != nil {
d.logger.Infof("error while waiting for death of %s : %s", container.Name, err)
}
}
delete(d.runningContainerState, idx)
}
} else {
log.Errorf("container list err: %s", err)
}
continue
}
for _, container := range runningContainer {
runningContainersID[container.ID] = true
// don't need to re eval an already monitored container
if _, ok := d.runningContainerState[container.ID]; ok {
continue
}
if containerConfig, ok := d.EvalContainer(container); ok {
monitChan <- containerConfig
}
}
for containerStateID, containerConfig := range d.runningContainerState {
if _, ok := runningContainersID[containerStateID]; !ok {
deleteChan <- containerConfig
}
}
d.logger.Tracef("Reading logs from %d containers", len(d.runningContainerState))
ticker.Reset(d.CheckIntervalDuration)
}
}
}
func (d *DockerSource) StreamingAcquisition(out chan types.Event, t *tomb.Tomb) error {
d.t = t
monitChan := make(chan *ContainerConfig)
deleteChan := make(chan *ContainerConfig)
d.logger.Infof("Starting docker acquisition")
t.Go(func() error {
return d.DockerManager(monitChan, deleteChan, out)
})
return d.WatchContainer(monitChan, deleteChan)
}
func (d *DockerSource) Dump() interface{} {
return d
}
func ReadTailScanner(scanner *bufio.Scanner, out chan string, t *tomb.Tomb) error {
for scanner.Scan() {
out <- scanner.Text()
}
return scanner.Err()
}
func (d *DockerSource) TailDocker(container *ContainerConfig, outChan chan types.Event, deleteChan chan *ContainerConfig) error {
container.logger.Infof("start tail for container %s", container.Name)
dockerReader, err := d.Client.ContainerLogs(context.Background(), container.ID, *d.containerLogsOptions)
if err != nil {
container.logger.Errorf("unable to read logs from container: %+v", err)
return err
}
var scanner *bufio.Scanner
// we use this library to normalize docker API logs (cf. https://ahmet.im/blog/docker-logs-api-binary-format-explained/)
if container.Tty {
scanner = bufio.NewScanner(dockerReader)
} else {
reader := dlog.NewReader(dockerReader)
scanner = bufio.NewScanner(reader)
}
readerChan := make(chan string)
readerTomb := &tomb.Tomb{}
readerTomb.Go(func() error {
return ReadTailScanner(scanner, readerChan, readerTomb)
})
for {
select {
case <-container.t.Dying():
readerTomb.Kill(nil)
container.logger.Infof("tail stopped for container %s", container.Name)
return nil
case line := <-readerChan:
if line == "" {
continue
}
l := types.Line{}
l.Raw = line
l.Labels = d.Config.Labels
l.Time = time.Now().UTC()
l.Src = container.Name
l.Process = true
l.Module = d.GetName()
var evt types.Event
if !d.Config.UseTimeMachine {
evt = types.Event{Line: l, Process: true, Type: types.LOG, ExpectMode: leaky.LIVE}
} else {
evt = types.Event{Line: l, Process: true, Type: types.LOG, ExpectMode: leaky.TIMEMACHINE}
}
linesRead.With(prometheus.Labels{"source": container.Name}).Inc()
outChan <- evt
d.logger.Debugf("Sent line to parsing: %+v", evt.Line.Raw)
case <-readerTomb.Dying():
//This case is to handle temporarily losing the connection to the docker socket
//The only known case currently is when using docker-socket-proxy (and maybe a docker daemon restart)
d.logger.Debugf("readerTomb dying for container %s, removing it from runningContainerState", container.Name)
deleteChan <- container
//Also reset the Since to avoid re-reading logs
d.Config.Since = time.Now().UTC().Format(time.RFC3339)
d.containerLogsOptions.Since = d.Config.Since
return nil
}
}
}
func (d *DockerSource) DockerManager(in chan *ContainerConfig, deleteChan chan *ContainerConfig, outChan chan types.Event) error {
d.logger.Info("DockerSource Manager started")
for {
select {
case newContainer := <-in:
if _, ok := d.runningContainerState[newContainer.ID]; !ok {
newContainer.t = &tomb.Tomb{}
newContainer.logger = d.logger.WithFields(log.Fields{"container_name": newContainer.Name})
newContainer.t.Go(func() error {
return d.TailDocker(newContainer, outChan, deleteChan)
})
d.runningContainerState[newContainer.ID] = newContainer
}
case containerToDelete := <-deleteChan:
if containerConfig, ok := d.runningContainerState[containerToDelete.ID]; ok {
log.Infof("container acquisition stopped for container '%s'", containerConfig.Name)
containerConfig.t.Kill(nil)
delete(d.runningContainerState, containerToDelete.ID)
}
case <-d.t.Dying():
for idx, container := range d.runningContainerState {
if d.runningContainerState[idx].t.Alive() {
d.logger.Infof("killing tail for container %s", container.Name)
d.runningContainerState[idx].t.Kill(nil)
if err := d.runningContainerState[idx].t.Wait(); err != nil {
d.logger.Infof("error while waiting for death of %s : %s", container.Name, err)
}
}
}
d.runningContainerState = nil
d.logger.Debugf("routine cleanup done, return")
return nil
}
}
}
crowdsec-1.4.6/pkg/acquisition/modules/docker/docker_test.go 0000664 0000000 0000000 00000017002 14371172257 0024231 0 ustar 00root root 0000000 0000000 package dockeracquisition
import (
"context"
"encoding/binary"
"fmt"
"io"
"os"
"runtime"
"strings"
"testing"
"time"
"github.com/crowdsecurity/crowdsec/pkg/cstest"
"github.com/crowdsecurity/crowdsec/pkg/types"
dockerTypes "github.com/docker/docker/api/types"
dockerContainer "github.com/docker/docker/api/types/container"
"github.com/docker/docker/client"
log "github.com/sirupsen/logrus"
"gopkg.in/tomb.v2"
"github.com/stretchr/testify/assert"
)
const testContainerName = "docker_test"
var readLogs = false
func TestConfigure(t *testing.T) {
log.Infof("Test 'TestConfigure'")
tests := []struct {
config string
expectedErr string
}{
{
config: `foobar: asd`,
expectedErr: "line 1: field foobar not found in type dockeracquisition.DockerConfiguration",
},
{
config: `
mode: tail
source: docker`,
expectedErr: "no containers names or containers ID configuration provided",
},
{
config: `
mode: cat
source: docker
container_name:
- toto`,
expectedErr: "",
},
}
subLogger := log.WithFields(log.Fields{
"type": "docker",
})
for _, test := range tests {
f := DockerSource{}
err := f.Configure([]byte(test.config), subLogger)
cstest.AssertErrorContains(t, err, test.expectedErr)
}
}
func TestConfigureDSN(t *testing.T) {
log.Infof("Test 'TestConfigureDSN'")
var dockerHost string
if runtime.GOOS == "windows" {
dockerHost = "npipe:////./pipe/docker_engine"
} else {
dockerHost = "unix:///var/run/podman/podman.sock"
}
tests := []struct {
name string
dsn string
expectedErr string
}{
{
name: "invalid DSN",
dsn: "asd://",
expectedErr: "invalid DSN asd:// for docker source, must start with docker://",
},
{
name: "empty DSN",
dsn: "docker://",
expectedErr: "empty docker:// DSN",
},
{
name: "DSN ok with log_level",
dsn: "docker://test_docker?log_level=warn",
expectedErr: "",
},
{
name: "DSN invalid log_level",
dsn: "docker://test_docker?log_level=foobar",
expectedErr: "unknown level foobar: not a valid logrus Level:",
},
{
name: "DSN ok with multiple parameters",
dsn: fmt.Sprintf("docker://test_docker?since=42min&docker_host=%s", dockerHost),
expectedErr: "",
},
}
subLogger := log.WithFields(log.Fields{
"type": "docker",
})
for _, test := range tests {
f := DockerSource{}
err := f.ConfigureByDSN(test.dsn, map[string]string{"type": "testtype"}, subLogger)
cstest.AssertErrorContains(t, err, test.expectedErr)
}
}
type mockDockerCli struct {
client.Client
}
func TestStreamingAcquisition(t *testing.T) {
log.SetOutput(os.Stdout)
log.SetLevel(log.InfoLevel)
log.Info("Test 'TestStreamingAcquisition'")
tests := []struct {
config string
expectedErr string
expectedOutput string
expectedLines int
logType string
logLevel log.Level
}{
{
config: `
source: docker
mode: cat
container_name:
- docker_test`,
expectedErr: "",
expectedOutput: "",
expectedLines: 3,
logType: "test",
logLevel: log.InfoLevel,
},
{
config: `
source: docker
mode: cat
container_name_regexp:
- docker_*`,
expectedErr: "",
expectedOutput: "",
expectedLines: 3,
logType: "test",
logLevel: log.InfoLevel,
},
}
for _, ts := range tests {
var logger *log.Logger
var subLogger *log.Entry
if ts.expectedOutput != "" {
logger.SetLevel(ts.logLevel)
subLogger = logger.WithFields(log.Fields{
"type": "docker",
})
} else {
subLogger = log.WithFields(log.Fields{
"type": "docker",
})
}
readLogs = false
dockerTomb := tomb.Tomb{}
out := make(chan types.Event)
dockerSource := DockerSource{}
err := dockerSource.Configure([]byte(ts.config), subLogger)
if err != nil {
t.Fatalf("Unexpected error : %s", err)
}
dockerSource.Client = new(mockDockerCli)
actualLines := 0
readerTomb := &tomb.Tomb{}
streamTomb := tomb.Tomb{}
streamTomb.Go(func() error {
return dockerSource.StreamingAcquisition(out, &dockerTomb)
})
readerTomb.Go(func() error {
time.Sleep(1 * time.Second)
ticker := time.NewTicker(1 * time.Second)
for {
select {
case <-out:
actualLines++
ticker.Reset(1 * time.Second)
case <-ticker.C:
log.Infof("no more line to read")
dockerSource.t.Kill(nil)
return nil
}
}
})
cstest.AssertErrorContains(t, err, ts.expectedErr)
if err := readerTomb.Wait(); err != nil {
t.Fatal(err)
}
if ts.expectedLines != 0 {
assert.Equal(t, ts.expectedLines, actualLines)
}
err = streamTomb.Wait()
if err != nil {
t.Fatalf("docker acquisition error: %s", err)
}
}
}
func (cli *mockDockerCli) ContainerList(ctx context.Context, options dockerTypes.ContainerListOptions) ([]dockerTypes.Container, error) {
if readLogs == true {
return []dockerTypes.Container{}, nil
}
containers := make([]dockerTypes.Container, 0)
container := &dockerTypes.Container{
ID: "12456",
Names: []string{testContainerName},
}
containers = append(containers, *container)
return containers, nil
}
func (cli *mockDockerCli) ContainerLogs(ctx context.Context, container string, options dockerTypes.ContainerLogsOptions) (io.ReadCloser, error) {
if readLogs == true {
return io.NopCloser(strings.NewReader("")), nil
}
readLogs = true
data := []string{"docker\n", "test\n", "1234\n"}
ret := ""
for _, line := range data {
startLineByte := make([]byte, 8)
binary.LittleEndian.PutUint32(startLineByte, 1) //stdout stream
binary.BigEndian.PutUint32(startLineByte[4:], uint32(len(line)))
ret += fmt.Sprintf("%s%s", startLineByte, line)
}
r := io.NopCloser(strings.NewReader(ret)) // r type is io.ReadCloser
return r, nil
}
func (cli *mockDockerCli) ContainerInspect(ctx context.Context, c string) (dockerTypes.ContainerJSON, error) {
r := dockerTypes.ContainerJSON{
Config: &dockerContainer.Config{
Tty: false,
},
}
return r, nil
}
func TestOneShot(t *testing.T) {
log.Infof("Test 'TestOneShot'")
tests := []struct {
dsn string
expectedErr string
expectedOutput string
expectedLines int
logType string
logLevel log.Level
}{
{
dsn: "docker://non_exist_docker",
expectedErr: "no container found named: non_exist_docker, can't run one shot acquisition",
expectedOutput: "",
expectedLines: 0,
logType: "test",
logLevel: log.InfoLevel,
},
{
dsn: "docker://" + testContainerName,
expectedErr: "",
expectedOutput: "",
expectedLines: 3,
logType: "test",
logLevel: log.InfoLevel,
},
}
for _, ts := range tests {
var subLogger *log.Entry
var logger *log.Logger
if ts.expectedOutput != "" {
logger.SetLevel(ts.logLevel)
subLogger = logger.WithFields(log.Fields{
"type": "docker",
})
} else {
log.SetLevel(ts.logLevel)
subLogger = log.WithFields(log.Fields{
"type": "docker",
})
}
readLogs = false
dockerClient := &DockerSource{}
labels := make(map[string]string)
labels["type"] = ts.logType
if err := dockerClient.ConfigureByDSN(ts.dsn, labels, subLogger); err != nil {
t.Fatalf("unable to configure dsn '%s': %s", ts.dsn, err)
}
dockerClient.Client = new(mockDockerCli)
out := make(chan types.Event, 100)
tomb := tomb.Tomb{}
err := dockerClient.OneShotAcquisition(out, &tomb)
cstest.AssertErrorContains(t, err, ts.expectedErr)
// else we do the check before actualLines is incremented ...
if ts.expectedLines != 0 {
assert.Equal(t, ts.expectedLines, len(out))
}
}
}
crowdsec-1.4.6/pkg/acquisition/modules/file/ 0000775 0000000 0000000 00000000000 14371172257 0021044 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/pkg/acquisition/modules/file/file.go 0000664 0000000 0000000 00000032700 14371172257 0022314 0 ustar 00root root 0000000 0000000 package fileacquisition
import (
"bufio"
"compress/gzip"
"fmt"
"io"
"net/url"
"os"
"path"
"path/filepath"
"regexp"
"strings"
"time"
"github.com/crowdsecurity/crowdsec/pkg/acquisition/configuration"
leaky "github.com/crowdsecurity/crowdsec/pkg/leakybucket"
"github.com/crowdsecurity/crowdsec/pkg/types"
"github.com/fsnotify/fsnotify"
"github.com/nxadm/tail"
"github.com/pkg/errors"
"github.com/prometheus/client_golang/prometheus"
log "github.com/sirupsen/logrus"
"gopkg.in/tomb.v2"
"gopkg.in/yaml.v2"
)
var linesRead = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "cs_filesource_hits_total",
Help: "Total lines that were read.",
},
[]string{"source"})
type FileConfiguration struct {
Filenames []string
ExcludeRegexps []string `yaml:"exclude_regexps"`
Filename string
ForceInotify bool `yaml:"force_inotify"`
configuration.DataSourceCommonCfg `yaml:",inline"`
}
type FileSource struct {
config FileConfiguration
watcher *fsnotify.Watcher
watchedDirectories map[string]bool
tails map[string]bool
logger *log.Entry
files []string
exclude_regexps []*regexp.Regexp
}
func (f *FileSource) Configure(Config []byte, logger *log.Entry) error {
fileConfig := FileConfiguration{}
f.logger = logger
f.watchedDirectories = make(map[string]bool)
f.tails = make(map[string]bool)
err := yaml.UnmarshalStrict(Config, &fileConfig)
if err != nil {
return errors.Wrap(err, "Cannot parse FileAcquisition configuration")
}
f.logger.Tracef("FileAcquisition configuration: %+v", fileConfig)
if len(fileConfig.Filename) != 0 {
fileConfig.Filenames = append(fileConfig.Filenames, fileConfig.Filename)
}
if len(fileConfig.Filenames) == 0 {
return fmt.Errorf("no filename or filenames configuration provided")
}
f.config = fileConfig
if f.config.Mode == "" {
f.config.Mode = configuration.TAIL_MODE
}
if f.config.Mode != configuration.CAT_MODE && f.config.Mode != configuration.TAIL_MODE {
return fmt.Errorf("unsupported mode %s for file source", f.config.Mode)
}
f.watcher, err = fsnotify.NewWatcher()
if err != nil {
return errors.Wrapf(err, "Could not create fsnotify watcher")
}
for _, exclude := range f.config.ExcludeRegexps {
re, err := regexp.Compile(exclude)
if err != nil {
return errors.Wrapf(err, "Could not compile regexp %s", exclude)
}
f.exclude_regexps = append(f.exclude_regexps, re)
}
f.logger.Tracef("Actual FileAcquisition Configuration %+v", f.config)
for _, pattern := range f.config.Filenames {
if f.config.ForceInotify {
directory := filepath.Dir(pattern)
f.logger.Infof("Force add watch on %s", directory)
if !f.watchedDirectories[directory] {
err = f.watcher.Add(directory)
if err != nil {
f.logger.Errorf("Could not create watch on directory %s : %s", directory, err)
continue
}
f.watchedDirectories[directory] = true
}
}
files, err := filepath.Glob(pattern)
if err != nil {
return errors.Wrap(err, "Glob failure")
}
if len(files) == 0 {
f.logger.Warnf("No matching files for pattern %s", pattern)
continue
}
for _, file := range files {
//check if file is excluded
excluded := false
for _, pattern := range f.exclude_regexps {
if pattern.MatchString(file) {
excluded = true
f.logger.Infof("Skipping file %s as it matches exclude pattern %s", file, pattern)
break
}
}
if excluded {
continue
}
if files[0] != pattern && f.config.Mode == configuration.TAIL_MODE { //we have a glob pattern
directory := filepath.Dir(file)
f.logger.Debugf("Will add watch to directory: %s", directory)
if !f.watchedDirectories[directory] {
err = f.watcher.Add(directory)
if err != nil {
f.logger.Errorf("Could not create watch on directory %s : %s", directory, err)
continue
}
f.watchedDirectories[directory] = true
} else {
f.logger.Debugf("Watch for directory %s already exists", directory)
}
}
f.logger.Infof("Adding file %s to datasources", file)
f.files = append(f.files, file)
}
}
return nil
}
func (f *FileSource) ConfigureByDSN(dsn string, labels map[string]string, logger *log.Entry) error {
if !strings.HasPrefix(dsn, "file://") {
return fmt.Errorf("invalid DSN %s for file source, must start with file://", dsn)
}
f.logger = logger
dsn = strings.TrimPrefix(dsn, "file://")
args := strings.Split(dsn, "?")
if len(args[0]) == 0 {
return fmt.Errorf("empty file:// DSN")
}
if len(args) == 2 && len(args[1]) != 0 {
params, err := url.ParseQuery(args[1])
if err != nil {
return errors.Wrap(err, "could not parse file args")
}
for key, value := range params {
if key != "log_level" {
return fmt.Errorf("unsupported key %s in file DSN", key)
}
if len(value) != 1 {
return errors.New("expected zero or one value for 'log_level'")
}
lvl, err := log.ParseLevel(value[0])
if err != nil {
return errors.Wrapf(err, "unknown level %s", value[0])
}
f.logger.Logger.SetLevel(lvl)
}
}
f.config = FileConfiguration{}
f.config.Labels = labels
f.config.Mode = configuration.CAT_MODE
f.logger.Debugf("Will try pattern %s", args[0])
files, err := filepath.Glob(args[0])
if err != nil {
return errors.Wrap(err, "Glob failure")
}
if len(files) == 0 {
return fmt.Errorf("no matching files for pattern %s", args[0])
}
if len(files) > 1 {
f.logger.Infof("Will read %d files", len(files))
}
for _, file := range files {
f.logger.Infof("Adding file %s to filelist", file)
f.files = append(f.files, file)
}
return nil
}
func (f *FileSource) GetMode() string {
return f.config.Mode
}
//SupportedModes returns the supported modes by the acquisition module
func (f *FileSource) SupportedModes() []string {
return []string{configuration.TAIL_MODE, configuration.CAT_MODE}
}
//OneShotAcquisition reads a set of file and returns when done
func (f *FileSource) OneShotAcquisition(out chan types.Event, t *tomb.Tomb) error {
f.logger.Debug("In oneshot")
for _, file := range f.files {
fi, err := os.Stat(file)
if err != nil {
return fmt.Errorf("could not stat file %s : %w", file, err)
}
if fi.IsDir() {
f.logger.Warnf("%s is a directory, ignoring it.", file)
continue
}
f.logger.Infof("reading %s at once", file)
err = f.readFile(file, out, t)
if err != nil {
return err
}
}
return nil
}
func (f *FileSource) GetMetrics() []prometheus.Collector {
return []prometheus.Collector{linesRead}
}
func (f *FileSource) GetAggregMetrics() []prometheus.Collector {
return []prometheus.Collector{linesRead}
}
func (f *FileSource) GetName() string {
return "file"
}
func (f *FileSource) CanRun() error {
return nil
}
func (f *FileSource) StreamingAcquisition(out chan types.Event, t *tomb.Tomb) error {
f.logger.Debug("Starting live acquisition")
t.Go(func() error {
return f.monitorNewFiles(out, t)
})
for _, file := range f.files {
//before opening the file, check if we need to specifically avoid it. (XXX)
skip := false
for _, pattern := range f.exclude_regexps {
if pattern.MatchString(file) {
f.logger.Infof("file %s matches exclusion pattern %s, skipping", file, pattern.String())
skip = true
break
}
}
if skip {
continue
}
//cf. https://github.com/crowdsecurity/crowdsec/issues/1168
//do not rely on stat, reclose file immediately as it's opened by Tail
fd, err := os.Open(file)
if err != nil {
f.logger.Errorf("unable to read %s : %s", file, err)
continue
}
if err := fd.Close(); err != nil {
f.logger.Errorf("unable to close %s : %s", file, err)
continue
}
fi, err := os.Stat(file)
if err != nil {
return fmt.Errorf("could not stat file %s : %w", file, err)
}
if fi.IsDir() {
f.logger.Warnf("%s is a directory, ignoring it.", file)
continue
}
tail, err := tail.TailFile(file, tail.Config{ReOpen: true, Follow: true, Poll: true, Location: &tail.SeekInfo{Offset: 0, Whence: io.SeekEnd}})
if err != nil {
f.logger.Errorf("Could not start tailing file %s : %s", file, err)
continue
}
f.tails[file] = true
t.Go(func() error {
defer types.CatchPanic("crowdsec/acquis/file/live/fsnotify")
return f.tailFile(out, t, tail)
})
}
return nil
}
func (f *FileSource) Dump() interface{} {
return f
}
func (f *FileSource) monitorNewFiles(out chan types.Event, t *tomb.Tomb) error {
logger := f.logger.WithField("goroutine", "inotify")
for {
select {
case event, ok := <-f.watcher.Events:
if !ok {
return nil
}
if event.Op&fsnotify.Create == fsnotify.Create {
fi, err := os.Stat(event.Name)
if err != nil {
logger.Errorf("Could not stat() new file %s, ignoring it : %s", event.Name, err)
continue
}
if fi.IsDir() {
continue
}
logger.Debugf("Detected new file %s", event.Name)
matched := false
for _, pattern := range f.config.Filenames {
logger.Debugf("Matching %s with %s", pattern, event.Name)
matched, err = path.Match(pattern, event.Name)
if err != nil {
logger.Errorf("Could not match pattern : %s", err)
continue
}
if matched {
break
}
}
if !matched {
continue
}
//before opening the file, check if we need to specifically avoid it. (XXX)
skip := false
for _, pattern := range f.exclude_regexps {
if pattern.MatchString(event.Name) {
f.logger.Infof("file %s matches exclusion pattern %s, skipping", event.Name, pattern.String())
skip = true
break
}
}
if skip {
continue
}
if f.tails[event.Name] {
//we already have a tail on it, do not start a new one
logger.Debugf("Already tailing file %s, not creating a new tail", event.Name)
break
}
//cf. https://github.com/crowdsecurity/crowdsec/issues/1168
//do not rely on stat, reclose file immediately as it's opened by Tail
fd, err := os.Open(event.Name)
if err != nil {
f.logger.Errorf("unable to read %s : %s", event.Name, err)
continue
}
if err := fd.Close(); err != nil {
f.logger.Errorf("unable to close %s : %s", event.Name, err)
continue
}
//Slightly different parameters for Location, as we want to read the first lines of the newly created file
tail, err := tail.TailFile(event.Name, tail.Config{ReOpen: true, Follow: true, Poll: true, Location: &tail.SeekInfo{Offset: 0, Whence: io.SeekStart}})
if err != nil {
logger.Errorf("Could not start tailing file %s : %s", event.Name, err)
break
}
f.tails[event.Name] = true
t.Go(func() error {
defer types.CatchPanic("crowdsec/acquis/tailfile")
return f.tailFile(out, t, tail)
})
}
case err, ok := <-f.watcher.Errors:
if !ok {
return nil
}
logger.Errorf("Error while monitoring folder: %s", err)
case <-t.Dying():
err := f.watcher.Close()
if err != nil {
return errors.Wrapf(err, "could not remove all inotify watches")
}
return nil
}
}
}
func (f *FileSource) tailFile(out chan types.Event, t *tomb.Tomb, tail *tail.Tail) error {
logger := f.logger.WithField("tail", tail.Filename)
logger.Debugf("-> Starting tail of %s", tail.Filename)
for {
select {
case <-t.Dying():
logger.Infof("File datasource %s stopping", tail.Filename)
if err := tail.Stop(); err != nil {
f.logger.Errorf("error in stop : %s", err)
return err
}
return nil
case <-tail.Tomb.Dying(): //our tailer is dying
logger.Warningf("File reader of %s died", tail.Filename)
t.Kill(fmt.Errorf("dead reader for %s", tail.Filename))
return fmt.Errorf("reader for %s is dead", tail.Filename)
case line := <-tail.Lines:
if line == nil {
logger.Warningf("tail for %s is empty", tail.Filename)
continue
}
if line.Err != nil {
logger.Warningf("fetch error : %v", line.Err)
return line.Err
}
if line.Text == "" { //skip empty lines
continue
}
linesRead.With(prometheus.Labels{"source": tail.Filename}).Inc()
l := types.Line{
Raw: trimLine(line.Text),
Labels: f.config.Labels,
Time: line.Time,
Src: tail.Filename,
Process: true,
Module: f.GetName(),
}
//we're tailing, it must be real time logs
logger.Debugf("pushing %+v", l)
expectMode := leaky.LIVE
if f.config.UseTimeMachine {
expectMode = leaky.TIMEMACHINE
}
out <- types.Event{Line: l, Process: true, Type: types.LOG, ExpectMode: expectMode}
}
}
}
func (f *FileSource) readFile(filename string, out chan types.Event, t *tomb.Tomb) error {
var scanner *bufio.Scanner
logger := f.logger.WithField("oneshot", filename)
fd, err := os.Open(filename)
if err != nil {
return errors.Wrapf(err, "failed opening %s", filename)
}
defer fd.Close()
if strings.HasSuffix(filename, ".gz") {
gz, err := gzip.NewReader(fd)
if err != nil {
logger.Errorf("Failed to read gz file: %s", err)
return errors.Wrapf(err, "failed to read gz %s", filename)
}
defer gz.Close()
scanner = bufio.NewScanner(gz)
} else {
scanner = bufio.NewScanner(fd)
}
scanner.Split(bufio.ScanLines)
for scanner.Scan() {
if scanner.Text() == "" {
continue
}
l := types.Line{
Raw: scanner.Text(),
Time: time.Now().UTC(),
Src: filename,
Labels: f.config.Labels,
Process: true,
Module: f.GetName(),
}
logger.Debugf("line %s", l.Raw)
linesRead.With(prometheus.Labels{"source": filename}).Inc()
//we're reading logs at once, it must be time-machine buckets
out <- types.Event{Line: l, Process: true, Type: types.LOG, ExpectMode: leaky.TIMEMACHINE}
}
t.Kill(nil)
return nil
}
crowdsec-1.4.6/pkg/acquisition/modules/file/file_test.go 0000664 0000000 0000000 00000026002 14371172257 0023351 0 ustar 00root root 0000000 0000000 package fileacquisition_test
import (
"fmt"
"os"
"runtime"
"testing"
"time"
fileacquisition "github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/file"
"github.com/crowdsecurity/crowdsec/pkg/cstest"
"github.com/crowdsecurity/crowdsec/pkg/types"
log "github.com/sirupsen/logrus"
"github.com/sirupsen/logrus/hooks/test"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gopkg.in/tomb.v2"
)
func TestBadConfiguration(t *testing.T) {
tests := []struct {
name string
config string
expectedErr string
}{
{
name: "extra configuration key",
config: "foobar: asd.log",
expectedErr: "line 1: field foobar not found in type fileacquisition.FileConfiguration",
},
{
name: "missing filenames",
config: "mode: tail",
expectedErr: "no filename or filenames configuration provided",
},
{
name: "glob syntax error",
config: `filename: "[asd-.log"`,
expectedErr: "Glob failure: syntax error in pattern",
},
{
name: "bad exclude regexp",
config: `filenames: ["asd.log"]
exclude_regexps: ["as[a-$d"]`,
expectedErr: "Could not compile regexp as",
},
}
subLogger := log.WithFields(log.Fields{
"type": "file",
})
for _, tc := range tests {
tc := tc
t.Run(tc.name, func(t *testing.T) {
f := fileacquisition.FileSource{}
err := f.Configure([]byte(tc.config), subLogger)
cstest.RequireErrorContains(t, err, tc.expectedErr)
})
}
}
func TestConfigureDSN(t *testing.T) {
file := "/etc/passwd"
if runtime.GOOS == "windows" {
file = `C:\Windows\System32\drivers\etc\hosts`
}
tests := []struct {
dsn string
expectedErr string
}{
{
dsn: "asd://",
expectedErr: "invalid DSN asd:// for file source, must start with file://",
},
{
dsn: "file://",
expectedErr: "empty file:// DSN",
},
{
dsn: fmt.Sprintf("file://%s?log_level=warn", file),
},
{
dsn: fmt.Sprintf("file://%s?log_level=foobar", file),
expectedErr: "unknown level foobar: not a valid logrus Level:",
},
}
subLogger := log.WithFields(log.Fields{
"type": "file",
})
for _, tc := range tests {
tc := tc
t.Run(tc.dsn, func(t *testing.T) {
f := fileacquisition.FileSource{}
err := f.ConfigureByDSN(tc.dsn, map[string]string{"type": "testtype"}, subLogger)
cstest.RequireErrorContains(t, err, tc.expectedErr)
})
}
}
func TestOneShot(t *testing.T) {
permDeniedFile := "/etc/shadow"
permDeniedError := "failed opening /etc/shadow: open /etc/shadow: permission denied"
if runtime.GOOS == "windows" {
// Technically, this is not a permission denied error, but we just want to test what happens
// if we do not have access to the file
permDeniedFile = `C:\Windows\System32\config\SAM`
permDeniedError = `failed opening C:\Windows\System32\config\SAM: open C:\Windows\System32\config\SAM: The process cannot access the file because it is being used by another process.`
}
tests := []struct {
name string
config string
expectedConfigErr string
expectedErr string
expectedOutput string
expectedLines int
logLevel log.Level
setup func()
afterConfigure func()
teardown func()
}{
{
name: "permission denied",
config: fmt.Sprintf(`
mode: cat
filename: %s`, permDeniedFile),
expectedErr: permDeniedError,
logLevel: log.WarnLevel,
expectedLines: 0,
},
{
name: "ignored directory",
config: `
mode: cat
filename: /`,
expectedOutput: "/ is a directory, ignoring it",
logLevel: log.WarnLevel,
expectedLines: 0,
},
{
name: "glob syntax error",
config: `
mode: cat
filename: "[*-.log"`,
expectedConfigErr: "Glob failure: syntax error in pattern",
logLevel: log.WarnLevel,
expectedLines: 0,
},
{
name: "no matching files",
config: `
mode: cat
filename: /do/not/exist`,
expectedOutput: "No matching files for pattern /do/not/exist",
logLevel: log.WarnLevel,
expectedLines: 0,
},
{
name: "test.log",
config: `
mode: cat
filename: test_files/test.log`,
expectedLines: 5,
logLevel: log.WarnLevel,
},
{
name: "test.log.gz",
config: `
mode: cat
filename: test_files/test.log.gz`,
expectedLines: 5,
logLevel: log.WarnLevel,
},
{
name: "unexpected end of gzip stream",
config: `
mode: cat
filename: test_files/bad.gz`,
expectedErr: "failed to read gz test_files/bad.gz: unexpected EOF",
expectedLines: 0,
logLevel: log.WarnLevel,
},
{
name: "deleted file",
config: `
mode: cat
filename: test_files/test_delete.log`,
setup: func() {
f, _ := os.Create("test_files/test_delete.log")
f.Close()
},
afterConfigure: func() {
os.Remove("test_files/test_delete.log")
},
expectedErr: "could not stat file test_files/test_delete.log",
},
}
for _, tc := range tests {
tc := tc
t.Run(tc.name, func(t *testing.T) {
logger, hook := test.NewNullLogger()
logger.SetLevel(tc.logLevel)
subLogger := logger.WithFields(log.Fields{
"type": "file",
})
tomb := tomb.Tomb{}
out := make(chan types.Event, 100)
f := fileacquisition.FileSource{}
if tc.setup != nil {
tc.setup()
}
err := f.Configure([]byte(tc.config), subLogger)
cstest.RequireErrorContains(t, err, tc.expectedConfigErr)
if tc.expectedConfigErr != "" {
return
}
if tc.afterConfigure != nil {
tc.afterConfigure()
}
err = f.OneShotAcquisition(out, &tomb)
actualLines := len(out)
cstest.RequireErrorContains(t, err, tc.expectedErr)
if tc.expectedLines != 0 {
assert.Equal(t, tc.expectedLines, actualLines)
}
if tc.expectedOutput != "" {
assert.Contains(t, hook.LastEntry().Message, tc.expectedOutput)
hook.Reset()
}
if tc.teardown != nil {
tc.teardown()
}
})
}
}
func TestLiveAcquisition(t *testing.T) {
permDeniedFile := "/etc/shadow"
permDeniedError := "unable to read /etc/shadow : open /etc/shadow: permission denied"
testPattern := "test_files/*.log"
if runtime.GOOS == "windows" {
// Technically, this is not a permission denied error, but we just want to test what happens
// if we do not have access to the file
permDeniedFile = `C:\Windows\System32\config\SAM`
permDeniedError = `unable to read C:\Windows\System32\config\SAM : open C:\Windows\System32\config\SAM: The process cannot access the file because it is being used by another process`
testPattern = `test_files\\*.log` // the \ must be escaped for the yaml config
}
tests := []struct {
name string
config string
expectedErr string
expectedOutput string
expectedLines int
logLevel log.Level
setup func()
afterConfigure func()
teardown func()
}{
{
config: fmt.Sprintf(`
mode: tail
filename: %s`, permDeniedFile),
expectedOutput: permDeniedError,
logLevel: log.InfoLevel,
expectedLines: 0,
name: "PermissionDenied",
},
{
config: `
mode: tail
filename: /`,
expectedOutput: "/ is a directory, ignoring it",
logLevel: log.WarnLevel,
expectedLines: 0,
name: "Directory",
},
{
config: `
mode: tail
filename: /do/not/exist`,
expectedOutput: "No matching files for pattern /do/not/exist",
logLevel: log.WarnLevel,
expectedLines: 0,
name: "badPattern",
},
{
config: fmt.Sprintf(`
mode: tail
filenames:
- %s
force_inotify: true`, testPattern),
expectedLines: 5,
logLevel: log.DebugLevel,
name: "basicGlob",
},
{
config: fmt.Sprintf(`
mode: tail
filenames:
- %s
force_inotify: true`, testPattern),
expectedLines: 0,
logLevel: log.DebugLevel,
name: "GlobInotify",
afterConfigure: func() {
f, _ := os.Create("test_files/a.log")
f.Close()
time.Sleep(1 * time.Second)
os.Remove("test_files/a.log")
},
},
{
config: fmt.Sprintf(`
mode: tail
filenames:
- %s
force_inotify: true`, testPattern),
expectedLines: 5,
logLevel: log.DebugLevel,
name: "GlobInotifyChmod",
afterConfigure: func() {
f, _ := os.Create("test_files/a.log")
f.Close()
time.Sleep(1 * time.Second)
os.Chmod("test_files/a.log", 0o000)
},
teardown: func() {
os.Chmod("test_files/a.log", 0o644)
os.Remove("test_files/a.log")
},
},
{
config: fmt.Sprintf(`
mode: tail
filenames:
- %s
force_inotify: true`, testPattern),
expectedLines: 5,
logLevel: log.DebugLevel,
name: "InotifyMkDir",
afterConfigure: func() {
os.Mkdir("test_files/pouet/", 0o700)
},
teardown: func() {
os.Remove("test_files/pouet/")
},
},
}
for _, tc := range tests {
tc := tc
t.Run(tc.name, func(t *testing.T) {
logger, hook := test.NewNullLogger()
logger.SetLevel(tc.logLevel)
subLogger := logger.WithFields(log.Fields{
"type": "file",
})
tomb := tomb.Tomb{}
out := make(chan types.Event)
f := fileacquisition.FileSource{}
if tc.setup != nil {
tc.setup()
}
err := f.Configure([]byte(tc.config), subLogger)
require.NoError(t, err)
if tc.afterConfigure != nil {
tc.afterConfigure()
}
actualLines := 0
if tc.expectedLines != 0 {
go func() {
for {
select {
case <-out:
actualLines++
case <-time.After(2 * time.Second):
return
}
}
}()
}
err = f.StreamingAcquisition(out, &tomb)
cstest.RequireErrorContains(t, err, tc.expectedErr)
if tc.expectedLines != 0 {
fd, err := os.Create("test_files/stream.log")
if err != nil {
t.Fatalf("could not create test file : %s", err)
}
for i := 0; i < 5; i++ {
_, err = fmt.Fprintf(fd, "%d\n", i)
if err != nil {
t.Fatalf("could not write test file : %s", err)
os.Remove("test_files/stream.log")
}
}
fd.Close()
// we sleep to make sure we detect the new file
time.Sleep(1 * time.Second)
os.Remove("test_files/stream.log")
assert.Equal(t, tc.expectedLines, actualLines)
}
if tc.expectedOutput != "" {
if hook.LastEntry() == nil {
t.Fatalf("expected output %s, but got nothing", tc.expectedOutput)
}
assert.Contains(t, hook.LastEntry().Message, tc.expectedOutput)
hook.Reset()
}
if tc.teardown != nil {
tc.teardown()
}
tomb.Kill(nil)
})
}
}
func TestExclusion(t *testing.T) {
config := `filenames: ["test_files/*.log*"]
exclude_regexps: ["\\.gz$"]`
logger, hook := test.NewNullLogger()
// logger.SetLevel(ts.logLevel)
subLogger := logger.WithFields(log.Fields{
"type": "file",
})
f := fileacquisition.FileSource{}
if err := f.Configure([]byte(config), subLogger); err != nil {
subLogger.Fatalf("unexpected error: %s", err)
}
expectedLogOutput := "Skipping file test_files/test.log.gz as it matches exclude pattern"
if runtime.GOOS == "windows" {
expectedLogOutput = `Skipping file test_files\test.log.gz as it matches exclude pattern \.gz`
}
if hook.LastEntry() == nil {
t.Fatalf("expected output %s, but got nothing", expectedLogOutput)
}
assert.Contains(t, hook.LastEntry().Message, expectedLogOutput)
hook.Reset()
}
crowdsec-1.4.6/pkg/acquisition/modules/file/tailline.go 0000664 0000000 0000000 00000000205 14371172257 0023171 0 ustar 00root root 0000000 0000000 // +build linux freebsd netbsd openbsd solaris !windows
package fileacquisition
func trimLine(text string) string {
return text
}
crowdsec-1.4.6/pkg/acquisition/modules/file/tailline_windows.go 0000664 0000000 0000000 00000000212 14371172257 0024741 0 ustar 00root root 0000000 0000000 // +build windows
package fileacquisition
import "strings"
func trimLine(text string) string {
return strings.TrimRight(text, "\r")
}
crowdsec-1.4.6/pkg/acquisition/modules/file/test_files/ 0000775 0000000 0000000 00000000000 14371172257 0023205 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/pkg/acquisition/modules/file/test_files/bad.gz 0000664 0000000 0000000 00000000003 14371172257 0024266 0 ustar 00root root 0000000 0000000 42
crowdsec-1.4.6/pkg/acquisition/modules/file/test_files/test.log 0000664 0000000 0000000 00000000012 14371172257 0024660 0 ustar 00root root 0000000 0000000 1
2
3
4
5
crowdsec-1.4.6/pkg/acquisition/modules/file/test_files/test.log.gz 0000664 0000000 0000000 00000000047 14371172257 0025307 0 ustar 00root root 0000000 0000000 ņŗ` test.log 3ä2â2æ2á2å Ŧij
crowdsec-1.4.6/pkg/acquisition/modules/journalctl/ 0000775 0000000 0000000 00000000000 14371172257 0022302 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/pkg/acquisition/modules/journalctl/journalctl.go 0000664 0000000 0000000 00000015437 14371172257 0025020 0 ustar 00root root 0000000 0000000 package journalctlacquisition
import (
"bufio"
"context"
"fmt"
"net/url"
"os/exec"
"strings"
"time"
"github.com/crowdsecurity/crowdsec/pkg/acquisition/configuration"
leaky "github.com/crowdsecurity/crowdsec/pkg/leakybucket"
"github.com/crowdsecurity/crowdsec/pkg/types"
"github.com/pkg/errors"
"github.com/prometheus/client_golang/prometheus"
log "github.com/sirupsen/logrus"
"gopkg.in/tomb.v2"
"gopkg.in/yaml.v2"
)
type JournalCtlConfiguration struct {
configuration.DataSourceCommonCfg `yaml:",inline"`
Filters []string `yaml:"journalctl_filter"`
}
type JournalCtlSource struct {
config JournalCtlConfiguration
logger *log.Entry
src string
args []string
}
const journalctlCmd string = "journalctl"
var (
journalctlArgsOneShot = []string{}
journalctlArgstreaming = []string{"--follow", "-n", "0"}
)
var linesRead = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "cs_journalctlsource_hits_total",
Help: "Total lines that were read.",
},
[]string{"source"})
func readLine(scanner *bufio.Scanner, out chan string, errChan chan error) error {
for scanner.Scan() {
txt := scanner.Text()
out <- txt
}
if errChan != nil && scanner.Err() != nil {
errChan <- scanner.Err()
close(errChan)
// the error is already consumed by runJournalCtl
return nil //nolint:nilerr
}
if errChan != nil {
close(errChan)
}
return nil
}
func (j *JournalCtlSource) runJournalCtl(out chan types.Event, t *tomb.Tomb) error {
ctx, cancel := context.WithCancel(context.Background())
cmd := exec.CommandContext(ctx, journalctlCmd, j.args...)
stdout, err := cmd.StdoutPipe()
if err != nil {
cancel()
return fmt.Errorf("could not get journalctl stdout: %s", err)
}
stderr, err := cmd.StderrPipe()
if err != nil {
cancel()
return fmt.Errorf("could not get journalctl stderr: %s", err)
}
stderrChan := make(chan string)
stdoutChan := make(chan string)
errChan := make(chan error, 1)
logger := j.logger.WithField("src", j.src)
logger.Infof("Running journalctl command: %s %s", cmd.Path, cmd.Args)
err = cmd.Start()
if err != nil {
cancel()
logger.Errorf("could not start journalctl command : %s", err)
return err
}
stdoutscanner := bufio.NewScanner(stdout)
if stdoutscanner == nil {
cancel()
cmd.Wait()
return fmt.Errorf("failed to create stdout scanner")
}
stderrScanner := bufio.NewScanner(stderr)
if stderrScanner == nil {
cancel()
cmd.Wait()
return fmt.Errorf("failed to create stderr scanner")
}
t.Go(func() error {
return readLine(stdoutscanner, stdoutChan, errChan)
})
t.Go(func() error {
//looks like journalctl closes stderr quite early, so ignore its status (but not its output)
return readLine(stderrScanner, stderrChan, nil)
})
for {
select {
case <-t.Dying():
logger.Infof("journalctl datasource %s stopping", j.src)
cancel()
cmd.Wait() //avoid zombie process
return nil
case stdoutLine := <-stdoutChan:
l := types.Line{}
l.Raw = stdoutLine
logger.Debugf("getting one line : %s", l.Raw)
l.Labels = j.config.Labels
l.Time = time.Now().UTC()
l.Src = j.src
l.Process = true
l.Module = j.GetName()
linesRead.With(prometheus.Labels{"source": j.src}).Inc()
var evt types.Event
if !j.config.UseTimeMachine {
evt = types.Event{Line: l, Process: true, Type: types.LOG, ExpectMode: leaky.LIVE}
} else {
evt = types.Event{Line: l, Process: true, Type: types.LOG, ExpectMode: leaky.TIMEMACHINE}
}
out <- evt
case stderrLine := <-stderrChan:
logger.Warnf("Got stderr message : %s", stderrLine)
err := fmt.Errorf("journalctl error : %s", stderrLine)
t.Kill(err)
case errScanner, ok := <-errChan:
if !ok {
logger.Debugf("errChan is closed, quitting")
t.Kill(nil)
}
if errScanner != nil {
t.Kill(errScanner)
}
}
}
}
func (j *JournalCtlSource) GetMetrics() []prometheus.Collector {
return []prometheus.Collector{linesRead}
}
func (j *JournalCtlSource) GetAggregMetrics() []prometheus.Collector {
return []prometheus.Collector{linesRead}
}
func (j *JournalCtlSource) Configure(yamlConfig []byte, logger *log.Entry) error {
config := JournalCtlConfiguration{}
j.logger = logger
err := yaml.UnmarshalStrict(yamlConfig, &config)
if err != nil {
return errors.Wrap(err, "Cannot parse JournalCtlSource configuration")
}
if config.Mode == "" {
config.Mode = configuration.TAIL_MODE
}
var args []string
if config.Mode == configuration.TAIL_MODE {
args = journalctlArgstreaming
} else {
args = journalctlArgsOneShot
}
if len(config.Filters) == 0 {
return fmt.Errorf("journalctl_filter is required")
}
j.args = append(args, config.Filters...)
j.src = fmt.Sprintf("journalctl-%s", strings.Join(config.Filters, "."))
j.config = config
return nil
}
func (j *JournalCtlSource) ConfigureByDSN(dsn string, labels map[string]string, logger *log.Entry) error {
j.logger = logger
j.config = JournalCtlConfiguration{}
j.config.Mode = configuration.CAT_MODE
j.config.Labels = labels
//format for the DSN is : journalctl://filters=FILTER1&filters=FILTER2
if !strings.HasPrefix(dsn, "journalctl://") {
return fmt.Errorf("invalid DSN %s for journalctl source, must start with journalctl://", dsn)
}
qs := strings.TrimPrefix(dsn, "journalctl://")
if len(qs) == 0 {
return fmt.Errorf("empty journalctl:// DSN")
}
params, err := url.ParseQuery(qs)
if err != nil {
return fmt.Errorf("could not parse journalctl DSN : %s", err)
}
for key, value := range params {
switch key {
case "filters":
j.config.Filters = append(j.config.Filters, value...)
case "log_level":
if len(value) != 1 {
return fmt.Errorf("expected zero or one value for 'log_level'")
}
lvl, err := log.ParseLevel(value[0])
if err != nil {
return errors.Wrapf(err, "unknown level %s", value[0])
}
j.logger.Logger.SetLevel(lvl)
case "since":
j.args = append(j.args, "--since", value[0])
default:
return fmt.Errorf("unsupported key %s in journalctl DSN", key)
}
}
j.args = append(j.args, j.config.Filters...)
return nil
}
func (j *JournalCtlSource) GetMode() string {
return j.config.Mode
}
func (j *JournalCtlSource) GetName() string {
return "journalctl"
}
func (j *JournalCtlSource) OneShotAcquisition(out chan types.Event, t *tomb.Tomb) error {
defer types.CatchPanic("crowdsec/acquis/journalctl/oneshot")
err := j.runJournalCtl(out, t)
j.logger.Debug("Oneshot journalctl acquisition is done")
return err
}
func (j *JournalCtlSource) StreamingAcquisition(out chan types.Event, t *tomb.Tomb) error {
t.Go(func() error {
defer types.CatchPanic("crowdsec/acquis/journalctl/streaming")
return j.runJournalCtl(out, t)
})
return nil
}
func (j *JournalCtlSource) CanRun() error {
//TODO: add a more precise check on version or something ?
_, err := exec.LookPath(journalctlCmd)
return err
}
func (j *JournalCtlSource) Dump() interface{} {
return j
}
crowdsec-1.4.6/pkg/acquisition/modules/journalctl/journalctl_test.go 0000664 0000000 0000000 00000014123 14371172257 0026046 0 ustar 00root root 0000000 0000000 package journalctlacquisition
import (
"os"
"os/exec"
"path/filepath"
"runtime"
"testing"
"time"
"github.com/crowdsecurity/crowdsec/pkg/cstest"
"github.com/crowdsecurity/crowdsec/pkg/types"
log "github.com/sirupsen/logrus"
"github.com/sirupsen/logrus/hooks/test"
"github.com/stretchr/testify/assert"
"gopkg.in/tomb.v2"
)
func TestBadConfiguration(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("Skipping test on windows")
}
tests := []struct {
config string
expectedErr string
}{
{
config: `foobar: asd.log`,
expectedErr: "line 1: field foobar not found in type journalctlacquisition.JournalCtlConfiguration",
},
{
config: `
mode: tail
source: journalctl`,
expectedErr: "journalctl_filter is required",
},
{
config: `
mode: cat
source: journalctl
journalctl_filter:
- _UID=42`,
expectedErr: "",
},
}
subLogger := log.WithFields(log.Fields{
"type": "journalctl",
})
for _, test := range tests {
f := JournalCtlSource{}
err := f.Configure([]byte(test.config), subLogger)
cstest.AssertErrorContains(t, err, test.expectedErr)
}
}
func TestConfigureDSN(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("Skipping test on windows")
}
tests := []struct {
dsn string
expectedErr string
}{
{
dsn: "asd://",
expectedErr: "invalid DSN asd:// for journalctl source, must start with journalctl://",
},
{
dsn: "journalctl://",
expectedErr: "empty journalctl:// DSN",
},
{
dsn: "journalctl://foobar=42",
expectedErr: "unsupported key foobar in journalctl DSN",
},
{
dsn: "journalctl://filters=%ZZ",
expectedErr: "could not parse journalctl DSN : invalid URL escape \"%ZZ\"",
},
{
dsn: "journalctl://filters=_UID=42?log_level=warn",
expectedErr: "",
},
{
dsn: "journalctl://filters=_UID=1000&log_level=foobar",
expectedErr: "unknown level foobar: not a valid logrus Level:",
},
{
dsn: "journalctl://filters=_UID=1000&log_level=warn&since=yesterday",
expectedErr: "",
},
}
subLogger := log.WithFields(log.Fields{
"type": "journalctl",
})
for _, test := range tests {
f := JournalCtlSource{}
err := f.ConfigureByDSN(test.dsn, map[string]string{"type": "testtype"}, subLogger)
cstest.AssertErrorContains(t, err, test.expectedErr)
}
}
func TestOneShot(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("Skipping test on windows")
}
tests := []struct {
config string
expectedErr string
expectedOutput string
expectedLines int
logLevel log.Level
}{
{
config: `
source: journalctl
mode: cat
journalctl_filter:
- "-_UID=42"`,
expectedErr: "",
expectedOutput: "journalctl: invalid option",
logLevel: log.WarnLevel,
expectedLines: 0,
},
{
config: `
source: journalctl
mode: cat
journalctl_filter:
- _SYSTEMD_UNIT=ssh.service`,
expectedErr: "",
expectedOutput: "",
logLevel: log.WarnLevel,
expectedLines: 14,
},
}
for _, ts := range tests {
var logger *log.Logger
var subLogger *log.Entry
var hook *test.Hook
if ts.expectedOutput != "" {
logger, hook = test.NewNullLogger()
logger.SetLevel(ts.logLevel)
subLogger = logger.WithFields(log.Fields{
"type": "journalctl",
})
} else {
subLogger = log.WithFields(log.Fields{
"type": "journalctl",
})
}
tomb := tomb.Tomb{}
out := make(chan types.Event, 100)
j := JournalCtlSource{}
err := j.Configure([]byte(ts.config), subLogger)
if err != nil {
t.Fatalf("Unexpected error : %s", err)
}
err = j.OneShotAcquisition(out, &tomb)
cstest.AssertErrorContains(t, err, ts.expectedErr)
if err != nil {
continue
}
if ts.expectedLines != 0 {
assert.Equal(t, ts.expectedLines, len(out))
}
if ts.expectedOutput != "" {
if hook.LastEntry() == nil {
t.Fatalf("Expected log output '%s' but got nothing !", ts.expectedOutput)
}
assert.Contains(t, hook.LastEntry().Message, ts.expectedOutput)
hook.Reset()
}
}
}
func TestStreaming(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("Skipping test on windows")
}
tests := []struct {
config string
expectedErr string
expectedOutput string
expectedLines int
logLevel log.Level
}{
{
config: `
source: journalctl
mode: cat
journalctl_filter:
- _SYSTEMD_UNIT=ssh.service`,
expectedErr: "",
expectedOutput: "",
logLevel: log.WarnLevel,
expectedLines: 14,
},
}
for _, ts := range tests {
var logger *log.Logger
var subLogger *log.Entry
var hook *test.Hook
if ts.expectedOutput != "" {
logger, hook = test.NewNullLogger()
logger.SetLevel(ts.logLevel)
subLogger = logger.WithFields(log.Fields{
"type": "journalctl",
})
} else {
subLogger = log.WithFields(log.Fields{
"type": "journalctl",
})
}
tomb := tomb.Tomb{}
out := make(chan types.Event)
j := JournalCtlSource{}
err := j.Configure([]byte(ts.config), subLogger)
if err != nil {
t.Fatalf("Unexpected error : %s", err)
}
actualLines := 0
if ts.expectedLines != 0 {
go func() {
READLOOP:
for {
select {
case <-out:
actualLines++
case <-time.After(1 * time.Second):
break READLOOP
}
}
}()
}
err = j.StreamingAcquisition(out, &tomb)
cstest.AssertErrorContains(t, err, ts.expectedErr)
if err != nil {
continue
}
if ts.expectedLines != 0 {
time.Sleep(1 * time.Second)
assert.Equal(t, ts.expectedLines, actualLines)
}
tomb.Kill(nil)
tomb.Wait()
output, _ := exec.Command("pgrep", "-x", "journalctl").CombinedOutput()
if string(output) != "" {
t.Fatalf("Found a journalctl process after killing the tomb !")
}
if ts.expectedOutput != "" {
if hook.LastEntry() == nil {
t.Fatalf("Expected log output '%s' but got nothing !", ts.expectedOutput)
}
assert.Contains(t, hook.LastEntry().Message, ts.expectedOutput)
hook.Reset()
}
}
}
func TestMain(m *testing.M) {
if os.Getenv("USE_SYSTEM_JOURNALCTL") == "" {
currentDir, _ := os.Getwd()
fullPath := filepath.Join(currentDir, "test_files")
os.Setenv("PATH", fullPath+":"+os.Getenv("PATH"))
}
os.Exit(m.Run())
}
crowdsec-1.4.6/pkg/acquisition/modules/journalctl/test_files/ 0000775 0000000 0000000 00000000000 14371172257 0024443 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/pkg/acquisition/modules/journalctl/test_files/journalctl 0000775 0000000 0000000 00000004323 14371172257 0026550 0 ustar 00root root 0000000 0000000 #!/usr/bin/env python3
import argparse
import time
import sys
class CustomParser(argparse.ArgumentParser):
#small hack to make argparse errors the same as journalctl
def error(self, message):
if 'unrecognized arguments:' in message:
sys.stderr.write("journalctl: invalid option -- '_'\n")
sys.stderr.flush()
exit(1)
else:
sys.stderr.write(message)
sys.stderr.flush()
exit(1)
LOGS = """-- Logs begin at Fri 2019-07-26 17:13:13 CEST, end at Mon 2020-11-23 09:17:34 CET. --
Nov 22 11:22:19 zeroed sshd[1480]: Invalid user wqeqwe from 127.0.0.1 port 55818
Nov 22 11:22:23 zeroed sshd[1480]: Failed password for invalid user wqeqwe from 127.0.0.1 port 55818 ssh2
Nov 22 11:23:22 zeroed sshd[1769]: Invalid user wqeqwe1 from 127.0.0.1 port 55824
Nov 22 11:23:24 zeroed sshd[1769]: Disconnecting invalid user wqeqwe1 127.0.0.1 port 55824: Too many authentication failures [preauth]
Nov 22 11:23:24 zeroed sshd[1777]: Invalid user wqeqwe2 from 127.0.0.1 port 55826
Nov 22 11:23:25 zeroed sshd[1777]: Disconnecting invalid user wqeqwe2 127.0.0.1 port 55826: Too many authentication failures [preauth]
Nov 22 11:23:25 zeroed sshd[1780]: Invalid user wqeqwe3 from 127.0.0.1 port 55828
Nov 22 11:23:26 zeroed sshd[1780]: Disconnecting invalid user wqeqwe3 127.0.0.1 port 55828: Too many authentication failures [preauth]
Nov 22 11:23:26 zeroed sshd[1786]: Invalid user wqeqwe4 from 127.0.0.1 port 55830
Nov 22 11:23:27 zeroed sshd[1786]: Failed password for invalid user wqeqwe4 from 127.0.0.1 port 55830 ssh2
Nov 22 11:23:27 zeroed sshd[1786]: Disconnecting invalid user wqeqwe4 127.0.0.1 port 55830: Too many authentication failures [preauth]
Nov 22 11:23:27 zeroed sshd[1791]: Invalid user wqeqwe5 from 127.0.0.1 port 55834
Nov 22 11:23:27 zeroed sshd[1791]: Failed password for invalid user wqeqwe5 from 127.0.0.1 port 55834 ssh2"""
parser = CustomParser()
parser.add_argument('filter', metavar='FILTER', type=str, nargs='?')
parser.add_argument('-n', dest='n', type=int)
parser.add_argument('--follow', dest='follow', action='store_true', default=False)
args = parser.parse_args()
for line in LOGS.split('\n'):
print(line)
if args.follow:
time.sleep(9999) crowdsec-1.4.6/pkg/acquisition/modules/kafka/ 0000775 0000000 0000000 00000000000 14371172257 0021202 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/pkg/acquisition/modules/kafka/kafka.go 0000664 0000000 0000000 00000014352 14371172257 0022613 0 ustar 00root root 0000000 0000000 package kafkaacquisition
import (
"context"
"crypto/tls"
"crypto/x509"
"fmt"
"io"
"os"
"strconv"
"time"
"github.com/crowdsecurity/crowdsec/pkg/acquisition/configuration"
"github.com/crowdsecurity/crowdsec/pkg/leakybucket"
"github.com/crowdsecurity/crowdsec/pkg/types"
"github.com/pkg/errors"
"github.com/prometheus/client_golang/prometheus"
"github.com/segmentio/kafka-go"
log "github.com/sirupsen/logrus"
"gopkg.in/tomb.v2"
"gopkg.in/yaml.v2"
)
var (
dataSourceName = "kafka"
)
var linesRead = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "cs_kafkasource_hits_total",
Help: "Total lines that were read from topic",
},
[]string{"topic"})
type KafkaConfiguration struct {
Brokers []string `yaml:"brokers"`
Topic string `yaml:"topic"`
GroupID string `yaml:"group_id"`
Timeout string `yaml:"timeout"`
TLS *TLSConfig `yaml:"tls"`
configuration.DataSourceCommonCfg `yaml:",inline"`
}
type TLSConfig struct {
InsecureSkipVerify bool `yaml:"insecure_skip_verify"`
ClientCert string `yaml:"client_cert"`
ClientKey string `yaml:"client_key"`
CaCert string `yaml:"ca_cert"`
}
type KafkaSource struct {
Config KafkaConfiguration
logger *log.Entry
Reader *kafka.Reader
}
func (k *KafkaSource) Configure(Config []byte, logger *log.Entry) error {
var err error
k.Config = KafkaConfiguration{}
k.logger = logger
err = yaml.UnmarshalStrict(Config, &k.Config)
if err != nil {
return errors.Wrapf(err, "cannot parse %s datasource configuration", dataSourceName)
}
if len(k.Config.Brokers) == 0 {
return fmt.Errorf("cannot create a %s reader with an empty list of broker addresses", dataSourceName)
}
if k.Config.Topic == "" {
return fmt.Errorf("cannot create a %s reader with am empty topic", dataSourceName)
}
if k.Config.Mode == "" {
k.Config.Mode = configuration.TAIL_MODE
}
dialer, err := k.Config.NewDialer()
if err != nil {
return errors.Wrapf(err, "cannot create %s dialer", dataSourceName)
}
k.Reader, err = k.Config.NewReader(dialer)
if err != nil {
return errors.Wrapf(err, "cannote create %s reader", dataSourceName)
}
if k.Reader == nil {
return fmt.Errorf("cannot create %s reader", dataSourceName)
}
return nil
}
func (k *KafkaSource) ConfigureByDSN(string, map[string]string, *log.Entry) error {
return fmt.Errorf("%s datasource does not support command-line acquisition", dataSourceName)
}
func (k *KafkaSource) GetMode() string {
return k.Config.Mode
}
func (k *KafkaSource) GetName() string {
return dataSourceName
}
func (k *KafkaSource) OneShotAcquisition(out chan types.Event, t *tomb.Tomb) error {
return fmt.Errorf("%s datasource does not support one-shot acquisition", dataSourceName)
}
func (k *KafkaSource) CanRun() error {
return nil
}
func (k *KafkaSource) GetMetrics() []prometheus.Collector {
return []prometheus.Collector{linesRead}
}
func (k *KafkaSource) GetAggregMetrics() []prometheus.Collector {
return []prometheus.Collector{linesRead}
}
func (k *KafkaSource) Dump() interface{} {
return k
}
func (k *KafkaSource) ReadMessage(out chan types.Event) error {
// Start processing from latest Offset
k.Reader.SetOffsetAt(context.Background(), time.Now())
for {
m, err := k.Reader.ReadMessage(context.Background())
if err != nil {
if err == io.EOF {
return nil
}
k.logger.Errorln(errors.Wrapf(err, "while reading %s message", dataSourceName))
}
l := types.Line{
Raw: string(m.Value),
Labels: k.Config.Labels,
Time: m.Time.UTC(),
Src: k.Config.Topic,
Process: true,
Module: k.GetName(),
}
linesRead.With(prometheus.Labels{"topic": k.Config.Topic}).Inc()
var evt types.Event
if !k.Config.UseTimeMachine {
evt = types.Event{Line: l, Process: true, Type: types.LOG, ExpectMode: leakybucket.LIVE}
} else {
evt = types.Event{Line: l, Process: true, Type: types.LOG, ExpectMode: leakybucket.TIMEMACHINE}
}
out <- evt
}
}
func (k *KafkaSource) RunReader(out chan types.Event, t *tomb.Tomb) error {
t.Go(func() error {
return k.ReadMessage(out)
})
//nolint //fp
for {
select {
case <-t.Dying():
k.logger.Infof("%s datasource topic %s stopping", dataSourceName, k.Config.Topic)
if err := k.Reader.Close(); err != nil {
return errors.Wrapf(err, "while closing %s reader on topic '%s'", dataSourceName, k.Config.Topic)
}
return nil
}
}
}
func (k *KafkaSource) StreamingAcquisition(out chan types.Event, t *tomb.Tomb) error {
k.logger.Infof("start reader on topic '%s'", k.Config.Topic)
t.Go(func() error {
defer types.CatchPanic("crowdsec/acquis/kafka/live")
return k.RunReader(out, t)
})
return nil
}
func (kc *KafkaConfiguration) NewTLSConfig() (*tls.Config, error) {
tlsConfig := tls.Config{
InsecureSkipVerify: kc.TLS.InsecureSkipVerify,
}
cert, err := tls.LoadX509KeyPair(kc.TLS.ClientCert, kc.TLS.ClientKey)
if err != nil {
return &tlsConfig, err
}
tlsConfig.Certificates = []tls.Certificate{cert}
caCert, err := os.ReadFile(kc.TLS.CaCert)
if err != nil {
return &tlsConfig, err
}
caCertPool := x509.NewCertPool()
caCertPool.AppendCertsFromPEM(caCert)
tlsConfig.RootCAs = caCertPool
tlsConfig.BuildNameToCertificate()
return &tlsConfig, err
}
func (kc *KafkaConfiguration) NewDialer() (*kafka.Dialer, error) {
dialer := &kafka.Dialer{}
var timeoutDuration time.Duration
timeoutDuration = time.Duration(10) * time.Second
if kc.Timeout != "" {
intTimeout, err := strconv.Atoi(kc.Timeout)
if err != nil {
return dialer, err
}
timeoutDuration = time.Duration(intTimeout) * time.Second
}
dialer = &kafka.Dialer{
Timeout: timeoutDuration,
DualStack: true,
}
if kc.TLS != nil {
tlsConfig, err := kc.NewTLSConfig()
if err != nil {
return dialer, err
}
dialer.TLS = tlsConfig
}
return dialer, nil
}
func (kc *KafkaConfiguration) NewReader(dialer *kafka.Dialer) (*kafka.Reader, error) {
rConf := kafka.ReaderConfig{
Brokers: kc.Brokers,
Topic: kc.Topic,
Dialer: dialer,
}
if kc.GroupID != "" {
rConf.GroupID = kc.GroupID
}
if err := rConf.Validate(); err != nil {
return &kafka.Reader{}, errors.Wrapf(err, "while validating reader configuration")
}
return kafka.NewReader(rConf), nil
}
crowdsec-1.4.6/pkg/acquisition/modules/kafka/kafka_test.go 0000664 0000000 0000000 00000012206 14371172257 0023646 0 ustar 00root root 0000000 0000000 package kafkaacquisition
import (
"context"
"net"
"runtime"
"strconv"
"testing"
"time"
"github.com/crowdsecurity/crowdsec/pkg/cstest"
"github.com/crowdsecurity/crowdsec/pkg/types"
"github.com/segmentio/kafka-go"
log "github.com/sirupsen/logrus"
"gopkg.in/tomb.v2"
"gotest.tools/v3/assert"
)
func TestConfigure(t *testing.T) {
tests := []struct {
config string
expectedErr string
}{
{
config: `
foobar: bla
source: kafka`,
expectedErr: "line 2: field foobar not found in type kafkaacquisition.KafkaConfiguration",
},
{
config: `source: kafka`,
expectedErr: "cannot create a kafka reader with an empty list of broker addresses",
},
{
config: `
source: kafka
brokers:
- bla
timeout: 5`,
expectedErr: "cannot create a kafka reader with am empty topic",
},
{
config: `
source: kafka
brokers:
- bla
topic: toto
timeout: aa`,
expectedErr: "cannot create kafka dialer: strconv.Atoi: parsing \"aa\": invalid syntax",
},
{
config: `
source: kafka
brokers:
- localhost:9092
topic: crowdsec`,
expectedErr: "",
},
}
subLogger := log.WithFields(log.Fields{
"type": "kafka",
})
for _, test := range tests {
k := KafkaSource{}
err := k.Configure([]byte(test.config), subLogger)
cstest.AssertErrorContains(t, err, test.expectedErr)
}
}
func writeToKafka(w *kafka.Writer, logs []string) {
for idx, log := range logs {
err := w.WriteMessages(context.Background(), kafka.Message{
Key: []byte(strconv.Itoa(idx)),
// create an arbitrary message payload for the value
Value: []byte(log),
})
if err != nil {
panic("could not write message " + err.Error())
}
}
}
func createTopic(topic string, broker string) {
conn, err := kafka.Dial("tcp", broker)
if err != nil {
panic(err)
}
defer conn.Close()
controller, err := conn.Controller()
if err != nil {
panic(err)
}
var controllerConn *kafka.Conn
controllerConn, err = kafka.Dial("tcp", net.JoinHostPort(controller.Host, strconv.Itoa(controller.Port)))
if err != nil {
panic(err)
}
defer controllerConn.Close()
topicConfigs := []kafka.TopicConfig{
{
Topic: topic,
NumPartitions: 1,
ReplicationFactor: 1,
},
}
err = controllerConn.CreateTopics(topicConfigs...)
if err != nil {
panic(err)
}
}
func TestStreamingAcquisition(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("Skipping test on windows")
}
tests := []struct {
name string
logs []string
expectedLines int
expectedErr string
}{
{
name: "valid msgs",
logs: []string{
"message 1",
"message 2",
"message 3",
},
expectedLines: 3,
},
}
subLogger := log.WithFields(log.Fields{
"type": "kafka",
})
createTopic("crowdsecplaintext", "localhost:9092")
w := kafka.NewWriter(kafka.WriterConfig{
Brokers: []string{"localhost:9092"},
Topic: "crowdsecplaintext",
})
if w == nil {
log.Fatalf("Unable to setup a kafka producer")
}
for _, ts := range tests {
ts := ts
t.Run(ts.name, func(t *testing.T) {
k := KafkaSource{}
err := k.Configure([]byte(`
source: kafka
brokers:
- localhost:9092
topic: crowdsecplaintext`), subLogger)
if err != nil {
t.Fatalf("could not configure kafka source : %s", err)
}
tomb := tomb.Tomb{}
out := make(chan types.Event)
err = k.StreamingAcquisition(out, &tomb)
cstest.AssertErrorContains(t, err, ts.expectedErr)
actualLines := 0
go writeToKafka(w, ts.logs)
READLOOP:
for {
select {
case <-out:
actualLines++
case <-time.After(2 * time.Second):
break READLOOP
}
}
assert.Equal(t, ts.expectedLines, actualLines)
tomb.Kill(nil)
tomb.Wait()
})
}
}
func TestStreamingAcquisitionWithSSL(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("Skipping test on windows")
}
tests := []struct {
name string
logs []string
expectedLines int
expectedErr string
}{
{
name: "valid msgs",
logs: []string{
"message 1",
"message 2",
},
expectedLines: 2,
},
}
subLogger := log.WithFields(log.Fields{
"type": "kafka",
})
createTopic("crowdsecssl", "localhost:9092")
w2 := kafka.NewWriter(kafka.WriterConfig{
Brokers: []string{"localhost:9092"},
Topic: "crowdsecssl",
})
if w2 == nil {
log.Fatalf("Unable to setup a kafka producer")
}
for _, ts := range tests {
ts := ts
t.Run(ts.name, func(t *testing.T) {
k := KafkaSource{}
err := k.Configure([]byte(`
source: kafka
brokers:
- localhost:9093
topic: crowdsecssl
tls:
insecure_skip_verify: true
client_cert: ./testdata/kafkaClient.certificate.pem
client_key: ./testdata/kafkaClient.key
ca_cert: ./testdata/snakeoil-ca-1.crt
`), subLogger)
if err != nil {
t.Fatalf("could not configure kafka source : %s", err)
}
tomb := tomb.Tomb{}
out := make(chan types.Event)
err = k.StreamingAcquisition(out, &tomb)
cstest.AssertErrorContains(t, err, ts.expectedErr)
actualLines := 0
go writeToKafka(w2, ts.logs)
READLOOP:
for {
select {
case <-out:
actualLines++
case <-time.After(2 * time.Second):
break READLOOP
}
}
assert.Equal(t, ts.expectedLines, actualLines)
tomb.Kill(nil)
tomb.Wait()
})
}
}
crowdsec-1.4.6/pkg/acquisition/modules/kafka/testdata/ 0000775 0000000 0000000 00000000000 14371172257 0023013 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/pkg/acquisition/modules/kafka/testdata/kafkaClient.certificate.pem 0000664 0000000 0000000 00000002533 14371172257 0030216 0 ustar 00root root 0000000 0000000 -----BEGIN CERTIFICATE-----
MIIDyDCCArCgAwIBAgIUZ3H0cvKHzTXDByikP2PLvhqCTSwwDQYJKoZIhvcNAQEL
BQAwbzEZMBcGA1UEAwwQQ3Jvd2RTZWMgVGVzdCBDQTERMA8GA1UECwwIQ3Jvd2Rz
ZWMxETAPBgNVBAoMCENyb3dkc2VjMQ4wDAYDVQQHDAVQYXJpczEPMA0GA1UECAwG
RnJhbmNlMQswCQYDVQQGEwJGUjAeFw0yMjA4MDExNjA5NDJaFw00OTEyMTYxNjA5
NDJaMGYxCzAJBgNVBAYTAkZSMQ8wDQYDVQQIEwZGcmFuY2UxDjAMBgNVBAcTBVBh
cmlzMREwDwYDVQQKEwhDcm93ZHNlYzENMAsGA1UECxMEVEVTVDEUMBIGA1UEAxML
a2Fma2FDbGllbnQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCG5R6r
xi9FcL6p8bD5+bpV3bTDNwRTF4b9psrhVY8MhvjvaoYODHvENJaDb3Z/ipDUdG6e
zjgigfjLRRwxvj2+E0nTn/TsfRQIlH2BYPEzCCrG33WKkcmG1K3LEbkXGyBcPljd
DPHb2nbZERDFBcIlqNM5N9+cbLFQnJNw3u7Nsv/e4jjUpAeYg30YVKwrr9P4mj8L
NR+ZALe0+2NUJpTYX0ZP4vPeYqTGWPshMrGHLrChhYnaYWzEvYITjVtzkHk9xtFg
uRGjgtwlpf0m2EM8GHhteHaLb2efU1C860QaFkTBK1JeGU1A2O6O8lOo8CMVG6h+
IBA1kspGRO0wmix5AgMBAAGjZTBjMCEGA1UdEQQaMBiCC2thZmthQ2xpZW50ggls
b2NhbGhvc3QwHQYDVR0OBBYEFD/QYQ2ppLhrC8qnfSjEsppvI/NPMB8GA1UdIwQY
MBaAFCCtzZtp2uUwxDCvIf8ETMpCtLxzMA0GCSqGSIb3DQEBCwUAA4IBAQA7Bly4
t1ob95om3h+d9tYOuqzERUhO9BZXjqGFmOxb4pmpg5ANa9j82VOy0PWvBPR4M6N5
uHwUKj6S4HWDLpabNNsBWYUzILBBQDqkiKgy0NmakZjv2fbFSIEpZF8sfyL3Z/ci
JRo6SqZWILh7B2BqysLmgTJeRFode3zqIKhLPIqYqEDBCwgSL1quX0afut2q86lx
x2RJB/N8QsNfXSjTOojXY3cJzLdW4XKGZKk75YhlpYt+v5235paVbocz32diQczk
9yCqfJfG8BBNA6WdPgtLhQHiDLO7UY+Y+jGIe2G41w7adT/b2Omeb2h3RbGeqKx9
WteVlQb955ItDXKI
-----END CERTIFICATE-----
crowdsec-1.4.6/pkg/acquisition/modules/kafka/testdata/kafkaClient.key 0000664 0000000 0000000 00000003474 14371172257 0025751 0 ustar 00root root 0000000 0000000 Bag Attributes
friendlyName: kafkaclient
localKeyID: 54 69 6D 65 20 31 36 35 39 33 37 30 31 38 36 30 33 36
Key Attributes:
-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCG5R6rxi9FcL6p
8bD5+bpV3bTDNwRTF4b9psrhVY8MhvjvaoYODHvENJaDb3Z/ipDUdG6ezjgigfjL
RRwxvj2+E0nTn/TsfRQIlH2BYPEzCCrG33WKkcmG1K3LEbkXGyBcPljdDPHb2nbZ
ERDFBcIlqNM5N9+cbLFQnJNw3u7Nsv/e4jjUpAeYg30YVKwrr9P4mj8LNR+ZALe0
+2NUJpTYX0ZP4vPeYqTGWPshMrGHLrChhYnaYWzEvYITjVtzkHk9xtFguRGjgtwl
pf0m2EM8GHhteHaLb2efU1C860QaFkTBK1JeGU1A2O6O8lOo8CMVG6h+IBA1kspG
RO0wmix5AgMBAAECggEAYLrdwomMDjxpzI2OvcJQ1w/zdmT2Ses+FpfLNchXmsES
swPs+xgCjFC1eaytCYpAjsirJl90K3KOCJ0XOahUt/822nUCea67deedE/CDJXf+
zLsim2otW+0YbtzXn/UIwHzI1kJZELFYthEhuFaHwN+OD6K8S3w5rjeJFtAV6BQc
AzMAwQ+6j8DG/V+5mgp4YCrcGXmJzvdXqJdiBmFwoEOAdp/ahTjMscmyI0ZQdMpF
t8e8x5WDVT65ScUA6nvKkSttYhJ5qpEWiMerR8rBJbkbNi529fUW7/sX1xTMYI5G
psOxGdXGSyH2i365DtFxHhtovSc+TgWpNJgZDcJLLQKBgQDdWi4sJO985m1hps7m
bKiXLJE/gbSYQekIlt4/KGbY01BrKSSvgsVTIZbC/m8GhAG48uLG1y51R30cGx6g
Wg8616duqnq+P7pvw+hWsGrtzYh4URx6T91SJi313Xi5ouqLsNiMznPoLXEnDgJv
xO17TkCnThU/Ms6ml1PFeccAYwKBgQCcAoFVqibXRyFoD2UVMpgekOQ+QxBZXItm
RoiBhimEahhx6DjJ9GPmAWUTJfAMom0TNoYa/ziM/+VdNruUMRUvAqBHV2vqllYE
Szhfxlh0RyCiZzrgEqZLVMdr0vxbeA4e5D2+26NH0YHGqCVacdX5659bSM5hcP/s
WO/fGIcAcwKBgQCIKv3UcjRRZX9MX01QOu/Lb8M6ihQKxiERA55cxAHgyL3j7j9/
KLcy2Krh8ZtjKrnUiLYxFBakVwID1DsW8dAEpr19Oqqfdpy+PIolKgndmF6nhV47
b/36lzoW0dN+f1ZB8NyGYkqzPaEqIVgmYcKl5BGp2kL/ycWOffEuvidJeQKBgHls
eb1y8Ugc1KNpWANnnX4sx3iuOctTfCveOeCVyzqEWQJO++Qzko0yCLkejfdHdB3A
EiBxBFK52Ir0TorIqPQt1xGvuQ6cc9ZjtTzV44Kc2YmNTwWXflajZZNGY6PNjS/9
9RDXYf5D0f4MYQZEE4axHRavU/IDQS1zCz9Yl7qBAoGAezrd5EziH9VNHMzPnfkH
+Lg2DCrRbyG1pHpuKJg2i98Ulwuu/9A5m5Vj1iYrANt9v4ycWimLikyd5vJXW60V
9PBb8FB/vjpXNJ1PZBGjlxgpWpzF13JGcJpFBK+z5yCPevzJlc/H+IAQbd3mS3WW
DDwAGG2L41aLYKmsjAtr76I=
-----END PRIVATE KEY-----
crowdsec-1.4.6/pkg/acquisition/modules/kafka/testdata/snakeoil-ca-1.crt 0000664 0000000 0000000 00000002517 14371172257 0026056 0 ustar 00root root 0000000 0000000 -----BEGIN CERTIFICATE-----
MIIDvzCCAqegAwIBAgIUSc+OZ8EjaDgzm0sjqlHVXjZ0od0wDQYJKoZIhvcNAQEL
BQAwbzEZMBcGA1UEAwwQQ3Jvd2RTZWMgVGVzdCBDQTERMA8GA1UECwwIQ3Jvd2Rz
ZWMxETAPBgNVBAoMCENyb3dkc2VjMQ4wDAYDVQQHDAVQYXJpczEPMA0GA1UECAwG
RnJhbmNlMQswCQYDVQQGEwJGUjAeFw0yMjA4MDExNjA5MzZaFw0yMzA4MDExNjA5
MzZaMG8xGTAXBgNVBAMMEENyb3dkU2VjIFRlc3QgQ0ExETAPBgNVBAsMCENyb3dk
c2VjMREwDwYDVQQKDAhDcm93ZHNlYzEOMAwGA1UEBwwFUGFyaXMxDzANBgNVBAgM
BkZyYW5jZTELMAkGA1UEBhMCRlIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQDAzj1gJzEhzcymL9dZX6+dTLdi9RDoII7PWtYCIoY5tqvewOzVBZMEDEhG
az8Btwo6Edr7u804Zule9ZVSaTkmse+thNthukXrtmTEuOuienym5KkVddNckTtr
w/5MLMkKK0Ux45BYW4H3wT1HpD56ezCUhxL5O3ACPjufw7yvMheHRnQxe3txmlNq
rd9swZH0sdZovWmW2Fj+C5qYbP/6hLzii9SNWOzOnKxlbw8CMBzK7KZgWp4qi4sz
tFCkGh2+Ya2QV3+q9Z6fD3hTZJfELEbDgP7ULYrvlGzLrrfLFcqAwmQ360PlsWiL
bg0+/rWkBRz/3wpma2RP+dGFfaj7AgMBAAGjUzBRMB0GA1UdDgQWBBQgrc2badrl
MMQwryH/BEzKQrS8czAfBgNVHSMEGDAWgBQgrc2badrlMMQwryH/BEzKQrS8czAP
BgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQAE6ct4k+X4hAw+TUpN
E/rVrEybHFv6qvgyE7Ay/LhpevU+r8UBChv3XZ/u3h4WKdqFrxPN4JDOvIXN0Jq2
Xs7Bs//hj+hULvJ3DWfQEQ6LivcxVxQsU47Sbxf6sUeCV3kXSxjFEcsvSx9kPNv6
3Bi1EwPrMiwNdpB1BDUG7Z2mFxhoHv1SUppE7Lhu/x/1b7LgYqNy2VWWOFg/TZI2
tdg45fMtNYp8kdQW+r18YxToQHUjXkkQqW9HSyxIqeabVqxuuptyY+OSIIFxBWaB
A4BbiHPKhJ0umCQa9mPeVKWUjUeXzRnHMXw2nPyqfK+1wQXt/7DZrBQLVe5Z9IHG
DZj/
-----END CERTIFICATE-----
crowdsec-1.4.6/pkg/acquisition/modules/kinesis/ 0000775 0000000 0000000 00000000000 14371172257 0021572 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/pkg/acquisition/modules/kinesis/kinesis.go 0000664 0000000 0000000 00000041233 14371172257 0023571 0 ustar 00root root 0000000 0000000 package kinesisacquisition
import (
"bytes"
"compress/gzip"
"encoding/json"
"fmt"
"io"
"strings"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/arn"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/kinesis"
"github.com/crowdsecurity/crowdsec/pkg/acquisition/configuration"
"github.com/crowdsecurity/crowdsec/pkg/leakybucket"
"github.com/crowdsecurity/crowdsec/pkg/types"
"github.com/pkg/errors"
"github.com/prometheus/client_golang/prometheus"
log "github.com/sirupsen/logrus"
"gopkg.in/tomb.v2"
"gopkg.in/yaml.v2"
)
type KinesisConfiguration struct {
configuration.DataSourceCommonCfg `yaml:",inline"`
StreamName string `yaml:"stream_name"`
StreamARN string `yaml:"stream_arn"`
UseEnhancedFanOut bool `yaml:"use_enhanced_fanout"` //Use RegisterStreamConsumer and SubscribeToShard instead of GetRecords
AwsProfile *string `yaml:"aws_profile"`
AwsRegion string `yaml:"aws_region"`
AwsEndpoint string `yaml:"aws_endpoint"`
ConsumerName string `yaml:"consumer_name"`
FromSubscription bool `yaml:"from_subscription"`
MaxRetries int `yaml:"max_retries"`
}
type KinesisSource struct {
Config KinesisConfiguration
logger *log.Entry
kClient *kinesis.Kinesis
shardReaderTomb *tomb.Tomb
}
type CloudWatchSubscriptionRecord struct {
MessageType string `json:"messageType"`
Owner string `json:"owner"`
LogGroup string `json:"logGroup"`
LogStream string `json:"logStream"`
SubscriptionFilters []string `json:"subscriptionFilters"`
LogEvents []CloudwatchSubscriptionLogEvent `json:"logEvents"`
}
type CloudwatchSubscriptionLogEvent struct {
ID string `json:"id"`
Message string `json:"message"`
Timestamp int64 `json:"timestamp"`
}
var linesRead = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "cs_kinesis_stream_hits_total",
Help: "Number of event read per stream.",
},
[]string{"stream"},
)
var linesReadShards = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "cs_kinesis_shards_hits_total",
Help: "Number of event read per shards.",
},
[]string{"stream", "shard"},
)
func (k *KinesisSource) newClient() error {
var sess *session.Session
if k.Config.AwsProfile != nil {
sess = session.Must(session.NewSessionWithOptions(session.Options{
SharedConfigState: session.SharedConfigEnable,
Profile: *k.Config.AwsProfile,
}))
} else {
sess = session.Must(session.NewSessionWithOptions(session.Options{
SharedConfigState: session.SharedConfigEnable,
}))
}
if sess == nil {
return fmt.Errorf("failed to create aws session")
}
config := aws.NewConfig()
if k.Config.AwsRegion != "" {
config = config.WithRegion(k.Config.AwsRegion)
}
if k.Config.AwsEndpoint != "" {
config = config.WithEndpoint(k.Config.AwsEndpoint)
}
k.kClient = kinesis.New(sess, config)
if k.kClient == nil {
return fmt.Errorf("failed to create kinesis client")
}
return nil
}
func (k *KinesisSource) GetMetrics() []prometheus.Collector {
return []prometheus.Collector{linesRead, linesReadShards}
}
func (k *KinesisSource) GetAggregMetrics() []prometheus.Collector {
return []prometheus.Collector{linesRead, linesReadShards}
}
func (k *KinesisSource) Configure(yamlConfig []byte, logger *log.Entry) error {
config := KinesisConfiguration{}
k.logger = logger
err := yaml.UnmarshalStrict(yamlConfig, &config)
if err != nil {
return errors.Wrap(err, "Cannot parse kinesis datasource configuration")
}
if config.Mode == "" {
config.Mode = configuration.TAIL_MODE
}
k.Config = config
if k.Config.StreamName == "" && !k.Config.UseEnhancedFanOut {
return fmt.Errorf("stream_name is mandatory when use_enhanced_fanout is false")
}
if k.Config.StreamARN == "" && k.Config.UseEnhancedFanOut {
return fmt.Errorf("stream_arn is mandatory when use_enhanced_fanout is true")
}
if k.Config.ConsumerName == "" && k.Config.UseEnhancedFanOut {
return fmt.Errorf("consumer_name is mandatory when use_enhanced_fanout is true")
}
if k.Config.StreamARN != "" && k.Config.StreamName != "" {
return fmt.Errorf("stream_arn and stream_name are mutually exclusive")
}
if k.Config.MaxRetries <= 0 {
k.Config.MaxRetries = 10
}
err = k.newClient()
if err != nil {
return errors.Wrap(err, "Cannot create kinesis client")
}
k.shardReaderTomb = &tomb.Tomb{}
return nil
}
func (k *KinesisSource) ConfigureByDSN(string, map[string]string, *log.Entry) error {
return fmt.Errorf("kinesis datasource does not support command-line acquisition")
}
func (k *KinesisSource) GetMode() string {
return k.Config.Mode
}
func (k *KinesisSource) GetName() string {
return "kinesis"
}
func (k *KinesisSource) OneShotAcquisition(out chan types.Event, t *tomb.Tomb) error {
return fmt.Errorf("kinesis datasource does not support one-shot acquisition")
}
func (k *KinesisSource) decodeFromSubscription(record []byte) ([]CloudwatchSubscriptionLogEvent, error) {
b := bytes.NewBuffer(record)
r, err := gzip.NewReader(b)
if err != nil {
k.logger.Error(err)
return nil, err
}
decompressed, err := io.ReadAll(r)
if err != nil {
k.logger.Error(err)
return nil, err
}
var subscriptionRecord CloudWatchSubscriptionRecord
err = json.Unmarshal(decompressed, &subscriptionRecord)
if err != nil {
k.logger.Error(err)
return nil, err
}
return subscriptionRecord.LogEvents, nil
}
func (k *KinesisSource) WaitForConsumerDeregistration(consumerName string, streamARN string) error {
maxTries := k.Config.MaxRetries
for i := 0; i < maxTries; i++ {
_, err := k.kClient.DescribeStreamConsumer(&kinesis.DescribeStreamConsumerInput{
ConsumerName: aws.String(consumerName),
StreamARN: aws.String(streamARN),
})
if err != nil {
switch err.(type) {
case *kinesis.ResourceNotFoundException:
return nil
default:
k.logger.Errorf("Error while waiting for consumer deregistration: %s", err)
return errors.Wrap(err, "Cannot describe stream consumer")
}
}
time.Sleep(time.Millisecond * 200 * time.Duration(i+1))
}
return fmt.Errorf("consumer %s is not deregistered after %d tries", consumerName, maxTries)
}
func (k *KinesisSource) DeregisterConsumer() error {
k.logger.Debugf("Deregistering consumer %s if it exists", k.Config.ConsumerName)
_, err := k.kClient.DeregisterStreamConsumer(&kinesis.DeregisterStreamConsumerInput{
ConsumerName: aws.String(k.Config.ConsumerName),
StreamARN: aws.String(k.Config.StreamARN),
})
if err != nil {
switch err.(type) {
case *kinesis.ResourceNotFoundException:
default:
return errors.Wrap(err, "Cannot deregister stream consumer")
}
}
err = k.WaitForConsumerDeregistration(k.Config.ConsumerName, k.Config.StreamARN)
if err != nil {
return errors.Wrap(err, "Cannot wait for consumer deregistration")
}
return nil
}
func (k *KinesisSource) WaitForConsumerRegistration(consumerARN string) error {
maxTries := k.Config.MaxRetries
for i := 0; i < maxTries; i++ {
describeOutput, err := k.kClient.DescribeStreamConsumer(&kinesis.DescribeStreamConsumerInput{
ConsumerARN: aws.String(consumerARN),
})
if err != nil {
return errors.Wrap(err, "Cannot describe stream consumer")
}
if *describeOutput.ConsumerDescription.ConsumerStatus == "ACTIVE" {
k.logger.Debugf("Consumer %s is active", consumerARN)
return nil
}
time.Sleep(time.Millisecond * 200 * time.Duration(i+1))
k.logger.Debugf("Waiting for consumer registration %d", i)
}
return fmt.Errorf("consumer %s is not active after %d tries", consumerARN, maxTries)
}
func (k *KinesisSource) RegisterConsumer() (*kinesis.RegisterStreamConsumerOutput, error) {
k.logger.Debugf("Registering consumer %s", k.Config.ConsumerName)
streamConsumer, err := k.kClient.RegisterStreamConsumer(&kinesis.RegisterStreamConsumerInput{
ConsumerName: aws.String(k.Config.ConsumerName),
StreamARN: aws.String(k.Config.StreamARN),
})
if err != nil {
return nil, errors.Wrap(err, "Cannot register stream consumer")
}
err = k.WaitForConsumerRegistration(*streamConsumer.Consumer.ConsumerARN)
if err != nil {
return nil, errors.Wrap(err, "Timeout while waiting for consumer to be active")
}
return streamConsumer, nil
}
func (k *KinesisSource) ParseAndPushRecords(records []*kinesis.Record, out chan types.Event, logger *log.Entry, shardId string) {
for _, record := range records {
if k.Config.StreamARN != "" {
linesReadShards.With(prometheus.Labels{"stream": k.Config.StreamARN, "shard": shardId}).Inc()
linesRead.With(prometheus.Labels{"stream": k.Config.StreamARN}).Inc()
} else {
linesReadShards.With(prometheus.Labels{"stream": k.Config.StreamName, "shard": shardId}).Inc()
linesRead.With(prometheus.Labels{"stream": k.Config.StreamName}).Inc()
}
var data []CloudwatchSubscriptionLogEvent
var err error
if k.Config.FromSubscription {
//The AWS docs says that the data is base64 encoded
//but apparently GetRecords decodes it for us ?
data, err = k.decodeFromSubscription(record.Data)
if err != nil {
logger.Errorf("Cannot decode data: %s", err)
continue
}
} else {
data = []CloudwatchSubscriptionLogEvent{{Message: string(record.Data)}}
}
for _, event := range data {
logger.Tracef("got record %s", event.Message)
l := types.Line{}
l.Raw = event.Message
l.Labels = k.Config.Labels
l.Time = time.Now().UTC()
l.Process = true
l.Module = k.GetName()
if k.Config.StreamARN != "" {
l.Src = k.Config.StreamARN
} else {
l.Src = k.Config.StreamName
}
var evt types.Event
if !k.Config.UseTimeMachine {
evt = types.Event{Line: l, Process: true, Type: types.LOG, ExpectMode: leakybucket.LIVE}
} else {
evt = types.Event{Line: l, Process: true, Type: types.LOG, ExpectMode: leakybucket.TIMEMACHINE}
}
out <- evt
}
}
}
func (k *KinesisSource) ReadFromSubscription(reader kinesis.SubscribeToShardEventStreamReader, out chan types.Event, shardId string, streamName string) error {
logger := k.logger.WithFields(log.Fields{"shard_id": shardId})
//ghetto sync, kinesis allows to subscribe to a closed shard, which will make the goroutine exit immediately
//and we won't be able to start a new one if this is the first one started by the tomb
//TODO: look into parent shards to see if a shard is closed before starting to read it ?
time.Sleep(time.Second)
for {
select {
case <-k.shardReaderTomb.Dying():
logger.Infof("Subscribed shard reader is dying")
err := reader.Close()
if err != nil {
return errors.Wrap(err, "Cannot close kinesis subscribed shard reader")
}
return nil
case event, ok := <-reader.Events():
if !ok {
logger.Infof("Event chan has been closed")
return nil
}
switch event := event.(type) {
case *kinesis.SubscribeToShardEvent:
k.ParseAndPushRecords(event.Records, out, logger, shardId)
case *kinesis.SubscribeToShardEventStreamUnknownEvent:
logger.Infof("got an unknown event, what to do ?")
}
}
}
}
func (k *KinesisSource) SubscribeToShards(arn arn.ARN, streamConsumer *kinesis.RegisterStreamConsumerOutput, out chan types.Event) error {
shards, err := k.kClient.ListShards(&kinesis.ListShardsInput{
StreamName: aws.String(arn.Resource[7:]),
})
if err != nil {
return errors.Wrap(err, "Cannot list shards for enhanced_read")
}
for _, shard := range shards.Shards {
shardId := *shard.ShardId
r, err := k.kClient.SubscribeToShard(&kinesis.SubscribeToShardInput{
ShardId: aws.String(shardId),
StartingPosition: &kinesis.StartingPosition{Type: aws.String(kinesis.ShardIteratorTypeLatest)},
ConsumerARN: streamConsumer.Consumer.ConsumerARN,
})
if err != nil {
return errors.Wrap(err, "Cannot subscribe to shard")
}
k.shardReaderTomb.Go(func() error {
return k.ReadFromSubscription(r.GetEventStream().Reader, out, shardId, arn.Resource[7:])
})
}
return nil
}
func (k *KinesisSource) EnhancedRead(out chan types.Event, t *tomb.Tomb) error {
parsedARN, err := arn.Parse(k.Config.StreamARN)
if err != nil {
return errors.Wrap(err, "Cannot parse stream ARN")
}
if !strings.HasPrefix(parsedARN.Resource, "stream/") {
return fmt.Errorf("resource part of stream ARN %s does not start with stream/", k.Config.StreamARN)
}
k.logger = k.logger.WithFields(log.Fields{"stream": parsedARN.Resource[7:]})
k.logger.Info("starting kinesis acquisition with enhanced fan-out")
err = k.DeregisterConsumer()
if err != nil {
return errors.Wrap(err, "Cannot deregister consumer")
}
streamConsumer, err := k.RegisterConsumer()
if err != nil {
return errors.Wrap(err, "Cannot register consumer")
}
for {
k.shardReaderTomb = &tomb.Tomb{}
err = k.SubscribeToShards(parsedARN, streamConsumer, out)
if err != nil {
return errors.Wrap(err, "Cannot subscribe to shards")
}
select {
case <-t.Dying():
k.logger.Infof("Kinesis source is dying")
k.shardReaderTomb.Kill(nil)
_ = k.shardReaderTomb.Wait() //we don't care about the error as we kill the tomb ourselves
err = k.DeregisterConsumer()
if err != nil {
return errors.Wrap(err, "Cannot deregister consumer")
}
return nil
case <-k.shardReaderTomb.Dying():
k.logger.Debugf("Kinesis subscribed shard reader is dying")
if k.shardReaderTomb.Err() != nil {
return k.shardReaderTomb.Err()
}
//All goroutines have exited without error, so a resharding event, start again
k.logger.Debugf("All reader goroutines have exited, resharding event or periodic resubscribe")
continue
}
}
}
func (k *KinesisSource) ReadFromShard(out chan types.Event, shardId string) error {
logger := k.logger.WithFields(log.Fields{"shard": shardId})
logger.Debugf("Starting to read shard")
sharIt, err := k.kClient.GetShardIterator(&kinesis.GetShardIteratorInput{ShardId: aws.String(shardId),
StreamName: &k.Config.StreamName,
ShardIteratorType: aws.String(kinesis.ShardIteratorTypeLatest)})
if err != nil {
logger.Errorf("Cannot get shard iterator: %s", err)
return errors.Wrap(err, "Cannot get shard iterator")
}
it := sharIt.ShardIterator
//AWS recommends to wait for a second between calls to GetRecords for a given shard
ticker := time.NewTicker(time.Second)
for {
select {
case <-ticker.C:
records, err := k.kClient.GetRecords(&kinesis.GetRecordsInput{ShardIterator: it})
it = records.NextShardIterator
if err != nil {
switch err.(type) {
case *kinesis.ProvisionedThroughputExceededException:
logger.Warn("Provisioned throughput exceeded")
//TODO: implement exponential backoff
continue
case *kinesis.ExpiredIteratorException:
logger.Warn("Expired iterator")
continue
default:
logger.Error("Cannot get records")
return errors.Wrap(err, "Cannot get records")
}
}
k.ParseAndPushRecords(records.Records, out, logger, shardId)
if it == nil {
logger.Warnf("Shard has been closed")
return nil
}
case <-k.shardReaderTomb.Dying():
logger.Infof("shardReaderTomb is dying, exiting ReadFromShard")
ticker.Stop()
return nil
}
}
}
func (k *KinesisSource) ReadFromStream(out chan types.Event, t *tomb.Tomb) error {
k.logger = k.logger.WithFields(log.Fields{"stream": k.Config.StreamName})
k.logger.Info("starting kinesis acquisition from shards")
for {
shards, err := k.kClient.ListShards(&kinesis.ListShardsInput{
StreamName: aws.String(k.Config.StreamName),
})
if err != nil {
return errors.Wrap(err, "Cannot list shards")
}
k.shardReaderTomb = &tomb.Tomb{}
for _, shard := range shards.Shards {
shardId := *shard.ShardId
k.shardReaderTomb.Go(func() error {
defer types.CatchPanic("crowdsec/acquis/kinesis/streaming/shard")
return k.ReadFromShard(out, shardId)
})
}
select {
case <-t.Dying():
k.logger.Info("kinesis source is dying")
k.shardReaderTomb.Kill(nil)
_ = k.shardReaderTomb.Wait() //we don't care about the error as we kill the tomb ourselves
return nil
case <-k.shardReaderTomb.Dying():
reason := k.shardReaderTomb.Err()
if reason != nil {
k.logger.Errorf("Unexpected error from shard reader : %s", reason)
return reason
}
k.logger.Infof("All shards have been closed, probably a resharding event, restarting acquisition")
continue
}
}
}
func (k *KinesisSource) StreamingAcquisition(out chan types.Event, t *tomb.Tomb) error {
t.Go(func() error {
defer types.CatchPanic("crowdsec/acquis/kinesis/streaming")
if k.Config.UseEnhancedFanOut {
return k.EnhancedRead(out, t)
} else {
return k.ReadFromStream(out, t)
}
})
return nil
}
func (k *KinesisSource) CanRun() error {
return nil
}
func (k *KinesisSource) Dump() interface{} {
return k
}
crowdsec-1.4.6/pkg/acquisition/modules/kinesis/kinesis_test.go 0000664 0000000 0000000 00000017531 14371172257 0024634 0 ustar 00root root 0000000 0000000 package kinesisacquisition
import (
"bytes"
"compress/gzip"
"encoding/json"
"fmt"
"net"
"os"
"runtime"
"strings"
"testing"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/kinesis"
"github.com/crowdsecurity/crowdsec/pkg/cstest"
"github.com/crowdsecurity/crowdsec/pkg/types"
log "github.com/sirupsen/logrus"
"github.com/stretchr/testify/assert"
"gopkg.in/tomb.v2"
)
func getLocalStackEndpoint() (string, error) {
endpoint := "http://localhost:4566"
if v := os.Getenv("AWS_ENDPOINT_FORCE"); v != "" {
v = strings.TrimPrefix(v, "http://")
_, err := net.Dial("tcp", v)
if err != nil {
return "", fmt.Errorf("while dialing %s : %s : aws endpoint isn't available", v, err)
}
}
return endpoint, nil
}
func GenSubObject(i int) []byte {
r := CloudWatchSubscriptionRecord{
MessageType: "subscription",
Owner: "test",
LogGroup: "test",
LogStream: "test",
SubscriptionFilters: []string{"filter1"},
LogEvents: []CloudwatchSubscriptionLogEvent{
{
ID: "testid",
Message: fmt.Sprintf("%d", i),
Timestamp: time.Now().UTC().Unix(),
},
},
}
body, err := json.Marshal(r)
if err != nil {
log.Fatal(err)
}
var b bytes.Buffer
gz := gzip.NewWriter(&b)
gz.Write(body)
gz.Close()
//AWS actually base64 encodes the data, but it looks like kinesis automatically decodes it at some point
//localstack does not do it, so let's just write a raw gzipped stream
return b.Bytes()
}
func WriteToStream(streamName string, count int, shards int, sub bool) {
endpoint, err := getLocalStackEndpoint()
if err != nil {
log.Fatal(err)
}
sess := session.Must(session.NewSession())
kinesisClient := kinesis.New(sess, aws.NewConfig().WithEndpoint(endpoint).WithRegion("us-east-1"))
for i := 0; i < count; i++ {
partition := "partition"
if shards != 1 {
partition = fmt.Sprintf("partition-%d", i%shards)
}
var data []byte
if sub {
data = GenSubObject(i)
} else {
data = []byte(fmt.Sprintf("%d", i))
}
_, err = kinesisClient.PutRecord(&kinesis.PutRecordInput{
Data: data,
PartitionKey: aws.String(partition),
StreamName: aws.String(streamName),
})
if err != nil {
fmt.Printf("Error writing to stream: %s\n", err)
log.Fatal(err)
}
}
}
func TestMain(m *testing.M) {
os.Setenv("AWS_ACCESS_KEY_ID", "foobar")
os.Setenv("AWS_SECRET_ACCESS_KEY", "foobar")
//delete_streams()
//create_streams()
code := m.Run()
//delete_streams()
os.Exit(code)
}
func TestBadConfiguration(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("Skipping test on windows")
}
tests := []struct {
config string
expectedErr string
}{
{
config: `source: kinesis`,
expectedErr: "stream_name is mandatory when use_enhanced_fanout is false",
},
{
config: `
source: kinesis
use_enhanced_fanout: true`,
expectedErr: "stream_arn is mandatory when use_enhanced_fanout is true",
},
{
config: `
source: kinesis
use_enhanced_fanout: true
stream_arn: arn:aws:kinesis:eu-west-1:123456789012:stream/my-stream`,
expectedErr: "consumer_name is mandatory when use_enhanced_fanout is true",
},
{
config: `
source: kinesis
stream_name: foobar
stream_arn: arn:aws:kinesis:eu-west-1:123456789012:stream/my-stream`,
expectedErr: "stream_arn and stream_name are mutually exclusive",
},
}
subLogger := log.WithFields(log.Fields{
"type": "kinesis",
})
for _, test := range tests {
f := KinesisSource{}
err := f.Configure([]byte(test.config), subLogger)
cstest.AssertErrorContains(t, err, test.expectedErr)
}
}
func TestReadFromStream(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("Skipping test on windows")
}
tests := []struct {
config string
count int
shards int
}{
{
config: `source: kinesis
aws_endpoint: %s
aws_region: us-east-1
stream_name: stream-1-shard`,
count: 10,
shards: 1,
},
}
endpoint, _ := getLocalStackEndpoint()
for _, test := range tests {
f := KinesisSource{}
config := fmt.Sprintf(test.config, endpoint)
err := f.Configure([]byte(config), log.WithFields(log.Fields{
"type": "kinesis",
}))
if err != nil {
t.Fatalf("Error configuring source: %s", err)
}
tomb := &tomb.Tomb{}
out := make(chan types.Event)
err = f.StreamingAcquisition(out, tomb)
if err != nil {
t.Fatalf("Error starting source: %s", err)
}
//Allow the datasource to start listening to the stream
time.Sleep(4 * time.Second)
WriteToStream(f.Config.StreamName, test.count, test.shards, false)
for i := 0; i < test.count; i++ {
e := <-out
assert.Equal(t, fmt.Sprintf("%d", i), e.Line.Raw)
}
tomb.Kill(nil)
tomb.Wait()
}
}
func TestReadFromMultipleShards(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("Skipping test on windows")
}
tests := []struct {
config string
count int
shards int
}{
{
config: `source: kinesis
aws_endpoint: %s
aws_region: us-east-1
stream_name: stream-2-shards`,
count: 10,
shards: 2,
},
}
endpoint, _ := getLocalStackEndpoint()
for _, test := range tests {
f := KinesisSource{}
config := fmt.Sprintf(test.config, endpoint)
err := f.Configure([]byte(config), log.WithFields(log.Fields{
"type": "kinesis",
}))
if err != nil {
t.Fatalf("Error configuring source: %s", err)
}
tomb := &tomb.Tomb{}
out := make(chan types.Event)
err = f.StreamingAcquisition(out, tomb)
if err != nil {
t.Fatalf("Error starting source: %s", err)
}
//Allow the datasource to start listening to the stream
time.Sleep(4 * time.Second)
WriteToStream(f.Config.StreamName, test.count, test.shards, false)
c := 0
for i := 0; i < test.count; i++ {
<-out
c += 1
}
assert.Equal(t, test.count, c)
tomb.Kill(nil)
tomb.Wait()
}
}
func TestFromSubscription(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("Skipping test on windows")
}
tests := []struct {
config string
count int
shards int
}{
{
config: `source: kinesis
aws_endpoint: %s
aws_region: us-east-1
stream_name: stream-1-shard
from_subscription: true`,
count: 10,
shards: 1,
},
}
endpoint, _ := getLocalStackEndpoint()
for _, test := range tests {
f := KinesisSource{}
config := fmt.Sprintf(test.config, endpoint)
err := f.Configure([]byte(config), log.WithFields(log.Fields{
"type": "kinesis",
}))
if err != nil {
t.Fatalf("Error configuring source: %s", err)
}
tomb := &tomb.Tomb{}
out := make(chan types.Event)
err = f.StreamingAcquisition(out, tomb)
if err != nil {
t.Fatalf("Error starting source: %s", err)
}
//Allow the datasource to start listening to the stream
time.Sleep(4 * time.Second)
WriteToStream(f.Config.StreamName, test.count, test.shards, true)
for i := 0; i < test.count; i++ {
e := <-out
assert.Equal(t, fmt.Sprintf("%d", i), e.Line.Raw)
}
tomb.Kill(nil)
tomb.Wait()
}
}
/*
func TestSubscribeToStream(t *testing.T) {
tests := []struct {
config string
count int
shards int
}{
{
config: `source: kinesis
aws_endpoint: %s
aws_region: us-east-1
stream_arn: arn:aws:kinesis:us-east-1:000000000000:stream/stream-1-shard
consumer_name: consumer-1
use_enhanced_fanout: true`,
count: 10,
shards: 1,
},
}
endpoint, _ := getLocalStackEndpoint()
for _, test := range tests {
f := KinesisSource{}
config := fmt.Sprintf(test.config, endpoint)
err := f.Configure([]byte(config), log.WithFields(log.Fields{
"type": "kinesis",
}))
if err != nil {
t.Fatalf("Error configuring source: %s", err)
}
tomb := &tomb.Tomb{}
out := make(chan types.Event)
err = f.StreamingAcquisition(out, tomb)
if err != nil {
t.Fatalf("Error starting source: %s", err)
}
//Allow the datasource to start listening to the stream
time.Sleep(10 * time.Second)
WriteToStream("stream-1-shard", test.count, test.shards)
for i := 0; i < test.count; i++ {
e := <-out
assert.Equal(t, fmt.Sprintf("%d", i), e.Line.Raw)
}
}
}
*/
crowdsec-1.4.6/pkg/acquisition/modules/syslog/ 0000775 0000000 0000000 00000000000 14371172257 0021445 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/pkg/acquisition/modules/syslog/internal/ 0000775 0000000 0000000 00000000000 14371172257 0023261 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/pkg/acquisition/modules/syslog/internal/parser/ 0000775 0000000 0000000 00000000000 14371172257 0024555 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/pkg/acquisition/modules/syslog/internal/parser/rfc3164/ 0000775 0000000 0000000 00000000000 14371172257 0025645 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/pkg/acquisition/modules/syslog/internal/parser/rfc3164/parse.go 0000664 0000000 0000000 00000011102 14371172257 0027301 0 ustar 00root root 0000000 0000000 package rfc3164
import (
"fmt"
"time"
"github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/syslog/internal/parser/utils"
)
type RFC3164Option func(*RFC3164)
type RFC3164 struct {
PRI int
Timestamp time.Time
Hostname string
Tag string
Message string
PID string
//
len int
position int
buf []byte
useCurrentYear bool //If no year is specified in the timestamp, use the current year
strictHostname bool //If the hostname contains invalid characters or is not an IP, return an error
}
const PRI_MAX_LEN = 3
//Order is important: format with the most information must be first because we will stop on the first match
var VALID_TIMESTAMPS = []string{
time.RFC3339,
"Jan 02 15:04:05 2006",
"Jan _2 15:04:05 2006",
"Jan 02 15:04:05",
"Jan _2 15:04:05",
}
func WithCurrentYear() RFC3164Option {
return func(r *RFC3164) {
r.useCurrentYear = true
}
}
func WithStrictHostname() RFC3164Option {
return func(r *RFC3164) {
r.strictHostname = true
}
}
func (r *RFC3164) parsePRI() error {
pri := 0
if r.buf[r.position] != '<' {
return fmt.Errorf("PRI must start with '<'")
}
r.position++
for r.position < r.len {
c := r.buf[r.position]
if c == '>' {
r.position++
break
}
if c < '0' || c > '9' {
return fmt.Errorf("PRI must be a number")
}
pri = pri*10 + int(c-'0')
r.position++
}
if pri > 999 {
return fmt.Errorf("PRI must be up to 3 characters long")
}
if r.position == r.len && r.buf[r.position-1] != '>' {
return fmt.Errorf("PRI must end with '>'")
}
r.PRI = pri
return nil
}
func (r *RFC3164) parseTimestamp() error {
validTs := false
for _, layout := range VALID_TIMESTAMPS {
tsLen := len(layout)
if r.position+tsLen > r.len {
continue
}
t, err := time.Parse(layout, string(r.buf[r.position:r.position+tsLen]))
if err == nil {
validTs = true
r.Timestamp = t
r.position += tsLen
break
}
}
if !validTs {
return fmt.Errorf("timestamp is not valid")
}
if r.useCurrentYear {
if r.Timestamp.Year() == 0 {
r.Timestamp = time.Date(time.Now().Year(), r.Timestamp.Month(), r.Timestamp.Day(), r.Timestamp.Hour(), r.Timestamp.Minute(), r.Timestamp.Second(), r.Timestamp.Nanosecond(), r.Timestamp.Location())
}
}
r.position++
return nil
}
func (r *RFC3164) parseHostname() error {
hostname := []byte{}
for r.position < r.len {
c := r.buf[r.position]
if c == ' ' {
r.position++
break
}
hostname = append(hostname, c)
r.position++
}
if r.strictHostname {
if !utils.IsValidHostnameOrIP(string(hostname)) {
return fmt.Errorf("hostname is not valid")
}
}
if len(hostname) == 0 {
return fmt.Errorf("hostname is empty")
}
r.Hostname = string(hostname)
return nil
}
//We do not enforce tag len as quite a lot of syslog client send tags with more than 32 chars
func (r *RFC3164) parseTag() error {
tag := []byte{}
tmpPid := []byte{}
pidEnd := false
hasPid := false
for r.position < r.len {
c := r.buf[r.position]
if !utils.IsAlphaNumeric(c) {
break
}
tag = append(tag, c)
r.position++
}
if len(tag) == 0 {
return fmt.Errorf("tag is empty")
}
r.Tag = string(tag)
if r.position == r.len {
return nil
}
c := r.buf[r.position]
if c == '[' {
hasPid = true
r.position++
for r.position < r.len {
c = r.buf[r.position]
if c == ']' {
pidEnd = true
r.position++
break
}
if c < '0' || c > '9' {
return fmt.Errorf("pid inside tag must be a number")
}
tmpPid = append(tmpPid, c)
r.position++
}
}
if hasPid && !pidEnd {
return fmt.Errorf("pid inside tag must be closed with ']'")
}
if hasPid {
r.PID = string(tmpPid)
}
return nil
}
func (r *RFC3164) parseMessage() error {
err := r.parseTag()
if err != nil {
return err
}
if r.position == r.len {
return fmt.Errorf("message is empty")
}
c := r.buf[r.position]
if c == ':' {
r.position++
}
for {
if r.position >= r.len {
return fmt.Errorf("message is empty")
}
c := r.buf[r.position]
if c != ' ' {
break
}
r.position++
}
message := r.buf[r.position:r.len]
r.Message = string(message)
return nil
}
func (r *RFC3164) Parse(message []byte) error {
r.len = len(message)
if r.len == 0 {
return fmt.Errorf("message is empty")
}
r.buf = message
err := r.parsePRI()
if err != nil {
return err
}
err = r.parseTimestamp()
if err != nil {
return err
}
err = r.parseHostname()
if err != nil {
return err
}
err = r.parseMessage()
if err != nil {
return err
}
return nil
}
func NewRFC3164Parser(opts ...RFC3164Option) *RFC3164 {
r := &RFC3164{}
for _, opt := range opts {
opt(r)
}
return r
}
crowdsec-1.4.6/pkg/acquisition/modules/syslog/internal/parser/rfc3164/parse_test.go 0000664 0000000 0000000 00000025704 14371172257 0030355 0 ustar 00root root 0000000 0000000 package rfc3164
import (
"testing"
"time"
)
func TestPri(t *testing.T) {
tests := []struct {
input string
expected int
expectedErr string
}{
{"<0>", 0, ""},
{"<19>", 19, ""},
{"<200>", 200, ""},
{"<4999>", 0, "PRI must be up to 3 characters long"},
{"<123", 0, "PRI must end with '>'"},
{"123>", 0, "PRI must start with '<'"},
{"", 0, "PRI must be a number"},
}
for _, test := range tests {
test := test
t.Run(test.input, func(t *testing.T) {
r := &RFC3164{}
r.buf = []byte(test.input)
r.len = len(r.buf)
err := r.parsePRI()
if err != nil {
if test.expectedErr != "" {
if err.Error() != test.expectedErr {
t.Errorf("expected error %s, got %s", test.expectedErr, err)
}
} else {
t.Errorf("unexpected error: %s", err)
}
} else {
if test.expectedErr != "" {
t.Errorf("expected error %s, got no error", test.expectedErr)
} else if r.PRI != test.expected {
t.Errorf("expected %d, got %d", test.expected, r.PRI)
}
}
})
}
}
func TestTimestamp(t *testing.T) {
tests := []struct {
input string
expected string
expectedErr string
currentYear bool
}{
{"May 20 09:33:54", "0000-05-20T09:33:54Z", "", false},
{"May 20 09:33:54", "2023-05-20T09:33:54Z", "", true},
{"May 20 09:33:54 2022", "2022-05-20T09:33:54Z", "", false},
{"May 1 09:33:54 2022", "2022-05-01T09:33:54Z", "", false},
{"May 01 09:33:54 2021", "2021-05-01T09:33:54Z", "", true},
{"foobar", "", "timestamp is not valid", false},
}
for _, test := range tests {
test := test
t.Run(test.input, func(t *testing.T) {
opts := []RFC3164Option{}
if test.currentYear {
opts = append(opts, WithCurrentYear())
}
r := NewRFC3164Parser(opts...)
r.buf = []byte(test.input)
r.len = len(r.buf)
err := r.parseTimestamp()
if err != nil {
if test.expectedErr != "" {
if err.Error() != test.expectedErr {
t.Errorf("expected error %s, got %s", test.expectedErr, err)
}
} else {
t.Errorf("unexpected error: %s", err)
}
} else {
if test.expectedErr != "" {
t.Errorf("expected error %s, got no error", test.expectedErr)
} else if r.Timestamp.Format(time.RFC3339) != test.expected {
t.Errorf("expected %s, got %s", test.expected, r.Timestamp.Format(time.RFC3339))
}
}
})
}
}
func TestHostname(t *testing.T) {
tests := []struct {
input string
expected string
expectedErr string
strictHostname bool
}{
{"127.0.0.1", "127.0.0.1", "", false},
{"::1", "::1", "", false},
{"foo.-bar", "", "hostname is not valid", true},
{"foo-.bar", "", "hostname is not valid", true},
{"foo123.bar", "foo123.bar", "", true},
{"a..", "", "hostname is not valid", true},
{"foo.bar", "foo.bar", "", false},
{"foo,bar", "foo,bar", "", false},
{"foo,bar", "", "hostname is not valid", true},
{"", "", "hostname is empty", false},
{".", ".", "", true},
{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "", "hostname is not valid", true},
{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.bla", "", "hostname is not valid", true},
{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.bla", "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.bla", "", false},
{"a.foo-", "", "hostname is not valid", true},
}
for _, test := range tests {
test := test
t.Run(test.input, func(t *testing.T) {
opts := []RFC3164Option{}
if test.strictHostname {
opts = append(opts, WithStrictHostname())
}
r := NewRFC3164Parser(opts...)
r.buf = []byte(test.input)
r.len = len(r.buf)
err := r.parseHostname()
if err != nil {
if test.expectedErr != "" {
if err.Error() != test.expectedErr {
t.Errorf("expected error %s, got %s", test.expectedErr, err)
}
} else {
t.Errorf("unexpected error: %s", err)
}
} else {
if test.expectedErr != "" {
t.Errorf("expected error %s, got no error", test.expectedErr)
} else if r.Hostname != test.expected {
t.Errorf("expected %s, got %s", test.expected, r.Hostname)
}
}
})
}
}
func TestTag(t *testing.T) {
tests := []struct {
input string
expected string
expectedPID string
expectedErr string
}{
{"foobar", "foobar", "", ""},
{"foobar[42]", "foobar", "42", ""},
{"", "", "", "tag is empty"},
{"foobar[", "", "", "pid inside tag must be closed with ']'"},
{"foobar[42", "", "", "pid inside tag must be closed with ']'"},
{"foobar[asd]", "foobar", "", "pid inside tag must be a number"},
}
for _, test := range tests {
test := test
t.Run(test.input, func(t *testing.T) {
r := &RFC3164{}
r.buf = []byte(test.input)
r.len = len(r.buf)
err := r.parseTag()
if err != nil {
if test.expectedErr != "" {
if err.Error() != test.expectedErr {
t.Errorf("expected error %s, got %s", test.expectedErr, err)
}
} else {
t.Errorf("unexpected error: %s", err)
}
} else {
if test.expectedErr != "" {
t.Errorf("expected error %s, got no error", test.expectedErr)
} else {
if r.Tag != test.expected {
t.Errorf("expected %s, got %s", test.expected, r.Tag)
}
if r.PID != test.expectedPID {
t.Errorf("expected %s, got %s", test.expected, r.Message)
}
}
}
})
}
}
func TestMessage(t *testing.T) {
tests := []struct {
input string
expected string
expectedErr string
}{
{"foobar: pouet", "pouet", ""},
{"foobar[42]: test", "test", ""},
{"foobar[123]: this is a test", "this is a test", ""},
{"foobar[123]: ", "", "message is empty"},
{"foobar[123]:", "", "message is empty"},
}
for _, test := range tests {
test := test
t.Run(test.input, func(t *testing.T) {
r := &RFC3164{}
r.buf = []byte(test.input)
r.len = len(r.buf)
err := r.parseMessage()
if err != nil {
if test.expectedErr != "" {
if err.Error() != test.expectedErr {
t.Errorf("expected error %s, got %s", test.expectedErr, err)
}
} else {
t.Errorf("unexpected error: %s", err)
}
} else {
if test.expectedErr != "" {
t.Errorf("expected error %s, got no error", test.expectedErr)
} else if r.Message != test.expected {
t.Errorf("expected message %s, got %s", test.expected, r.Tag)
}
}
})
}
}
func TestParse(t *testing.T) {
type expected struct {
Timestamp time.Time
Hostname string
Tag string
PID string
Message string
PRI int
}
tests := []struct {
input string
expected expected
expectedErr string
opts []RFC3164Option
}{
{
"<12>May 20 09:33:54 UDMPRO,a2edd0c6ae48,udm-1.10.0.3686 kernel: foo", expected{
Timestamp: time.Date(0, time.May, 20, 9, 33, 54, 0, time.UTC),
Hostname: "UDMPRO,a2edd0c6ae48,udm-1.10.0.3686",
Tag: "kernel",
PID: "",
Message: "foo",
PRI: 12,
}, "", []RFC3164Option{},
},
{
"<12>May 20 09:33:54 UDMPRO,a2edd0c6ae48,udm-1.10.0.3686 kernel: foo", expected{
Timestamp: time.Date(2023, time.May, 20, 9, 33, 54, 0, time.UTC),
Hostname: "UDMPRO,a2edd0c6ae48,udm-1.10.0.3686",
Tag: "kernel",
PID: "",
Message: "foo",
PRI: 12,
}, "", []RFC3164Option{WithCurrentYear()},
},
{
"<12>May 20 09:33:54 UDMPRO,a2edd0c6ae48,udm-1.10.0.3686 kernel: foo", expected{}, "hostname is not valid", []RFC3164Option{WithStrictHostname()},
},
{
"foobar", expected{}, "PRI must start with '<'", []RFC3164Option{},
},
{
"<12>", expected{}, "timestamp is not valid", []RFC3164Option{},
},
{
"<12 May 02 09:33:54 foo.bar", expected{}, "PRI must be a number", []RFC3164Option{},
},
{
"<12>May 02 09:33:54", expected{}, "hostname is empty", []RFC3164Option{},
},
{
"<12>May 02 09:33:54 foo.bar", expected{}, "tag is empty", []RFC3164Option{},
},
{
"<12>May 02 09:33:54 foo.bar bla[42", expected{}, "pid inside tag must be closed with ']'", []RFC3164Option{},
},
{
"<12>May 02 09:33:54 foo.bar bla[42]", expected{}, "message is empty", []RFC3164Option{},
},
{
"<12>May 02 09:33:54 foo.bar bla[42]: ", expected{}, "message is empty", []RFC3164Option{},
},
{
"<12>May 02 09:33:54 foo.bar bla", expected{}, "message is empty", []RFC3164Option{},
},
{
"<12>May 02 09:33:54 foo.bar bla:", expected{}, "message is empty", []RFC3164Option{},
},
{
"", expected{}, "message is empty", []RFC3164Option{},
},
{
`<13>1 2021-05-18T11:58:40.828081+02:00 mantis sshd 49340 - [timeQuality isSynced="0" tzKnown="1"] blabla`, expected{}, "timestamp is not valid", []RFC3164Option{},
},
{
`<46>Jun 2 06:55:39 localhost haproxy[27213]: Connect from 100.100.100.99:52611 to 100.100.100.99:443 (https_shared-merged/HTTP)\\n 10.0.0.1}`, expected{
Timestamp: time.Date(time.Now().Year(), time.June, 2, 6, 55, 39, 0, time.UTC),
Hostname: "localhost",
Tag: "haproxy",
PID: "27213",
Message: `Connect from 100.100.100.99:52611 to 100.100.100.99:443 (https_shared-merged/HTTP)\\n 10.0.0.1}`,
PRI: 46,
}, "", []RFC3164Option{WithCurrentYear()},
},
{
`<46>Jun 2 06:55:39 2022 localhost haproxy[27213]: Connect from 100.100.100.99:52611 to 100.100.100.99:443 (https_shared-merged/HTTP)\\n 10.0.0.1}`, expected{
Timestamp: time.Date(2022, time.June, 2, 6, 55, 39, 0, time.UTC),
Hostname: "localhost",
Tag: "haproxy",
PID: "27213",
Message: `Connect from 100.100.100.99:52611 to 100.100.100.99:443 (https_shared-merged/HTTP)\\n 10.0.0.1}`,
PRI: 46,
}, "", []RFC3164Option{},
},
}
for _, test := range tests {
test := test
t.Run(test.input, func(t *testing.T) {
r := NewRFC3164Parser(test.opts...)
err := r.Parse([]byte(test.input))
if err != nil {
if test.expectedErr != "" {
if err.Error() != test.expectedErr {
t.Errorf("expected error '%s', got '%s'", test.expectedErr, err)
}
} else {
t.Errorf("unexpected error: '%s'", err)
}
} else {
if test.expectedErr != "" {
t.Errorf("expected error '%s', got no error", test.expectedErr)
} else {
if r.Timestamp != test.expected.Timestamp {
t.Errorf("expected timestamp '%s', got '%s'", test.expected.Timestamp, r.Timestamp)
}
if r.Hostname != test.expected.Hostname {
t.Errorf("expected hostname '%s', got '%s'", test.expected.Hostname, r.Hostname)
}
if r.Tag != test.expected.Tag {
t.Errorf("expected tag '%s', got '%s'", test.expected.Tag, r.Tag)
}
if r.PID != test.expected.PID {
t.Errorf("expected pid '%s', got '%s'", test.expected.PID, r.PID)
}
if r.Message != test.expected.Message {
t.Errorf("expected message '%s', got '%s'", test.expected.Message, r.Message)
}
if r.PRI != test.expected.PRI {
t.Errorf("expected pri '%d', got '%d'", test.expected.PRI, r.PRI)
}
}
}
})
}
}
crowdsec-1.4.6/pkg/acquisition/modules/syslog/internal/parser/rfc3164/perf_test.go 0000664 0000000 0000000 00000003116 14371172257 0030170 0 ustar 00root root 0000000 0000000 package rfc3164
import "testing"
var e error
func BenchmarkParse(b *testing.B) {
tests := []struct {
input []byte
opts []RFC3164Option
}{
{
[]byte("<12>May 20 09:33:54 UDMPRO,a2edd0c6ae48,udm-1.10.0.3686 kernel: [1670546.400229] IN=eth9 OUT= MAC=24:5a:4c:7b:0a:4c:34:27:92:67:0f:2b:08:00 SRC=79.124.62.34 DST=x.x.x.x LEN=44 TOS=0x00 PREC=0x00 TTL=243 ID=37520 PROTO=TCP SPT=55055 DPT=51443 WINDOW=1024 RES=0x00 SYN URGP=0"), []RFC3164Option{},
},
{
[]byte("<12>May 20 09:33:54 UDMPRO,a2edd0c6ae48,udm-1.10.0.3686 kernel: foo"), []RFC3164Option{WithCurrentYear()},
},
{
[]byte("<12>May 20 09:33:54 UDMPRO,a2edd0c6ae48,udm-1.10.0.3686 kernel: foo"), []RFC3164Option{WithStrictHostname()},
},
{
[]byte("foobar"), []RFC3164Option{},
},
{
[]byte("<12>"), []RFC3164Option{},
},
{
[]byte("<12>May 02 09:33:54"), []RFC3164Option{},
},
{
[]byte("<12>May 02 09:33:54 foo.bar"), []RFC3164Option{},
},
{
[]byte("<12>May 02 09:33:54 foo.bar bla[42"), []RFC3164Option{},
},
{
[]byte("<12>May 02 09:33:54 foo.bar bla[42]"), []RFC3164Option{},
},
{
[]byte("<12>May 02 09:33:54 foo.bar bla[42]: "), []RFC3164Option{},
},
{
[]byte("<12>May 02 09:33:54 foo.bar bla"), []RFC3164Option{},
},
{
[]byte("<12>May 02 09:33:54 foo.bar bla:"), []RFC3164Option{},
},
{
[]byte(""), []RFC3164Option{},
},
}
var err error
for _, test := range tests {
test := test
b.Run(string(test.input), func(b *testing.B) {
for i := 0; i < b.N; i++ {
r := NewRFC3164Parser(test.opts...)
err = r.Parse(test.input)
}
})
}
e = err
}
crowdsec-1.4.6/pkg/acquisition/modules/syslog/internal/parser/rfc5424/ 0000775 0000000 0000000 00000000000 14371172257 0025646 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/pkg/acquisition/modules/syslog/internal/parser/rfc5424/parse.go 0000664 0000000 0000000 00000014731 14371172257 0027315 0 ustar 00root root 0000000 0000000 package rfc5424
import (
"fmt"
"time"
"github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/syslog/internal/parser/utils"
)
type RFC5424Option func(*RFC5424)
type RFC5424 struct {
PRI int
Timestamp time.Time
Hostname string
Tag string
Message string
PID string
MsgID string
//
len int
position int
buf []byte
useCurrentYear bool //If no year is specified in the timestamp, use the current year
strictHostname bool //If the hostname contains invalid characters or is not an IP, return an error
}
const PRI_MAX_LEN = 3
const NIL_VALUE = '-'
var VALID_TIMESTAMPS = []string{
time.RFC3339,
}
const VALID_TIMESTAMP = time.RFC3339Nano
func WithCurrentYear() RFC5424Option {
return func(r *RFC5424) {
r.useCurrentYear = true
}
}
func WithStrictHostname() RFC5424Option {
return func(r *RFC5424) {
r.strictHostname = true
}
}
func (r *RFC5424) parsePRI() error {
pri := 0
if r.buf[r.position] != '<' {
return fmt.Errorf("PRI must start with '<'")
}
r.position++
for r.position < r.len {
c := r.buf[r.position]
if c == '>' {
r.position++
break
}
if c < '0' || c > '9' {
return fmt.Errorf("PRI must be a number")
}
pri = pri*10 + int(c-'0')
r.position++
}
if pri > 999 {
return fmt.Errorf("PRI must be up to 3 characters long")
}
if r.position == r.len && r.buf[r.position-1] != '>' {
return fmt.Errorf("PRI must end with '>'")
}
r.PRI = pri
return nil
}
func (r *RFC5424) parseVersion() error {
if r.buf[r.position] != '1' {
return fmt.Errorf("version must be 1")
}
r.position += 2
if r.position >= r.len {
return fmt.Errorf("version must be followed by a space")
}
return nil
}
func (r *RFC5424) parseTimestamp() error {
timestamp := []byte{}
if r.buf[r.position] == NIL_VALUE {
r.Timestamp = time.Now().UTC().Round(0)
r.position += 2
return nil
}
for r.position < r.len {
c := r.buf[r.position]
if c == ' ' {
break
}
timestamp = append(timestamp, c)
r.position++
}
if len(timestamp) == 0 {
return fmt.Errorf("timestamp is empty")
}
if r.position == r.len {
return fmt.Errorf("EOL after timestamp")
}
date, err := time.Parse(VALID_TIMESTAMP, string(timestamp))
if err != nil {
return fmt.Errorf("timestamp is not valid")
}
r.Timestamp = date
r.position++
if r.position >= r.len {
return fmt.Errorf("EOL after timestamp")
}
return nil
}
func (r *RFC5424) parseHostname() error {
if r.buf[r.position] == NIL_VALUE {
r.Hostname = ""
r.position += 2
return nil
}
hostname := []byte{}
for r.position < r.len {
c := r.buf[r.position]
if c == ' ' {
r.position++
break
}
hostname = append(hostname, c)
r.position++
}
if r.strictHostname {
if !utils.IsValidHostnameOrIP(string(hostname)) {
return fmt.Errorf("hostname is not valid")
}
}
if len(hostname) == 0 {
return fmt.Errorf("hostname is empty")
}
r.Hostname = string(hostname)
return nil
}
func (r *RFC5424) parseAppName() error {
if r.buf[r.position] == NIL_VALUE {
r.Tag = ""
r.position += 2
return nil
}
appname := []byte{}
for r.position < r.len {
c := r.buf[r.position]
if c == ' ' {
r.position++
break
}
appname = append(appname, c)
r.position++
}
if len(appname) == 0 {
return fmt.Errorf("appname is empty")
}
if len(appname) > 48 {
return fmt.Errorf("appname is too long")
}
r.Tag = string(appname)
return nil
}
func (r *RFC5424) parseProcID() error {
if r.buf[r.position] == NIL_VALUE {
r.PID = ""
r.position += 2
return nil
}
procid := []byte{}
for r.position < r.len {
c := r.buf[r.position]
if c == ' ' {
r.position++
break
}
procid = append(procid, c)
r.position++
}
if len(procid) == 0 {
return fmt.Errorf("procid is empty")
}
if len(procid) > 128 {
return fmt.Errorf("procid is too long")
}
r.PID = string(procid)
return nil
}
func (r *RFC5424) parseMsgID() error {
if r.buf[r.position] == NIL_VALUE {
r.MsgID = ""
r.position += 2
return nil
}
msgid := []byte{}
for r.position < r.len {
c := r.buf[r.position]
if c == ' ' {
r.position++
break
}
msgid = append(msgid, c)
r.position++
}
if len(msgid) == 0 {
return fmt.Errorf("msgid is empty")
}
if len(msgid) > 32 {
return fmt.Errorf("msgid is too long")
}
r.MsgID = string(msgid)
return nil
}
func (r *RFC5424) parseStructuredData() error {
done := false
if r.buf[r.position] == NIL_VALUE {
r.position += 2
return nil
}
if r.buf[r.position] != '[' {
return fmt.Errorf("structured data must start with '[' or be '-'")
}
prev := byte(0)
for r.position < r.len {
done = false
c := r.buf[r.position]
if c == ']' && prev != '\\' {
done = true
r.position++
if r.position < r.len && r.buf[r.position] == ' ' {
break
}
}
prev = c
r.position++
}
r.position++
if !done {
return fmt.Errorf("structured data must end with ']'")
}
return nil
}
func (r *RFC5424) parseMessage() error {
if r.position == r.len {
return fmt.Errorf("message is empty")
}
message := []byte{}
for r.position < r.len {
c := r.buf[r.position]
message = append(message, c)
r.position++
}
r.Message = string(message)
return nil
}
func (r *RFC5424) Parse(message []byte) error {
r.len = len(message)
if r.len == 0 {
return fmt.Errorf("syslog line is empty")
}
r.buf = message
err := r.parsePRI()
if err != nil {
return err
}
if r.position >= r.len {
return fmt.Errorf("EOL after PRI")
}
err = r.parseVersion()
if err != nil {
return err
}
if r.position >= r.len {
return fmt.Errorf("EOL after Version")
}
err = r.parseTimestamp()
if err != nil {
return err
}
if r.position >= r.len {
return fmt.Errorf("EOL after Timestamp")
}
err = r.parseHostname()
if err != nil {
return err
}
if r.position >= r.len {
return fmt.Errorf("EOL after hostname")
}
err = r.parseAppName()
if err != nil {
return err
}
if r.position >= r.len {
return fmt.Errorf("EOL after appname")
}
err = r.parseProcID()
if err != nil {
return err
}
if r.position >= r.len {
return fmt.Errorf("EOL after ProcID")
}
err = r.parseMsgID()
if err != nil {
return err
}
if r.position >= r.len {
return fmt.Errorf("EOL after MSGID")
}
err = r.parseStructuredData()
if err != nil {
return err
}
if r.position >= r.len {
return fmt.Errorf("EOL after SD")
}
err = r.parseMessage()
if err != nil {
return err
}
return nil
}
func NewRFC5424Parser(opts ...RFC5424Option) *RFC5424 {
r := &RFC5424{}
for _, opt := range opts {
opt(r)
}
return r
}
crowdsec-1.4.6/pkg/acquisition/modules/syslog/internal/parser/rfc5424/parse_test.go 0000664 0000000 0000000 00000025252 14371172257 0030354 0 ustar 00root root 0000000 0000000 package rfc5424
import (
"testing"
"time"
)
func TestPri(t *testing.T) {
tests := []struct {
input string
expected int
expectedErr string
}{
{"<0>", 0, ""},
{"<19>", 19, ""},
{"<200>", 200, ""},
{"<4999>", 0, "PRI must be up to 3 characters long"},
{"<123", 0, "PRI must end with '>'"},
{"123>", 0, "PRI must start with '<'"},
{"", 0, "PRI must be a number"},
}
for _, test := range tests {
test := test
t.Run(test.input, func(t *testing.T) {
r := &RFC5424{}
r.buf = []byte(test.input)
r.len = len(r.buf)
err := r.parsePRI()
if err != nil {
if test.expectedErr != "" {
if err.Error() != test.expectedErr {
t.Errorf("expected error %s, got %s", test.expectedErr, err)
}
} else {
t.Errorf("unexpected error: %s", err)
}
} else {
if test.expectedErr != "" {
t.Errorf("expected error %s, got no error", test.expectedErr)
} else if r.PRI != test.expected {
t.Errorf("expected %d, got %d", test.expected, r.PRI)
}
}
})
}
}
func TestHostname(t *testing.T) {
tests := []struct {
input string
expected string
expectedErr string
strictHostname bool
}{
{"127.0.0.1", "127.0.0.1", "", false},
{"::1", "::1", "", false},
{"-", "", "", false},
{"foo.-bar", "", "hostname is not valid", true},
{"foo-.bar", "", "hostname is not valid", true},
{"foo123.bar", "foo123.bar", "", true},
{"a..", "", "hostname is not valid", true},
{"foo.bar", "foo.bar", "", false},
{"foo,bar", "foo,bar", "", false},
{"foo,bar", "", "hostname is not valid", true},
{".", ".", "", true},
{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "", "hostname is not valid", true},
{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.bla", "", "hostname is not valid", true},
{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.bla", "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.bla", "", false},
{"a.foo-", "", "hostname is not valid", true},
}
for _, test := range tests {
test := test
t.Run(test.input, func(t *testing.T) {
opts := []RFC5424Option{}
if test.strictHostname {
opts = append(opts, WithStrictHostname())
}
r := NewRFC5424Parser(opts...)
r.buf = []byte(test.input)
r.len = len(r.buf)
err := r.parseHostname()
if err != nil {
if test.expectedErr != "" {
if err.Error() != test.expectedErr {
t.Errorf("expected error %s, got %s", test.expectedErr, err)
}
} else {
t.Errorf("unexpected error: %s", err)
}
} else {
if test.expectedErr != "" {
t.Errorf("expected error %s, got no error", test.expectedErr)
} else if r.Hostname != test.expected {
t.Errorf("expected %s, got %s", test.expected, r.Hostname)
}
}
})
}
}
func TestParse(t *testing.T) {
type expected struct {
Timestamp time.Time
Hostname string
Tag string
PID string
Message string
PRI int
MsgID string
}
tests := []struct {
name string
input string
expected expected
expectedErr string
opts []RFC5424Option
}{
{
"valid msg",
`<13>1 2021-05-18T11:58:40.828081+02:42 mantis sshd 49340 - [timeQuality isSynced="0" tzKnown="1"] blabla`, expected{
Timestamp: time.Date(2021, 5, 18, 11, 58, 40, 828081000, time.FixedZone("+0242", 9720)),
Hostname: "mantis",
Tag: "sshd",
PID: "49340",
MsgID: "",
Message: "blabla",
PRI: 13,
}, "", []RFC5424Option{},
},
{
"valid msg with msgid",
`<13>1 2021-05-18T11:58:40.828081+02:42 mantis foobar 49340 123123 [timeQuality isSynced="0" tzKnown="1"] blabla`, expected{
Timestamp: time.Date(2021, 5, 18, 11, 58, 40, 828081000, time.FixedZone("+0242", 9720)),
Hostname: "mantis",
Tag: "foobar",
PID: "49340",
MsgID: "123123",
Message: "blabla",
PRI: 13,
}, "", []RFC5424Option{},
},
{
"valid msg with repeating SD",
`<13>1 2021-05-18T11:58:40.828081+02:42 mantis foobar 49340 123123 [timeQuality isSynced="0" tzKnown="1"][foo="bar][a] blabla`, expected{
Timestamp: time.Date(2021, 5, 18, 11, 58, 40, 828081000, time.FixedZone("+0242", 9720)),
Hostname: "mantis",
Tag: "foobar",
PID: "49340",
MsgID: "123123",
Message: "blabla",
PRI: 13,
}, "", []RFC5424Option{},
},
{
"invalid SD",
`<13>1 2021-05-18T11:58:40.828081+02:00 mantis foobar 49340 123123 [timeQuality asd`, expected{}, "structured data must end with ']'", []RFC5424Option{},
},
{
"invalid version",
`<13>42 2021-05-18T11:58:40.828081+02:00 mantis foobar 49340 123123 [timeQuality isSynced="0" tzKnown="1"] blabla`, expected{}, "version must be 1", []RFC5424Option{},
},
{
"invalid message",
`<13>1`, expected{}, "version must be followed by a space", []RFC5424Option{},
},
{
"valid msg with empty fields",
`<13>1 - foo - - - - blabla`, expected{
Timestamp: time.Now().UTC().Round(0),
Hostname: "foo",
PRI: 13,
Message: "blabla",
}, "", []RFC5424Option{},
},
{
"valid msg with empty fields",
`<13>1 - - - - - - blabla`, expected{
Timestamp: time.Now().UTC().Round(0),
PRI: 13,
Message: "blabla",
}, "", []RFC5424Option{},
},
{
"valid msg with escaped SD",
`<13>1 2022-05-24T10:57:39Z testhostname unknown - sn="msgid" [foo="\]" bar="a\""][a b="[\]" c] testmessage`,
expected{
PRI: 13,
Timestamp: time.Date(2022, 5, 24, 10, 57, 39, 0, time.UTC),
Tag: "unknown",
Hostname: "testhostname",
MsgID: `sn="msgid"`,
Message: `testmessage`,
}, "", []RFC5424Option{},
},
{
"valid complex msg",
`<13>1 2022-05-24T10:57:39Z myhostname unknown - sn="msgid" [all@0 request="/dist/precache-manifest.58b57debe6bc4f96698da0dc314461e9.js" src_ip_geo_country="DE" MONTH="May" COMMONAPACHELOG="1.1.1.1 - - [24/May/2022:10:57:37 +0200\] \"GET /dist/precache-manifest.58b57debe6bc4f96698da0dc314461e9.js HTTP/2.0\" 304 0" auth="-" HOUR="10" gl2_remote_ip="172.31.32.142" ident="-" gl2_remote_port="43375" BASE10NUM="[2.0, 304, 0\]" pid="-1" program="nginx" gl2_source_input="623ed3440183476d61cff974" INT="+0200" is_private_ip="false" YEAR="2022" src_ip_geo_city="Achern" clientip="1.1.1.1" USERNAME="-" src_ip_geo_location="48.6306,8.0743" gl2_source_node="8620c2bb-dbb7-4535-b1ce-83df223acd8d" MINUTE="57" timestamp="2022-05-24T08:57:37.000Z" src_ip_asn="3320" level="5" IP="1.1.1.1" IPV4="1.1.1.1" verb="GET" gl2_message_id="01G3TMJFAMFS4H60QSF7M029R0" TIME="10:57:37" USER="-" src_ip_asn_owner="Deutsche Telekom AG" response="304" bytes="0" SECOND="37" httpversion="2.0" _id="906ce155-db3f-11ec-b25f-0a189ba2c64e" facility="user" MONTHDAY="24"] source: sn="www.foobar.com" | message: 1.1.1.1 - - [24/May/2022:10:57:37 +0200] "GET /dist/precache-manifest.58b57debe6bc4f96698da0dc314461e9.js HTTP/2.0" 304 0 "https://www.foobar.com/sw.js" "Mozilla/5.0 (Linux; Android 9; ANE-LX1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.61 Mobile Safari/537.36" "-" "www.foobar.com" sn="www.foobar.com" rt=0.000 ua="-" us="-" ut="-" ul="-" cs=HIT { request: /dist/precache-manifest.58b57debe6bc4f96698da0dc314461e9.js | src_ip_geo_country: DE | MONTH: May | COMMONAPACHELOG: 1.1.1.1 - - [24/May/2022:10:57:37 +0200] "GET /dist/precache-manifest.58b57debe6bc4f96698da0dc314461e9.js HTTP/2.0" 304 0 | auth: - | HOUR: 10 | gl2_remote_ip: 172.31.32.142 | ident: - | gl2_remote_port: 43375 | BASE10NUM: [2.0, 304, 0] | pid: -1 | program: nginx | gl2_source_input: 623ed3440183476d61cff974 | INT: +0200 | is_private_ip: false | YEAR: 2022 | src_ip_geo_city: Achern | clientip: 1.1.1.1 | USERNAME:`,
expected{
Timestamp: time.Date(2022, 5, 24, 10, 57, 39, 0, time.UTC),
Hostname: "myhostname",
Tag: "unknown",
PRI: 13,
MsgID: `sn="msgid"`,
Message: `source: sn="www.foobar.com" | message: 1.1.1.1 - - [24/May/2022:10:57:37 +0200] "GET /dist/precache-manifest.58b57debe6bc4f96698da0dc314461e9.js HTTP/2.0" 304 0 "https://www.foobar.com/sw.js" "Mozilla/5.0 (Linux; Android 9; ANE-LX1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.61 Mobile Safari/537.36" "-" "www.foobar.com" sn="www.foobar.com" rt=0.000 ua="-" us="-" ut="-" ul="-" cs=HIT { request: /dist/precache-manifest.58b57debe6bc4f96698da0dc314461e9.js | src_ip_geo_country: DE | MONTH: May | COMMONAPACHELOG: 1.1.1.1 - - [24/May/2022:10:57:37 +0200] "GET /dist/precache-manifest.58b57debe6bc4f96698da0dc314461e9.js HTTP/2.0" 304 0 | auth: - | HOUR: 10 | gl2_remote_ip: 172.31.32.142 | ident: - | gl2_remote_port: 43375 | BASE10NUM: [2.0, 304, 0] | pid: -1 | program: nginx | gl2_source_input: 623ed3440183476d61cff974 | INT: +0200 | is_private_ip: false | YEAR: 2022 | src_ip_geo_city: Achern | clientip: 1.1.1.1 | USERNAME:`,
}, "", []RFC5424Option{},
},
{
"partial message",
`<13>1 2022-05-24T10:57:39Z foo bar -`,
expected{},
"EOL after ProcID",
[]RFC5424Option{},
},
{
"partial message",
`<13>1 2022-05-24T10:57:39Z foo bar `,
expected{},
"EOL after appname",
[]RFC5424Option{},
},
}
for _, test := range tests {
test := test
t.Run(test.name, func(t *testing.T) {
r := NewRFC5424Parser(test.opts...)
err := r.Parse([]byte(test.input))
if err != nil {
if test.expectedErr != "" {
if err.Error() != test.expectedErr {
t.Errorf("expected error '%s', got '%s'", test.expectedErr, err)
}
} else {
t.Errorf("unexpected error: '%s'", err)
}
} else {
if test.expectedErr != "" {
t.Errorf("expected error '%s', got no error", test.expectedErr)
} else {
if r.Timestamp.Round(time.Second).String() != test.expected.Timestamp.Round(time.Second).String() {
t.Errorf("expected timestamp '%s', got '%s'", test.expected.Timestamp, r.Timestamp)
}
if r.Hostname != test.expected.Hostname {
t.Errorf("expected hostname '%s', got '%s'", test.expected.Hostname, r.Hostname)
}
if r.Tag != test.expected.Tag {
t.Errorf("expected tag '%s', got '%s'", test.expected.Tag, r.Tag)
}
if r.PID != test.expected.PID {
t.Errorf("expected pid '%s', got '%s'", test.expected.PID, r.PID)
}
if r.Message != test.expected.Message {
t.Errorf("expected message '%s', got '%s'", test.expected.Message, r.Message)
}
if r.PRI != test.expected.PRI {
t.Errorf("expected pri '%d', got '%d'", test.expected.PRI, r.PRI)
}
if r.MsgID != test.expected.MsgID {
t.Errorf("expected msgid '%s', got '%s'", test.expected.MsgID, r.MsgID)
}
}
}
})
}
}
crowdsec-1.4.6/pkg/acquisition/modules/syslog/internal/parser/rfc5424/perf_test.go 0000664 0000000 0000000 00000012503 14371172257 0030171 0 ustar 00root root 0000000 0000000 package rfc5424
import "testing"
var e error
const BOM = "\xEF\xBB\xBF"
//Test case are from https://github.com/influxdata/go-syslog (the parser we used previously)
func BenchmarkParse(b *testing.B) {
tests := []struct {
label string
input []byte
}{
{
label: "[no] empty input",
input: []byte(``),
},
{
label: "[no] multiple syslog messages on multiple lines",
input: []byte("<1>1 - - - - - -\x0A<2>1 - - - - - -"),
},
{
label: "[no] impossible timestamp",
input: []byte(`<101>11 2003-09-31T22:14:15.003Z`),
},
{
label: "[no] malformed structured data",
input: []byte("<1>1 - - - - - X"),
},
{
label: "[no] with duplicated structured data id",
input: []byte("<165>3 2003-10-11T22:14:15.003Z example.com evnts - ID27 [id1][id1]"),
},
{
label: "[ok] minimal",
input: []byte(`<1>1 - - - - - -`),
},
{
label: "[ok] average message",
input: []byte(`<29>1 2016-02-21T04:32:57+00:00 web1 someservice - - [origin x-service="someservice"][meta sequenceId="14125553"] 127.0.0.1 - - 1456029177 "GET /v1/ok HTTP/1.1" 200 145 "-" "hacheck 0.9.0" 24306 127.0.0.1:40124 575`),
},
{
label: "[ok] complicated message",
input: []byte(`<78>1 2016-01-15T00:04:01Z host1 CROND 10391 - [meta sequenceId="29" sequenceBlah="foo"][my key="value"] some_message`),
},
{
label: "[ok] very long message",
input: []byte(`<190>1 2016-02-21T01:19:11+00:00 batch6sj - - - [meta sequenceId="21881798" x-group="37051387"][origin x-service="tracking"] metascutellar conversationalist nephralgic exogenetic graphy streng outtaken acouasm amateurism prenotice Lyonese bedull antigrammatical diosphenol gastriloquial bayoneteer sweetener naggy roughhouser dighter addend sulphacid uneffectless ferroprussiate reveal Mazdaist plaudite Australasian distributival wiseman rumness Seidel topazine shahdom sinsion mesmerically pinguedinous ophthalmotonometer scuppler wound eciliate expectedly carriwitchet dictatorialism bindweb pyelitic idic atule kokoon poultryproof rusticial seedlip nitrosate splenadenoma holobenthic uneternal Phocaean epigenic doubtlessly indirection torticollar robomb adoptedly outspeak wappenschawing talalgia Goop domitic savola unstrafed carded unmagnified mythologically orchester obliteration imperialine undisobeyed galvanoplastical cycloplegia quinquennia foremean umbonal marcgraviaceous happenstance theoretical necropoles wayworn Igbira pseudoangelic raising unfrounced lamasary centaurial Japanolatry microlepidoptera`),
},
{
label: "[ok] all max length and complete",
input: []byte(`<191>999 2018-12-31T23:59:59.999999-23:59 abcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabc abcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdef abcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzab abcdefghilmnopqrstuvzabcdefghilm [an@id key1="val1" key2="val2"][another@id key1="val1"] Some message "GET"`),
},
{
label: "[ok] all max length except structured data and message",
input: []byte(`<191>999 2018-12-31T23:59:59.999999-23:59 abcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabc abcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdef abcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzabcdefghilmnopqrstuvzab abcdefghilmnopqrstuvzabcdefghilm -`),
},
{
label: "[ok] minimal with message containing newline",
input: []byte("<1>1 - - - - - - x\x0Ay"),
},
{
label: "[ok] w/o procid, w/o structured data, with message starting with BOM",
input: []byte("<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - " + BOM + "'su root' failed for lonvick on /dev/pts/8"),
},
{
label: "[ok] minimal with UTF-8 message",
input: []byte("<0>1 - - - - - - â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â Ĩâ â â â â "),
},
{
label: "[ok] minimal with UTF-8 message starting with BOM",
input: []byte("<0>1 - - - - - - " + BOM + "â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â Ĩâ â â â â "),
},
{
label: "[ok] with structured data id, w/o structured data params",
input: []byte(`<29>50 2016-01-15T01:00:43Z hn S - - [my@id]`),
},
{
label: "[ok] with multiple structured data",
input: []byte(`<29>50 2016-01-15T01:00:43Z hn S - - [my@id1 k="v"][my@id2 c="val"]`),
},
{
label: "[ok] with escaped backslash within structured data param value, with message",
input: []byte(`<29>50 2016-01-15T01:00:43Z hn S - - [meta es="\\valid"] 1452819643`),
},
{
label: "[ok] with UTF-8 structured data param value, with message",
input: []byte(`<78>1 2016-01-15T00:04:01+00:00 host1 CROND 10391 - [sdid x="â"] some_message`),
},
}
var err error
for _, test := range tests {
test := test
b.Run(test.label, func(b *testing.B) {
for i := 0; i < b.N; i++ {
r := NewRFC5424Parser()
err = r.Parse(test.input)
}
})
}
e = err
}
crowdsec-1.4.6/pkg/acquisition/modules/syslog/internal/parser/utils/ 0000775 0000000 0000000 00000000000 14371172257 0025715 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/pkg/acquisition/modules/syslog/internal/parser/utils/utils.go 0000664 0000000 0000000 00000003506 14371172257 0027410 0 ustar 00root root 0000000 0000000 package utils
import "net"
func isValidIP(ip string) bool {
return net.ParseIP(ip) != nil
}
func IsAlphaNumeric(c byte) bool {
return 'a' <= c && c <= 'z' || 'A' <= c && c <= 'Z' || '0' <= c && c <= '9'
}
//This function is lifted from go source
//See https://github.com/golang/go/blob/master/src/net/dnsclient.go#L75
func isValidHostname(s string) bool {
// The root domain name is valid. See golang.org/issue/45715.
if s == "." {
return true
}
// See RFC 1035, RFC 3696.
// Presentation format has dots before every label except the first, and the
// terminal empty label is optional here because we assume fully-qualified
// (absolute) input. We must therefore reserve space for the first and last
// labels' length octets in wire format, where they are necessary and the
// maximum total length is 255.
// So our _effective_ maximum is 253, but 254 is not rejected if the last
// character is a dot.
l := len(s)
if l == 0 || l > 254 || l == 254 && s[l-1] != '.' {
return false
}
last := byte('.')
nonNumeric := false // true once we've seen a letter or hyphen
partlen := 0
for i := 0; i < len(s); i++ {
c := s[i]
switch {
default:
return false
case 'a' <= c && c <= 'z' || 'A' <= c && c <= 'Z' || c == '_':
nonNumeric = true
partlen++
case '0' <= c && c <= '9':
// fine
partlen++
case c == '-':
// Byte before dash cannot be dot.
if last == '.' {
return false
}
partlen++
nonNumeric = true
case c == '.':
// Byte before dot cannot be dot, dash.
if last == '.' || last == '-' {
return false
}
if partlen > 63 || partlen == 0 {
return false
}
partlen = 0
}
last = c
}
if last == '-' || partlen > 63 {
return false
}
return nonNumeric
}
func IsValidHostnameOrIP(hostname string) bool {
return isValidIP(hostname) || isValidHostname(hostname)
}
crowdsec-1.4.6/pkg/acquisition/modules/syslog/internal/server/ 0000775 0000000 0000000 00000000000 14371172257 0024567 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/pkg/acquisition/modules/syslog/internal/server/syslogserver.go 0000664 0000000 0000000 00000004237 14371172257 0027673 0 ustar 00root root 0000000 0000000 package syslogserver
import (
"fmt"
"net"
"strings"
"time"
"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
"gopkg.in/tomb.v2"
)
type SyslogServer struct {
listenAddr string
port int
channel chan SyslogMessage
udpConn *net.UDPConn
Logger *log.Entry
MaxMessageLen int
}
type SyslogMessage struct {
Message []byte
Client string
}
func (s *SyslogServer) Listen(listenAddr string, port int) error {
s.listenAddr = listenAddr
s.port = port
udpAddr, err := net.ResolveUDPAddr("udp", fmt.Sprintf("%s:%d", s.listenAddr, s.port))
if err != nil {
return errors.Wrapf(err, "could not resolve addr %s", s.listenAddr)
}
udpConn, err := net.ListenUDP("udp", udpAddr)
if err != nil {
return errors.Wrapf(err, "could not listen on port %d", s.port)
}
s.Logger.Debugf("listening on %s:%d", s.listenAddr, s.port)
s.udpConn = udpConn
err = s.udpConn.SetReadDeadline(time.Now().UTC().Add(100 * time.Millisecond))
if err != nil {
return errors.Wrap(err, "could not set read deadline on UDP socket")
}
return nil
}
func (s *SyslogServer) SetChannel(c chan SyslogMessage) {
s.channel = c
}
func (s *SyslogServer) StartServer() *tomb.Tomb {
t := tomb.Tomb{}
t.Go(func() error {
for {
select {
case <-t.Dying():
s.Logger.Info("Syslog server tomb is dying")
err := s.KillServer()
return err
default:
//RFC3164 says 1024 bytes max
//RFC5424 says 480 bytes minimum, and should support up to 2048 bytes
b := make([]byte, s.MaxMessageLen)
n, addr, err := s.udpConn.ReadFrom(b)
if err != nil && !strings.Contains(err.Error(), "i/o timeout") {
s.Logger.Errorf("error while reading from socket : %s", err)
s.udpConn.Close()
return err
}
if err == nil {
s.channel <- SyslogMessage{Message: b[:n], Client: strings.Split(addr.String(), ":")[0]}
}
err = s.udpConn.SetReadDeadline(time.Now().UTC().Add(100 * time.Millisecond))
if err != nil {
return err
}
}
}
})
return &t
}
func (s *SyslogServer) KillServer() error {
err := s.udpConn.Close()
if err != nil {
return errors.Wrap(err, "could not close UDP connection")
}
close(s.channel)
return nil
}
crowdsec-1.4.6/pkg/acquisition/modules/syslog/syslog.go 0000664 0000000 0000000 00000014225 14371172257 0023320 0 ustar 00root root 0000000 0000000 package syslogacquisition
import (
"fmt"
"net"
"strings"
"time"
"github.com/crowdsecurity/crowdsec/pkg/acquisition/configuration"
"github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/syslog/internal/parser/rfc3164"
"github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/syslog/internal/parser/rfc5424"
syslogserver "github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/syslog/internal/server"
leaky "github.com/crowdsecurity/crowdsec/pkg/leakybucket"
"github.com/crowdsecurity/crowdsec/pkg/types"
"github.com/pkg/errors"
"github.com/prometheus/client_golang/prometheus"
log "github.com/sirupsen/logrus"
"gopkg.in/tomb.v2"
"gopkg.in/yaml.v2"
)
type SyslogConfiguration struct {
Proto string `yaml:"protocol,omitempty"`
Port int `yaml:"listen_port,omitempty"`
Addr string `yaml:"listen_addr,omitempty"`
MaxMessageLen int `yaml:"max_message_len,omitempty"`
configuration.DataSourceCommonCfg `yaml:",inline"`
}
type SyslogSource struct {
config SyslogConfiguration
logger *log.Entry
server *syslogserver.SyslogServer
serverTomb *tomb.Tomb
}
var linesReceived = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "cs_syslogsource_hits_total",
Help: "Total lines that were received.",
},
[]string{"source"})
var linesParsed = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "cs_syslogsource_parsed_total",
Help: "Total lines that were successfully parsed",
},
[]string{"source", "type"})
func (s *SyslogSource) GetName() string {
return "syslog"
}
func (s *SyslogSource) GetMode() string {
return s.config.Mode
}
func (s *SyslogSource) Dump() interface{} {
return s
}
func (s *SyslogSource) CanRun() error {
return nil
}
func (s *SyslogSource) GetMetrics() []prometheus.Collector {
return []prometheus.Collector{linesReceived, linesParsed}
}
func (s *SyslogSource) GetAggregMetrics() []prometheus.Collector {
return []prometheus.Collector{linesReceived, linesParsed}
}
func (s *SyslogSource) ConfigureByDSN(dsn string, labels map[string]string, logger *log.Entry) error {
return fmt.Errorf("syslog datasource does not support one shot acquisition")
}
func (s *SyslogSource) OneShotAcquisition(out chan types.Event, t *tomb.Tomb) error {
return fmt.Errorf("syslog datasource does not support one shot acquisition")
}
func validatePort(port int) bool {
return port > 0 && port <= 65535
}
func validateAddr(addr string) bool {
return net.ParseIP(addr) != nil
}
func (s *SyslogSource) Configure(yamlConfig []byte, logger *log.Entry) error {
s.logger = logger
s.logger.Infof("Starting syslog datasource configuration")
syslogConfig := SyslogConfiguration{}
syslogConfig.Mode = configuration.TAIL_MODE
err := yaml.UnmarshalStrict(yamlConfig, &syslogConfig)
if err != nil {
return errors.Wrap(err, "Cannot parse syslog configuration")
}
if syslogConfig.Addr == "" {
syslogConfig.Addr = "127.0.0.1" //do we want a usable or secure default ?
}
if syslogConfig.Port == 0 {
syslogConfig.Port = 514
}
if syslogConfig.MaxMessageLen == 0 {
syslogConfig.MaxMessageLen = 2048
}
if !validatePort(syslogConfig.Port) {
return fmt.Errorf("invalid port %d", syslogConfig.Port)
}
if !validateAddr(syslogConfig.Addr) {
return fmt.Errorf("invalid listen IP %s", syslogConfig.Addr)
}
s.config = syslogConfig
return nil
}
func (s *SyslogSource) StreamingAcquisition(out chan types.Event, t *tomb.Tomb) error {
c := make(chan syslogserver.SyslogMessage)
s.server = &syslogserver.SyslogServer{Logger: s.logger.WithField("syslog", "internal"), MaxMessageLen: s.config.MaxMessageLen}
s.server.SetChannel(c)
err := s.server.Listen(s.config.Addr, s.config.Port)
if err != nil {
return errors.Wrap(err, "could not start syslog server")
}
s.serverTomb = s.server.StartServer()
t.Go(func() error {
defer types.CatchPanic("crowdsec/acquis/syslog/live")
return s.handleSyslogMsg(out, t, c)
})
return nil
}
func (s *SyslogSource) buildLogFromSyslog(ts time.Time, hostname string,
appname string, pid string, msg string) string {
ret := ""
if !ts.IsZero() {
ret += ts.Format("Jan 2 15:04:05")
} else {
s.logger.Tracef("%s - missing TS", msg)
ret += time.Now().UTC().Format("Jan 2 15:04:05")
}
if hostname != "" {
ret += " " + hostname
} else {
s.logger.Tracef("%s - missing host", msg)
ret += " unknownhost"
}
if appname != "" {
ret += " " + appname
}
if pid != "" {
ret += "[" + pid + "]: "
} else {
ret += ": "
}
if msg != "" {
ret += msg
}
return ret
}
func (s *SyslogSource) handleSyslogMsg(out chan types.Event, t *tomb.Tomb, c chan syslogserver.SyslogMessage) error {
killed := false
for {
select {
case <-t.Dying():
if !killed {
s.logger.Info("Syslog datasource is dying")
s.serverTomb.Kill(nil)
killed = true
}
case <-s.serverTomb.Dead():
s.logger.Info("Syslog server has exited")
return nil
case syslogLine := <-c:
var line string
var ts time.Time
logger := s.logger.WithField("client", syslogLine.Client)
logger.Tracef("raw: %s", syslogLine)
linesReceived.With(prometheus.Labels{"source": syslogLine.Client}).Inc()
p := rfc3164.NewRFC3164Parser(rfc3164.WithCurrentYear())
err := p.Parse(syslogLine.Message)
if err != nil {
logger.Debugf("could not parse as RFC3164 (%s)", err)
p2 := rfc5424.NewRFC5424Parser()
err = p2.Parse(syslogLine.Message)
if err != nil {
logger.Errorf("could not parse message: %s", err)
logger.Debugf("could not parse as RFC5424 (%s) : %s", err, syslogLine.Message)
continue
}
line = s.buildLogFromSyslog(p2.Timestamp, p2.Hostname, p2.Tag, p2.PID, p2.Message)
} else {
line = s.buildLogFromSyslog(p.Timestamp, p.Hostname, p.Tag, p.PID, p.Message)
}
line = strings.TrimSuffix(line, "\n")
l := types.Line{}
l.Raw = line
l.Module = s.GetName()
l.Labels = s.config.Labels
l.Time = ts
l.Src = syslogLine.Client
l.Process = true
if !s.config.UseTimeMachine {
out <- types.Event{Line: l, Process: true, Type: types.LOG, ExpectMode: leaky.LIVE}
} else {
out <- types.Event{Line: l, Process: true, Type: types.LOG, ExpectMode: leaky.TIMEMACHINE}
}
}
}
}
crowdsec-1.4.6/pkg/acquisition/modules/syslog/syslog_test.go 0000664 0000000 0000000 00000007540 14371172257 0024361 0 ustar 00root root 0000000 0000000 package syslogacquisition
import (
"fmt"
"net"
"runtime"
"testing"
"time"
"github.com/crowdsecurity/crowdsec/pkg/cstest"
"github.com/crowdsecurity/crowdsec/pkg/types"
log "github.com/sirupsen/logrus"
"gopkg.in/tomb.v2"
"github.com/stretchr/testify/assert"
)
func TestConfigure(t *testing.T) {
tests := []struct {
config string
expectedErr string
}{
{
config: `
foobar: bla
source: syslog`,
expectedErr: "line 2: field foobar not found in type syslogacquisition.SyslogConfiguration",
},
{
config: `source: syslog`,
expectedErr: "",
},
{
config: `
source: syslog
listen_port: asd`,
expectedErr: "cannot unmarshal !!str `asd` into int",
},
{
config: `
source: syslog
listen_port: 424242`,
expectedErr: "invalid port 424242",
},
{
config: `
source: syslog
listen_addr: 10.0.0`,
expectedErr: "invalid listen IP 10.0.0",
},
}
subLogger := log.WithFields(log.Fields{
"type": "syslog",
})
for _, test := range tests {
s := SyslogSource{}
err := s.Configure([]byte(test.config), subLogger)
cstest.AssertErrorContains(t, err, test.expectedErr)
}
}
func writeToSyslog(logs []string) {
conn, err := net.Dial("udp", "127.0.0.1:4242")
if err != nil {
fmt.Printf("could not establish connection to syslog server : %s", err)
return
}
for _, log := range logs {
n, err := fmt.Fprint(conn, log)
if err != nil {
fmt.Printf("could not write to syslog server : %s", err)
return
}
if n != len(log) {
fmt.Printf("could not write to syslog server : %s", err)
return
}
}
}
func TestStreamingAcquisition(t *testing.T) {
tests := []struct {
name string
config string
expectedErr string
logs []string
expectedLines int
}{
{
name: "invalid msgs",
config: `source: syslog
listen_port: 4242
listen_addr: 127.0.0.1`,
logs: []string{"foobar", "bla", "pouet"},
},
{
name: "RFC5424",
config: `source: syslog
listen_port: 4242
listen_addr: 127.0.0.1`,
expectedLines: 2,
logs: []string{`<13>1 2021-05-18T11:58:40.828081+02:00 mantis sshd 49340 - [timeQuality isSynced="0" tzKnown="1"] blabla`,
`<13>1 2021-05-18T12:12:37.560695+02:00 mantis sshd 49340 - [timeQuality isSynced="0" tzKnown="1"] blabla2[foobar]`},
},
{
name: "RFC3164",
config: `source: syslog
listen_port: 4242
listen_addr: 127.0.0.1`,
expectedLines: 3,
logs: []string{`<13>May 18 12:37:56 mantis sshd[49340]: blabla2[foobar]`,
`<13>May 18 12:37:56 mantis sshd[49340]: blabla2`,
`<13>May 18 12:37:56 mantis sshd: blabla2`,
`<13>May 18 12:37:56 mantis sshd`},
},
}
if runtime.GOOS != "windows" {
tests = append(tests, struct {
name string
config string
expectedErr string
logs []string
expectedLines int
}{
name: "privileged port",
config: `source: syslog`,
expectedErr: "could not start syslog server: could not listen on port 514: listen udp 127.0.0.1:514: bind: permission denied",
})
}
for _, ts := range tests {
ts := ts
t.Run(ts.name, func(t *testing.T) {
subLogger := log.WithFields(log.Fields{
"type": "syslog",
})
s := SyslogSource{}
err := s.Configure([]byte(ts.config), subLogger)
if err != nil {
t.Fatalf("could not configure syslog source : %s", err)
}
tomb := tomb.Tomb{}
out := make(chan types.Event)
err = s.StreamingAcquisition(out, &tomb)
cstest.AssertErrorContains(t, err, ts.expectedErr)
if ts.expectedErr != "" {
return
}
if err != nil && ts.expectedErr == "" {
t.Fatalf("unexpected error while starting syslog server: %s", err)
return
}
actualLines := 0
go writeToSyslog(ts.logs)
READLOOP:
for {
select {
case <-out:
actualLines++
case <-time.After(2 * time.Second):
break READLOOP
}
}
assert.Equal(t, ts.expectedLines, actualLines)
tomb.Kill(nil)
tomb.Wait()
})
}
}
crowdsec-1.4.6/pkg/acquisition/modules/wineventlog/ 0000775 0000000 0000000 00000000000 14371172257 0022466 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/pkg/acquisition/modules/wineventlog/wineventlog.go 0000664 0000000 0000000 00000002543 14371172257 0025362 0 ustar 00root root 0000000 0000000 //go:build !windows
package wineventlogacquisition
import (
"errors"
"github.com/crowdsecurity/crowdsec/pkg/acquisition/configuration"
"github.com/crowdsecurity/crowdsec/pkg/types"
"github.com/prometheus/client_golang/prometheus"
log "github.com/sirupsen/logrus"
"gopkg.in/tomb.v2"
)
type WinEventLogSource struct{}
func (w *WinEventLogSource) Configure(yamlConfig []byte, logger *log.Entry) error {
return nil
}
func (w *WinEventLogSource) ConfigureByDSN(dsn string, labels map[string]string, logger *log.Entry) error {
return nil
}
func (w *WinEventLogSource) GetMode() string {
return ""
}
func (w *WinEventLogSource) SupportedModes() []string {
return []string{configuration.TAIL_MODE, configuration.CAT_MODE}
}
func (w *WinEventLogSource) OneShotAcquisition(out chan types.Event, t *tomb.Tomb) error {
return nil
}
func (w *WinEventLogSource) GetMetrics() []prometheus.Collector {
return nil
}
func (w *WinEventLogSource) GetAggregMetrics() []prometheus.Collector {
return nil
}
func (w *WinEventLogSource) GetName() string {
return "wineventlog"
}
func (w *WinEventLogSource) CanRun() error {
return errors.New("windows event log acquisition is only supported on Windows")
}
func (w *WinEventLogSource) StreamingAcquisition(out chan types.Event, t *tomb.Tomb) error {
return nil
}
func (w *WinEventLogSource) Dump() interface{} {
return w
}
crowdsec-1.4.6/pkg/acquisition/modules/wineventlog/wineventlog_test.go 0000664 0000000 0000000 00000012332 14371172257 0026416 0 ustar 00root root 0000000 0000000 //go:build windows
// +build windows
package wineventlogacquisition
import (
"runtime"
"testing"
"time"
"github.com/crowdsecurity/crowdsec/pkg/exprhelpers"
"github.com/crowdsecurity/crowdsec/pkg/types"
log "github.com/sirupsen/logrus"
"github.com/stretchr/testify/assert"
"golang.org/x/sys/windows/svc/eventlog"
"gopkg.in/tomb.v2"
)
func TestBadConfiguration(t *testing.T) {
if runtime.GOOS != "windows" {
t.Skip("Skipping test on non-windows OS")
}
tests := []struct {
config string
expectedErr string
}{
{
config: `source: wineventlog
foobar: 42`,
expectedErr: "field foobar not found in type wineventlogacquisition.WinEventLogConfiguration",
},
{
config: `source: wineventlog`,
expectedErr: "event_channel or xpath_query must be set",
},
{
config: `source: wineventlog
event_channel: Security
event_level: blabla`,
expectedErr: "buildXpathQuery failed: invalid log level",
},
{
config: `source: wineventlog
event_channel: Security
event_level: blabla`,
expectedErr: "buildXpathQuery failed: invalid log level",
},
{
config: `source: wineventlog
event_channel: foo
xpath_query: test`,
expectedErr: "event_channel and xpath_query are mutually exclusive",
},
}
subLogger := log.WithFields(log.Fields{
"type": "windowseventlog",
})
for _, test := range tests {
f := WinEventLogSource{}
err := f.Configure([]byte(test.config), subLogger)
assert.Contains(t, err.Error(), test.expectedErr)
}
}
func TestQueryBuilder(t *testing.T) {
if runtime.GOOS != "windows" {
t.Skip("Skipping test on non-windows OS")
}
tests := []struct {
config string
expectedQuery string
expectedErr string
}{
{
config: `source: wineventlog
event_channel: Security
event_level: Information`,
expectedQuery: "",
expectedErr: "",
},
{
config: `source: wineventlog
event_channel: Security
event_level: Error
event_ids:
- 42`,
expectedQuery: "",
expectedErr: "",
},
{
config: `source: wineventlog
event_channel: Security
event_level: Error
event_ids:
- 42
- 43`,
expectedQuery: "",
expectedErr: "",
},
{
config: `source: wineventlog
event_channel: Security`,
expectedQuery: "",
expectedErr: "",
},
{
config: `source: wineventlog
event_channel: Security
event_level: bla`,
expectedQuery: "",
expectedErr: "invalid log level",
},
}
subLogger := log.WithFields(log.Fields{
"type": "windowseventlog",
})
for _, test := range tests {
f := WinEventLogSource{}
f.Configure([]byte(test.config), subLogger)
q, err := f.buildXpathQuery()
if test.expectedErr != "" {
if err == nil {
t.Fatalf("expected error '%s' but got none", test.expectedErr)
}
assert.Contains(t, err.Error(), test.expectedErr)
} else {
assert.NoError(t, err)
assert.Equal(t, test.expectedQuery, q)
}
}
}
func TestLiveAcquisition(t *testing.T) {
if runtime.GOOS != "windows" {
t.Skip("Skipping test on non-windows OS")
}
tests := []struct {
config string
expectedLines []string
}{
{
config: `source: wineventlog
xpath_query: |
`,
expectedLines: []string{
"blabla",
"test",
"aaaa",
"bbbbb",
},
},
{
config: `source: wineventlog
xpath_query: |
asdfsdf`,
expectedLines: nil,
},
{
config: `source: wineventlog
event_channel: Application
event_level: Information
event_ids:
- 42`,
expectedLines: []string{
"testmessage",
},
},
{
config: `source: wineventlog
event_channel: Application
event_level: Information
event_ids:
- 43`,
expectedLines: nil,
},
}
subLogger := log.WithFields(log.Fields{
"type": "windowseventlog",
})
evthandler, err := eventlog.Open("Application")
if err != nil {
t.Fatalf("failed to open event log: %s", err)
}
for _, test := range tests {
to := &tomb.Tomb{}
c := make(chan types.Event)
f := WinEventLogSource{}
f.Configure([]byte(test.config), subLogger)
f.StreamingAcquisition(c, to)
time.Sleep(time.Second)
lines := test.expectedLines
go func() {
for _, line := range lines {
evthandler.Info(42, line)
}
}()
ticker := time.NewTicker(time.Second * 5)
linesRead := make([]string, 0)
READLOOP:
for {
select {
case <-ticker.C:
if test.expectedLines == nil {
break READLOOP
}
t.Fatalf("timeout")
case e := <-c:
linesRead = append(linesRead, exprhelpers.XMLGetNodeValue(e.Line.Raw, "/Event/EventData[1]/Data"))
if len(linesRead) == len(lines) {
break READLOOP
}
}
}
if test.expectedLines == nil {
assert.Equal(t, 0, len(linesRead))
} else {
assert.Equal(t, len(test.expectedLines), len(linesRead))
assert.Equal(t, test.expectedLines, linesRead)
}
to.Kill(nil)
to.Wait()
}
}
crowdsec-1.4.6/pkg/acquisition/modules/wineventlog/wineventlog_windows.go 0000664 0000000 0000000 00000021575 14371172257 0027142 0 ustar 00root root 0000000 0000000 package wineventlogacquisition
import (
"encoding/xml"
"errors"
"fmt"
"runtime"
"strings"
"syscall"
"time"
"github.com/crowdsecurity/crowdsec/pkg/acquisition/configuration"
leaky "github.com/crowdsecurity/crowdsec/pkg/leakybucket"
"github.com/crowdsecurity/crowdsec/pkg/types"
"github.com/google/winops/winlog"
"github.com/google/winops/winlog/wevtapi"
"github.com/prometheus/client_golang/prometheus"
log "github.com/sirupsen/logrus"
"golang.org/x/sys/windows"
"gopkg.in/tomb.v2"
"gopkg.in/yaml.v2"
)
type WinEventLogConfiguration struct {
configuration.DataSourceCommonCfg `yaml:",inline"`
EventChannel string `yaml:"event_channel"`
EventLevel string `yaml:"event_level"`
EventIDs []int `yaml:"event_ids"`
XPathQuery string `yaml:"xpath_query"`
EventFile string `yaml:"event_file"`
PrettyName string `yaml:"pretty_name"`
}
type WinEventLogSource struct {
config WinEventLogConfiguration
logger *log.Entry
evtConfig *winlog.SubscribeConfig
query string
name string
}
type QueryList struct {
Select Select `xml:"Query>Select"`
}
type Select struct {
Path string `xml:"Path,attr"`
Query string `xml:",chardata"`
}
var linesRead = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "cs_winevtlogsource_hits_total",
Help: "Total event that were read.",
},
[]string{"source"})
func logLevelToInt(logLevel string) ([]string, error) {
switch strings.ToUpper(logLevel) {
case "CRITICAL":
return []string{"1"}, nil
case "ERROR":
return []string{"2"}, nil
case "WARNING":
return []string{"3"}, nil
case "INFORMATION":
return []string{"0", "4"}, nil
case "VERBOSE":
return []string{"5"}, nil
default:
return nil, errors.New("invalid log level")
}
}
//This is lifted from winops/winlog, but we only want to render the basic XML string, we don't need the extra fluff
func (w *WinEventLogSource) getXMLEvents(config *winlog.SubscribeConfig, publisherCache map[string]windows.Handle, resultSet windows.Handle, maxEvents int) ([]string, error) {
var events = make([]windows.Handle, maxEvents)
var returned uint32
// Get handles to events from the result set.
err := wevtapi.EvtNext(
resultSet, // Handle to query or subscription result set.
uint32(len(events)), // The number of events to attempt to retrieve.
&events[0], // Pointer to the array of event handles.
2000, // Timeout in milliseconds to wait.
0, // Reserved. Must be zero.
&returned) // The number of handles in the array that are set by the API.
if err == windows.ERROR_NO_MORE_ITEMS {
return nil, err
} else if err != nil {
return nil, fmt.Errorf("wevtapi.EvtNext failed: %v", err)
}
// Event handles must be closed after they are returned by EvtNext whether or not we use them.
defer func() {
for _, event := range events[:returned] {
winlog.Close(event)
}
}()
// Render events.
var renderedEvents []string
for _, event := range events[:returned] {
// Render the basic XML representation of the event.
fragment, err := winlog.RenderFragment(event, wevtapi.EvtRenderEventXml)
if err != nil {
w.logger.Errorf("Failed to render event with RenderFragment, skipping: %v", err)
continue
}
w.logger.Tracef("Rendered event: %s", fragment)
renderedEvents = append(renderedEvents, fragment)
}
return renderedEvents, err
}
func (w *WinEventLogSource) buildXpathQuery() (string, error) {
var query string
queryComponents := [][]string{}
if w.config.EventIDs != nil {
eventIds := []string{}
for _, id := range w.config.EventIDs {
eventIds = append(eventIds, fmt.Sprintf("EventID=%d", id))
}
queryComponents = append(queryComponents, eventIds)
}
if w.config.EventLevel != "" {
levels, err := logLevelToInt(w.config.EventLevel)
logLevels := []string{}
if err != nil {
return "", err
}
for _, level := range levels {
logLevels = append(logLevels, fmt.Sprintf("Level=%s", level))
}
queryComponents = append(queryComponents, logLevels)
}
if len(queryComponents) > 0 {
andList := []string{}
for _, component := range queryComponents {
andList = append(andList, fmt.Sprintf("(%s)", strings.Join(component, " or ")))
}
query = fmt.Sprintf("*[System[%s]]", strings.Join(andList, " and "))
} else {
query = "*"
}
queryList := QueryList{Select: Select{Path: w.config.EventChannel, Query: query}}
xpathQuery, err := xml.Marshal(queryList)
if err != nil {
w.logger.Errorf("Marshal failed: %v", err)
return "", err
}
w.logger.Debugf("xpathQuery: %s", xpathQuery)
return string(xpathQuery), nil
}
func (w *WinEventLogSource) getEvents(out chan types.Event, t *tomb.Tomb) error {
subscription, err := winlog.Subscribe(w.evtConfig)
if err != nil {
w.logger.Errorf("Failed to subscribe to event log: %s", err)
return err
}
defer winlog.Close(subscription)
publisherCache := make(map[string]windows.Handle)
defer func() {
for _, h := range publisherCache {
winlog.Close(h)
}
}()
for {
select {
case <-t.Dying():
w.logger.Infof("wineventlog is dying")
return nil
default:
status, err := windows.WaitForSingleObject(w.evtConfig.SignalEvent, 1000)
if err != nil {
w.logger.Errorf("WaitForSingleObject failed: %s", err)
return err
}
if status == syscall.WAIT_OBJECT_0 {
renderedEvents, err := w.getXMLEvents(w.evtConfig, publisherCache, subscription, 500)
if err == windows.ERROR_NO_MORE_ITEMS {
windows.ResetEvent(w.evtConfig.SignalEvent)
} else if err != nil {
w.logger.Errorf("getXMLEvents failed: %v", err)
continue
}
for _, event := range renderedEvents {
linesRead.With(prometheus.Labels{"source": w.name}).Inc()
l := types.Line{}
l.Raw = event
l.Module = w.GetName()
l.Labels = w.config.Labels
l.Time = time.Now()
l.Src = w.name
l.Process = true
if !w.config.UseTimeMachine {
out <- types.Event{Line: l, Process: true, Type: types.LOG, ExpectMode: leaky.LIVE}
} else {
out <- types.Event{Line: l, Process: true, Type: types.LOG, ExpectMode: leaky.TIMEMACHINE}
}
}
}
}
}
}
func (w *WinEventLogSource) generateConfig(query string) (*winlog.SubscribeConfig, error) {
var config winlog.SubscribeConfig
var err error
// Create a subscription signaler.
config.SignalEvent, err = windows.CreateEvent(
nil, // Default security descriptor.
1, // Manual reset.
1, // Initial state is signaled.
nil) // Optional name.
if err != nil {
return &config, fmt.Errorf("windows.CreateEvent failed: %v", err)
}
config.Flags = wevtapi.EvtSubscribeToFutureEvents
config.Query, err = syscall.UTF16PtrFromString(query)
if err != nil {
return &config, fmt.Errorf("syscall.UTF16PtrFromString failed: %v", err)
}
return &config, nil
}
func (w *WinEventLogSource) Configure(yamlConfig []byte, logger *log.Entry) error {
config := WinEventLogConfiguration{}
w.logger = logger
err := yaml.UnmarshalStrict(yamlConfig, &config)
if err != nil {
return fmt.Errorf("unable to parse configuration: %v", err)
}
if config.EventChannel != "" && config.XPathQuery != "" {
return fmt.Errorf("event_channel and xpath_query are mutually exclusive")
}
if config.EventChannel == "" && config.XPathQuery == "" {
return fmt.Errorf("event_channel or xpath_query must be set")
}
config.Mode = configuration.TAIL_MODE
w.config = config
if config.XPathQuery != "" {
w.query = config.XPathQuery
} else {
w.query, err = w.buildXpathQuery()
if err != nil {
return fmt.Errorf("buildXpathQuery failed: %v", err)
}
}
w.evtConfig, err = w.generateConfig(w.query)
if err != nil {
return err
}
if config.PrettyName != "" {
w.name = config.PrettyName
} else {
w.name = w.query
}
return nil
}
func (w *WinEventLogSource) ConfigureByDSN(dsn string, labels map[string]string, logger *log.Entry) error {
return nil
}
func (w *WinEventLogSource) GetMode() string {
return w.config.Mode
}
func (w *WinEventLogSource) SupportedModes() []string {
return []string{configuration.TAIL_MODE}
}
func (w *WinEventLogSource) OneShotAcquisition(out chan types.Event, t *tomb.Tomb) error {
return nil
}
func (w *WinEventLogSource) GetMetrics() []prometheus.Collector {
return []prometheus.Collector{linesRead}
}
func (w *WinEventLogSource) GetAggregMetrics() []prometheus.Collector {
return []prometheus.Collector{linesRead}
}
func (w *WinEventLogSource) GetName() string {
return "wineventlog"
}
func (w *WinEventLogSource) CanRun() error {
if runtime.GOOS != "windows" {
return errors.New("windows event log acquisition is only supported on Windows")
}
return nil
}
func (w *WinEventLogSource) StreamingAcquisition(out chan types.Event, t *tomb.Tomb) error {
t.Go(func() error {
defer types.CatchPanic("crowdsec/acquis/wineventlog/streaming")
return w.getEvents(out, t)
})
return nil
}
func (w *WinEventLogSource) Dump() interface{} {
return w
}
crowdsec-1.4.6/pkg/acquisition/test_files/ 0000775 0000000 0000000 00000000000 14371172257 0020616 5 ustar 00root root 0000000 0000000 crowdsec-1.4.6/pkg/acquisition/test_files/backward_compat.yaml 0000664 0000000 0000000 00000000370 14371172257 0024623 0 ustar 00root root 0000000 0000000 filename: /tmp/test.log
labels:
type: syslog
---
filenames:
- /tmp/test*.log
labels:
type: syslog
---
# to be uncommented when we reimplement back journalctl
# journalctl_filter:
# - "_SYSTEMD_UNIT=ssh.service"
# labels:
# type: syslog
---
crowdsec-1.4.6/pkg/acquisition/test_files/bad_filetype.yaml 0000664 0000000 0000000 00000000125 14371172257 0024127 0 ustar 00root root 0000000 0000000 type: file
filenames: /tmp/tltlt.log #it should be an array
labels:
type: syslog
crowdsec-1.4.6/pkg/acquisition/test_files/bad_source.yaml 0000664 0000000 0000000 00000000073 14371172257 0023610 0 ustar 00root root 0000000 0000000 source: does_not_exist
labels:
type: syslog
foobar: toto
crowdsec-1.4.6/pkg/acquisition/test_files/badyaml.yaml 0000664 0000000 0000000 00000000006 14371172257 0023107 0 ustar 00root root 0000000 0000000