pax_global_header 0000666 0000000 0000000 00000000064 14452002130 0014501 g ustar 00root root 0000000 0000000 52 comment=cb3fabd3ef656df8ff416b5a7fe09a6ad9b0fb05
gluon-0.17.0/ 0000775 0000000 0000000 00000000000 14452002130 0012712 5 ustar 00root root 0000000 0000000 gluon-0.17.0/.github/ 0000775 0000000 0000000 00000000000 14452002130 0014252 5 ustar 00root root 0000000 0000000 gluon-0.17.0/.github/workflows/ 0000775 0000000 0000000 00000000000 14452002130 0016307 5 ustar 00root root 0000000 0000000 gluon-0.17.0/.github/workflows/pull-request.yml 0000664 0000000 0000000 00000002577 14452002130 0021507 0 ustar 00root root 0000000 0000000 name: Test
on:
pull_request:
branches: dev
jobs:
test:
strategy:
matrix:
os: [ubuntu-20.04, macos-12, windows-latest]
runs-on: ${{ matrix.os }}
steps:
- name: Get sources
uses: actions/checkout@v3
- name: Set up Go 1.20
uses: actions/setup-go@v3
with:
go-version: '1.20'
- name: Run go mod tidy
run: go mod tidy
- name: Run golangci-lint
run: |
go install github.com/golangci/golangci-lint/cmd/golangci-lint@251ceaa228607dd3e0371694a1ab2c45d21cb744
golangci-lint run --timeout=500s
- name: Run tests
run: go test -timeout 15m -v ./...
- name: Run tests with race check
if: runner.os != 'Windows'
run: go test -race -v ./tests
Fuzzing:
runs-on: ubuntu-latest
steps:
- name: Build Fuzzers
id: build
uses: google/oss-fuzz/infra/cifuzz/actions/build_fuzzers@master
with:
oss-fuzz-project-name: 'gluon'
language: go
- name: Run Fuzzers
uses: google/oss-fuzz/infra/cifuzz/actions/run_fuzzers@master
with:
oss-fuzz-project-name: 'gluon'
language: go
fuzz-seconds: 300
- name: Upload Crash
uses: actions/upload-artifact@v3
if: failure() && steps.build.outcome == 'success'
with:
name: artifacts
path: ./out/artifacts
gluon-0.17.0/.github/workflows/release.yml 0000664 0000000 0000000 00000000431 14452002130 0020450 0 ustar 00root root 0000000 0000000 name: Release workflow
on:
push:
branches: master
jobs:
release:
runs-on: ubuntu-latest
steps:
- name: Get sources
uses: actions/checkout@v3
- uses: cycjimmy/semantic-release-action@v3
env:
GITHUB_TOKEN: ${{ github.token }}
gluon-0.17.0/.gitignore 0000664 0000000 0000000 00000000016 14452002130 0014677 0 ustar 00root root 0000000 0000000 .vscode
.idea
gluon-0.17.0/.golangci.yml 0000664 0000000 0000000 00000001065 14452002130 0015300 0 ustar 00root root 0000000 0000000 run:
go: 1.18
skip-files:
- internal/parser/parser.go
- internal/parser/extern
linters:
presets:
- bugs
- comment
enable:
- wsl
disable:
- godox # Annoying, we have too many TODOs at the moment :p
- scopelint # Deprecated, replaced by exportloopref, which is enabled by default.
- errorlint # Too many false positives
issues:
exclude-rules:
- path: benchmarks
linters:
- gosec
- dupword
- path: tests
linters:
- dupword
- path: _test\.go
linters:
- dupword
gluon-0.17.0/.releaserc 0000664 0000000 0000000 00000000362 14452002130 0014661 0 ustar 00root root 0000000 0000000 {
"plugins": [
"@semantic-release/commit-analyzer",
"@semantic-release/release-notes-generator",
"@semantic-release/changelog",
["@semantic-release/git", {
"assets": ["internal/parser/lib"]
}],
"@semantic-release/github"
]
}
gluon-0.17.0/COPYING_NOTES.md 0000664 0000000 0000000 00000013074 14452002130 0015321 0 ustar 00root root 0000000 0000000 # Copying
The MIT License (MIT)
Copyright (c) 2020 James Houlahan
Copyright (c) 2021 Proton AG
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
# Dependencies
Gluon includes the following 3rd party software:
* [The Go Project libraries](https://golang.org/project/) | Available under [BSD license](https://golang.org/LICENSE)
* [ent](https://entgo.io/ent) available under [license](https://pkg.go.dev/entgo.io/ent?tab=licenses)
* [go-mbox](https://github.com/ProtonMail/go-mbox) available under [license](https://github.com/ProtonMail/go-mbox/blob/master/LICENSE)
* [juniper](https://github.com/bradenaw/juniper) available under [license](https://github.com/bradenaw/juniper/blob/master/LICENSE)
* [go-imap](https://github.com/emersion/go-imap) available under [license](https://github.com/emersion/go-imap/blob/master/LICENSE)
* [go-imap-uidplus](https://github.com/emersion/go-imap-uidplus) available under [license](https://github.com/emersion/go-imap-uidplus/blob/master/LICENSE)
* [mock](https://github.com/golang/mock) available under [license](https://github.com/golang/mock/blob/master/LICENSE)
* [uuid](https://github.com/google/uuid) available under [license](https://github.com/google/uuid/blob/master/LICENSE)
* [go-sqlite3](https://github.com/mattn/go-sqlite3) available under [license](https://github.com/mattn/go-sqlite3/blob/master/LICENSE)
* [lz4](https://github.com/pierrec/lz4/v4) available under [license](https://github.com/pierrec/lz4/v4/blob/master/LICENSE)
* [profile](https://github.com/pkg/profile) available under [license](https://github.com/pkg/profile/blob/master/LICENSE)
* [logrus](https://github.com/sirupsen/logrus) available under [license](https://github.com/sirupsen/logrus/blob/master/LICENSE)
* [testify](https://github.com/stretchr/testify) available under [license](https://github.com/stretchr/testify/blob/master/LICENSE)
* [goleak](https://go.uber.org/goleak) available under [license](https://pkg.go.dev/go.uber.org/goleak?tab=licenses)
* [exp](https://golang.org/x/exp) available under [license](https://cs.opensource.google/go/x/exp/+/master:LICENSE)
* [sys](https://golang.org/x/sys) available under [license](https://cs.opensource.google/go/x/sys/+/master:LICENSE)
* [text](https://golang.org/x/text) available under [license](https://cs.opensource.google/go/x/text/+/master:LICENSE)
* [yaml](https://gopkg.in/yaml.v3) available under [license](https://github.com/go-yaml/yaml/blob/v3.0.1/LICENSE)
* [atlas](https://ariga.io/atlas) available under [license](https://github.com/ariga/atlas/blob/master/LICENSE)
* [levenshtein](https://github.com/agext/levenshtein) available under [license](https://github.com/agext/levenshtein/blob/master/LICENSE)
* [go-textseg](https://github.com/apparentlymart/go-textseg/v13) available under [license](https://github.com/apparentlymart/go-textseg/v13/blob/master/LICENSE)
* [go-spew](https://github.com/davecgh/go-spew) available under [license](https://github.com/davecgh/go-spew/blob/master/LICENSE)
* [go-sasl](https://github.com/emersion/go-sasl) available under [license](https://github.com/emersion/go-sasl/blob/master/LICENSE)
* [fgprof](https://github.com/felixge/fgprof) available under [license](https://github.com/felixge/fgprof/blob/master/LICENSE)
* [inflect](https://github.com/go-openapi/inflect) available under [license](https://github.com/go-openapi/inflect/blob/master/LICENSE)
* [go-cmp](https://github.com/google/go-cmp) available under [license](https://github.com/google/go-cmp/blob/master/LICENSE)
* [pprof](https://github.com/google/pprof) available under [license](https://github.com/google/pprof/blob/master/LICENSE)
* [hcl](https://github.com/hashicorp/hcl/v2) available under [license](https://github.com/hashicorp/hcl/v2/blob/master/LICENSE)
* [pretty](https://github.com/kr/pretty) available under [license](https://github.com/kr/pretty/blob/master/LICENSE)
* [go-wordwrap](https://github.com/mitchellh/go-wordwrap) available under [license](https://github.com/mitchellh/go-wordwrap/blob/master/LICENSE)
* [go-difflib](https://github.com/pmezard/go-difflib) available under [license](https://github.com/pmezard/go-difflib/blob/master/LICENSE)
* [go-cty](https://github.com/zclconf/go-cty) available under [license](https://github.com/zclconf/go-cty/blob/master/LICENSE)
* [mod](https://golang.org/x/mod) available under [license](https://cs.opensource.google/go/x/mod/+/master:LICENSE)
* [sync](https://golang.org/x/sync) available under [license](https://cs.opensource.google/go/x/sync/+/master:LICENSE)
* [check](https://gopkg.in/check.v1) available under [license](https://github.com/go-check/check/blob/v1/LICENSE)
gluon-0.17.0/LICENSE 0000664 0000000 0000000 00000002126 14452002130 0013720 0 ustar 00root root 0000000 0000000 The MIT License (MIT)
Copyright (c) 2020 James Houlahan
Copyright (c) 2021 Proton AG
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
gluon-0.17.0/README.md 0000664 0000000 0000000 00000006004 14452002130 0014171 0 ustar 00root root 0000000 0000000
Gluon
An IMAP4rev1 library focusing on correctness, stability and performance.
# Demo
The demo server can be started with:
```
$ GLUON_LOG_LEVEL=trace go run demo/demo.go
DEBU[0000] Applying update update="MailboxCreated: Mailbox.ID = 0, Mailbox.Name = INBOX" user-id=ac8970c5-cdb7-4043-ad85-ad9b9defcfb8
DEBU[0000] Applying update update="MessagesCreated: MessageCount=0 Messages=[]" user-id=ac8970c5-cdb7-4043-ad85-ad9b9defcfb8
INFO[0000] User added to server userID=ac8970c5-cdb7-4043-ad85-ad9b9defcfb8
DEBU[0000] Applying update update="MailboxCreated: Mailbox.ID = 0, Mailbox.Name = INBOX" user-id=a51fad46-9bde-462a-a467-6c30f9a40a63
DEBU[0000] Applying update update="MessagesCreated: MessageCount=0 Messages=[]" user-id=a51fad46-9bde-462a-a467-6c30f9a40a63
INFO[0000] User added to server userID=a51fad46-9bde-462a-a467-6c30f9a40a63
INFO[0000] Server is listening on 127.0.0.1:1143
```
By default, the demo server includes two demo users, both with password `pass`.
The first has addresses `user1@example.com` and `alias1@example.com`.
The second has addresses `user2@example.com` and `alias2@example.com`.
Once started, connect to the demo server with an email client (e.g. thunderbird) or via telnet:
```
$ telnet 127.0.0.1 1143
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
* OK [CAPABILITY IDLE IMAP4rev1 MOVE UIDPLUS UNSELECT] 00.00.00 - gluon session ID 2
tag login user1@example.com pass
tag OK [CAPABILITY IDLE IMAP4rev1 MOVE UIDPLUS UNSELECT] Logged in
tag append inbox (\Seen) {14}
+ Ready
To: user@pm.me
tag OK [APPENDUID 1 1] APPEND
tag select inbox
* FLAGS (\Answered \Deleted \Flagged \Seen)
* 1 EXISTS
* 1 RECENT
* OK [PERMANENTFLAGS (\Answered \Deleted \Flagged \Seen)] Flags permitted
* OK [UIDNEXT 2] Predicted next UID
* OK [UIDVALIDITY 1] UIDs valid
tag OK [READ-WRITE] SELECT
tag fetch 1:* (UID BODY.PEEK[])
* 1 FETCH (UID 1 BODY[] {32}
X-Pm-Gluon-Id: 1
To: user@pm.me)
tag OK command completed in 1.030958ms
```
# Changing DB schema
Do not forget to re-generate ent code after changing the DB schema in `./internal/db/ent/schema`.
```
pushd ./internal/db/ent && go generate . && popd
gluon-0.17.0/async/ 0000775 0000000 0000000 00000000000 14452002130 0014027 5 ustar 00root root 0000000 0000000 gluon-0.17.0/async/bool.go 0000664 0000000 0000000 00000000767 14452002130 0015323 0 ustar 00root root 0000000 0000000 package async
import "sync/atomic"
// atomicBool is an atomic boolean value.
// The zero value is false.
type atomicBool struct {
v uint32
}
// Load atomically loads and returns the value stored in x.
func (x *atomicBool) load() bool { return atomic.LoadUint32(&x.v) != 0 }
// Store atomically stores val into x.
func (x *atomicBool) store(val bool) { atomic.StoreUint32(&x.v, b32(val)) }
// b32 returns a uint32 0 or 1 representing b.
func b32(b bool) uint32 {
if b {
return 1
}
return 0
}
gluon-0.17.0/async/context.go 0000664 0000000 0000000 00000002370 14452002130 0016044 0 ustar 00root root 0000000 0000000 package async
import (
"context"
"sync"
)
// Abortable collects groups of functions that can be aborted by calling Abort.
type Abortable struct {
abortFunc []context.CancelFunc
abortLock sync.RWMutex
}
func (a *Abortable) Do(ctx context.Context, fn func(context.Context)) {
fn(a.newCancelCtx(ctx))
}
func (a *Abortable) Abort() {
a.abortLock.RLock()
defer a.abortLock.RUnlock()
for _, fn := range a.abortFunc {
fn()
}
}
func (a *Abortable) newCancelCtx(ctx context.Context) context.Context {
a.abortLock.Lock()
defer a.abortLock.Unlock()
ctx, cancel := context.WithCancel(ctx)
a.abortFunc = append(a.abortFunc, cancel)
return ctx
}
// RangeContext iterates over the given channel until the context is canceled or the
// channel is closed.
func RangeContext[T any](ctx context.Context, ch <-chan T, fn func(T)) {
for {
select {
case v, ok := <-ch:
if !ok {
return
}
fn(v)
case <-ctx.Done():
return
}
}
}
// ForwardContext forwards all values from the src channel to the dst channel until the
// context is canceled or the src channel is closed.
func ForwardContext[T any](ctx context.Context, dst chan<- T, src <-chan T) {
RangeContext(ctx, src, func(v T) {
select {
case dst <- v:
case <-ctx.Done():
}
})
}
gluon-0.17.0/async/group.go 0000664 0000000 0000000 00000011050 14452002130 0015507 0 ustar 00root root 0000000 0000000 package async
import (
"context"
"math/rand"
"sync"
"time"
)
// Group is forked and improved version of "github.com/bradenaw/juniper/xsync.Group".
//
// It manages a group of goroutines. The main change to original is posibility
// to wait passed function to finish without canceling it's context and adding
// PanicHandler.
type Group struct {
baseCtx context.Context
ctx context.Context
jobCtx context.Context
cancel context.CancelFunc
finish context.CancelFunc
wg sync.WaitGroup
panicHandler PanicHandler
}
// NewGroup returns a Group ready for use. The context passed to any of the f functions will be a
// descendant of ctx.
func NewGroup(ctx context.Context, panicHandler PanicHandler) *Group {
bgCtx, cancel := context.WithCancel(ctx)
jobCtx, finish := context.WithCancel(ctx)
return &Group{
baseCtx: ctx,
ctx: bgCtx,
jobCtx: jobCtx,
cancel: cancel,
finish: finish,
panicHandler: panicHandler,
}
}
// Once calls f once from another goroutine.
func (g *Group) Once(f func(ctx context.Context)) {
g.wg.Add(1)
go func() {
defer HandlePanic(g.panicHandler)
defer g.wg.Done()
f(g.ctx)
}()
}
// jitterDuration returns a random duration in [d - jitter, d + jitter].
func jitterDuration(d time.Duration, jitter time.Duration) time.Duration {
return d + time.Duration(float64(jitter)*((rand.Float64()*2)-1)) //nolint:gosec
}
// Periodic spawns a goroutine that calls f once per interval +/- jitter.
func (g *Group) Periodic(
interval time.Duration,
jitter time.Duration,
f func(ctx context.Context),
) {
g.wg.Add(1)
go func() {
defer HandlePanic(g.panicHandler)
defer g.wg.Done()
t := time.NewTimer(jitterDuration(interval, jitter))
defer t.Stop()
for {
if g.ctx.Err() != nil {
return
}
select {
case <-g.jobCtx.Done():
return
case <-t.C:
}
t.Reset(jitterDuration(interval, jitter))
f(g.ctx)
}
}()
}
// Trigger spawns a goroutine which calls f whenever the returned function is called. If f is
// already running when triggered, f will run again immediately when it finishes.
func (g *Group) Trigger(f func(ctx context.Context)) func() {
c := make(chan struct{}, 1)
g.wg.Add(1)
go func() {
defer HandlePanic(g.panicHandler)
defer g.wg.Done()
for {
if g.ctx.Err() != nil {
return
}
select {
case <-g.jobCtx.Done():
return
case <-c:
}
f(g.ctx)
}
}()
return func() {
select {
case c <- struct{}{}:
default:
}
}
}
// PeriodicOrTrigger spawns a goroutine which calls f whenever the returned function is called. If
// f is already running when triggered, f will run again immediately when it finishes. Also calls f
// when it has been interval+/-jitter since the last trigger.
func (g *Group) PeriodicOrTrigger(
interval time.Duration,
jitter time.Duration,
f func(ctx context.Context),
) func() {
c := make(chan struct{}, 1)
g.wg.Add(1)
go func() {
defer HandlePanic(g.panicHandler)
defer g.wg.Done()
t := time.NewTimer(jitterDuration(interval, jitter))
defer t.Stop()
for {
if g.ctx.Err() != nil {
return
}
select {
case <-g.jobCtx.Done():
return
case <-t.C:
t.Reset(jitterDuration(interval, jitter))
case <-c:
if !t.Stop() {
<-t.C
}
t.Reset(jitterDuration(interval, jitter))
}
f(g.ctx)
}
}()
return func() {
select {
case c <- struct{}{}:
default:
}
}
}
func (g *Group) resetCtx() {
g.jobCtx, g.finish = context.WithCancel(g.baseCtx)
g.ctx, g.cancel = context.WithCancel(g.baseCtx)
}
// Cancel is send to all of the spawn goroutines and ends periodic
// or trigger routines.
func (g *Group) Cancel() {
g.cancel()
g.finish()
g.resetCtx()
}
// Finish will ends all periodic or polls routines. It will let
// currently running functions to finish (cancel is not sent).
//
// It is not safe to call Wait concurrently with any other method on g.
func (g *Group) Finish() {
g.finish()
g.jobCtx, g.finish = context.WithCancel(g.baseCtx)
}
// CancelAndWait cancels the context passed to any of the spawned goroutines and waits for all spawned
// goroutines to exit.
//
// It is not safe to call Wait concurrently with any other method on g.
func (g *Group) CancelAndWait() {
g.finish()
g.cancel()
g.wg.Wait()
g.resetCtx()
}
// WaitToFinish will ends all periodic or polls routines. It will wait for
// currently running functions to finish (cancel is not sent).
//
// It is not safe to call Wait concurrently with any other method on g.
func (g *Group) WaitToFinish() {
g.finish()
g.wg.Wait()
g.jobCtx, g.finish = context.WithCancel(g.baseCtx)
}
gluon-0.17.0/async/logging.go 0000664 0000000 0000000 00000000451 14452002130 0016004 0 ustar 00root root 0000000 0000000 package async
import (
"context"
"github.com/ProtonMail/gluon/logging"
)
func GoAnnotated(ctx context.Context, panicHandler PanicHandler, fn func(context.Context), labelMap ...logging.Labels) {
go func() {
defer HandlePanic(panicHandler)
logging.DoAnnotated(ctx, fn, labelMap...)
}()
}
gluon-0.17.0/async/panic_handler.go 0000664 0000000 0000000 00000000640 14452002130 0017145 0 ustar 00root root 0000000 0000000 package async
type PanicHandler interface {
HandlePanic(interface{})
}
type NoopPanicHandler struct{}
func (n NoopPanicHandler) HandlePanic(r interface{}) {}
func HandlePanic(panicHandler PanicHandler) {
if panicHandler == nil {
return
}
if _, ok := panicHandler.(NoopPanicHandler); ok {
return
}
if _, ok := panicHandler.(*NoopPanicHandler); ok {
return
}
panicHandler.HandlePanic(recover())
}
gluon-0.17.0/async/panic_handler_test.go 0000664 0000000 0000000 00000001534 14452002130 0020207 0 ustar 00root root 0000000 0000000 package async
import (
"fmt"
"testing"
"github.com/stretchr/testify/require"
)
type recoverHandler struct{}
func (h recoverHandler) HandlePanic(r interface{}) {
fmt.Println("recoverHandler", r)
}
func TestPanicHandler(t *testing.T) {
require.NotPanics(t, func() {
defer HandlePanic(recoverHandler{})
panic("there")
})
require.PanicsWithValue(t, "where", func() {
defer HandlePanic(NoopPanicHandler{})
panic("where")
})
require.PanicsWithValue(t, "everywhere", func() {
defer HandlePanic(nil)
panic("everywhere")
})
require.NotPanics(t, func() {
defer HandlePanic(recoverHandler{})
panic(nil)
})
require.NotPanics(t, func() {
defer HandlePanic(recoverHandler{})
})
require.NotPanics(t, func() {
defer HandlePanic(NoopPanicHandler{})
})
require.NotPanics(t, func() {
defer HandlePanic(&NoopPanicHandler{})
})
}
gluon-0.17.0/async/queued_channel.go 0000664 0000000 0000000 00000004317 14452002130 0017343 0 ustar 00root root 0000000 0000000 package async
import (
"context"
"sync"
)
// QueuedChannel represents a channel on which queued items can be published without having to worry if the reader
// has actually consumed existing items first or if there's no way of knowing ahead of time what the ideal channel
// buffer size should be.
type QueuedChannel[T any] struct {
ch chan T
stopCh chan struct{}
items []T
cond *sync.Cond
closed atomicBool // Should use atomic.Bool once we use Go 1.19!
}
func NewQueuedChannel[T any](chanBufferSize, queueCapacity int, panicHandler PanicHandler) *QueuedChannel[T] {
queue := &QueuedChannel[T]{
ch: make(chan T, chanBufferSize),
stopCh: make(chan struct{}),
items: make([]T, 0, queueCapacity),
cond: sync.NewCond(&sync.Mutex{}),
}
// The queue is initially not closed.
queue.closed.store(false)
// Start the queue consumer.
GoAnnotated(context.Background(), panicHandler, func(ctx context.Context) {
defer close(queue.ch)
for {
item, ok := queue.pop()
if !ok {
return
}
select {
case queue.ch <- item:
continue
case <-queue.stopCh:
return
}
}
})
return queue
}
func (q *QueuedChannel[T]) Enqueue(items ...T) bool {
if q.closed.load() {
return false
}
q.cond.L.Lock()
defer q.cond.L.Unlock()
q.items = append(q.items, items...)
q.cond.Broadcast()
return true
}
func (q *QueuedChannel[T]) GetChannel() <-chan T {
return q.ch
}
func (q *QueuedChannel[T]) Close() {
q.closed.store(true)
q.cond.L.Lock()
defer q.cond.L.Unlock()
q.cond.Broadcast()
}
// CloseAndDiscardQueued force closes the channel and does not guarantee that the remaining queued items will be read.
func (q *QueuedChannel[T]) CloseAndDiscardQueued() {
close(q.stopCh)
q.Close()
}
func (q *QueuedChannel[T]) pop() (T, bool) {
q.cond.L.Lock()
defer q.cond.L.Unlock()
var item T
// Wait until there are items to pop, returning false immediately if the queue is closed.
// This allows the queue to continue popping elements if it's closed,
// but will prevent it from hanging indefinitely once it runs out of items.
for len(q.items) == 0 {
if q.closed.load() {
return item, false
}
q.cond.Wait()
}
item, q.items = q.items[0], q.items[1:]
return item, true
}
gluon-0.17.0/async/queued_channel_test.go 0000664 0000000 0000000 00000002365 14452002130 0020403 0 ustar 00root root 0000000 0000000 package async
import (
"testing"
"github.com/stretchr/testify/require"
"go.uber.org/goleak"
)
func TestQueuedChannel(t *testing.T) {
defer goleak.VerifyNone(t)
// Create a new queued channel.
queue := NewQueuedChannel[int](3, 3, NoopPanicHandler{})
// Push some items to the queue.
require.True(t, queue.Enqueue(1, 2, 3))
// Get a go channel to read from the queue.
resCh := queue.GetChannel()
// Check we can initially read items off the channel.
require.Equal(t, 1, <-resCh)
require.Equal(t, 2, <-resCh)
require.Equal(t, 3, <-resCh)
// Push some more items to the queue.
require.True(t, queue.Enqueue(4, 5, 6))
// Close the queue before reading the items.
queue.Close()
// Check we can still read the three items.
require.Equal(t, 4, <-resCh)
require.Equal(t, 5, <-resCh)
require.Equal(t, 6, <-resCh)
// Enqueuing more items after the queue is closed should return false.
require.False(t, queue.Enqueue(7, 8, 9))
}
func TestQueuedChannelDoesNotLeakIfThereAreNoReadersOnCloseAndDiscard(t *testing.T) {
defer goleak.VerifyNone(t)
// Create a new queued channel.
queue := NewQueuedChannel[int](1, 3, NoopPanicHandler{})
// Push some items to the queue.
require.True(t, queue.Enqueue(1, 2, 3))
queue.CloseAndDiscardQueued()
}
gluon-0.17.0/async/wait_group.go 0000664 0000000 0000000 00000000623 14452002130 0016537 0 ustar 00root root 0000000 0000000 package async
import "sync"
type WaitGroup struct {
wg sync.WaitGroup
panicHandler PanicHandler
}
func MakeWaitGroup(panicHandler PanicHandler) WaitGroup {
return WaitGroup{panicHandler: panicHandler}
}
func (wg *WaitGroup) Go(f func()) {
wg.wg.Add(1)
go func() {
defer HandlePanic(wg.panicHandler)
defer wg.wg.Done()
f()
}()
}
func (wg *WaitGroup) Wait() {
wg.wg.Wait()
}
gluon-0.17.0/benchmarks/ 0000775 0000000 0000000 00000000000 14452002130 0015027 5 ustar 00root root 0000000 0000000 gluon-0.17.0/benchmarks/README.md 0000664 0000000 0000000 00000002670 14452002130 0016313 0 ustar 00root root 0000000 0000000 # Gluon Benchmarks
Collection of benchmarks and/or tools to test and/or profile Gluon performance. See each folder's `README.md` for more information.
## Capturing Profiles
You can use the gluon demo binary as a way to profile Gluon. Be sure to build the binary with `go build`.
You need to run the benchmark for each type of profile you wish to generate as it is not possible to profile
multiple items at the same time.
Example:
```bash
./demo -profile-cpu -profile-path=$PWD # Generates $PWD/cpu.pprof - CPU Profiling.
./demo -profile-mem -profile-path=$PWD # Generates $PWD/mem.pprof - Memory Profiling.
./demo -profile-lock -profile-path=$PWD # Generates $PWD/block.pprof - Locking/blocking profiling.
```
When the benchmark has finished execution kill the demo binary and the profile data will be written out.
## Analysing the data
You need to install the pprof tool (`go install github.com/google/pprof`) to analyse the generated data.
After the tool is installed you can analyse the data with `pprof `. See [this blog post](1) and
[this tutorial](2) for an introduction to pprof.
For a more comprehensible overview you can also launch pprof's web interface which includes many additions
such as a flamegraph.
```bash
pprof -http 127.0.0.1:8080
```
## Available Benchmarks/Tools
* [imaptest](imaptest)
## References
[1]: https://www.youtube.com/watch?v=N3PWzBeLX2M
[2]: https://go.dev/blog/pprof gluon-0.17.0/benchmarks/gluon_bench/ 0000775 0000000 0000000 00000000000 14452002130 0017312 5 ustar 00root root 0000000 0000000 gluon-0.17.0/benchmarks/gluon_bench/README.md 0000664 0000000 0000000 00000002256 14452002130 0020576 0 ustar 00root root 0000000 0000000 # Gluon Bench - IMAP benchmarks
Gluon bench provides a collection of benchmarks that operate either at the IMAP client level or directly on Gluon
itself (e.g: sync).
All IMAP command related benchmarks can be run against a local gluon server which will be started with the benchmark or
an externally running IMAP server.
If running against a local server, it's possible to record the execution times of every individual command.
Finally, it is also possible to produce a JSON report rather than printing to the console.
## Building
```bash
# In benchmarks/gluon_bench
go build main.go -o gluon_bench
```
## Running Gluon Bench
To run Gluon Bench specify a set of options followed by a set of benchmarks you wish to run:
```bash
gluon_bench -verbose -parallel-client=4 fetch append
```
Please consult the output of `gluon_bench -h` for all available options/modifiers and benchmarks.
## Integrating Gluon Bench in other projects
When integrating Gluon Bench in other projects which may contain other gluon connectors:
* Register your connector with `utils.RegisterConnector()`
* Specify the connector with the option `-connector=<...>`
* In your `main` call `benchmark.RunMain()` gluon-0.17.0/benchmarks/gluon_bench/benchmark/ 0000775 0000000 0000000 00000000000 14452002130 0021244 5 ustar 00root root 0000000 0000000 gluon-0.17.0/benchmarks/gluon_bench/benchmark/bench_dir_config.go 0000664 0000000 0000000 00000001551 14452002130 0025037 0 ustar 00root root 0000000 0000000 package benchmark
import (
"os"
)
// BenchDirConfig controls the directory where the benchmark data will be generated.
type BenchDirConfig interface {
Get() (string, error)
}
// FixedBenchDirConfig always returns a known path.
type FixedBenchDirConfig struct {
path string
}
func NewFixedBenchDirConfig(path string) *FixedBenchDirConfig {
return &FixedBenchDirConfig{path: path}
}
func (p *FixedBenchDirConfig) Get() (string, error) {
if err := os.MkdirAll(p.path, 0o777); err != nil {
return "", err
}
return p.path, nil
}
// TmpBenchDirConfig returns a temporary path that is generated on first use.
type TmpBenchDirConfig struct {
path string
}
func (t *TmpBenchDirConfig) Get() (string, error) {
if len(t.path) == 0 {
path, err := os.MkdirTemp("", "gluon-bench-*")
if err != nil {
return "", err
}
t.path = path
}
return t.path, nil
}
gluon-0.17.0/benchmarks/gluon_bench/benchmark/benchmark.go 0000664 0000000 0000000 00000001555 14452002130 0023533 0 ustar 00root root 0000000 0000000 package benchmark
import (
"context"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/reporter"
)
type Benchmark interface {
// Name should return the name of the benchmark. It will also be used to match against cli args.
Name() string
// Setup sets up the benchmark state, this is not timed.
Setup(ctx context.Context, benchmarkDir string) error
// Run performs the actual benchmark, this is timed.
Run(ctx context.Context) (*reporter.BenchmarkRun, error)
// TearDown clear the benchmark state, this is not timed.
TearDown(ctx context.Context) error
}
var benchmarks = make(map[string]Benchmark)
func RegisterBenchmark(benchmark Benchmark) {
if _, ok := benchmarks[benchmark.Name()]; ok {
panic("Benchmark with this name already exists")
}
benchmarks[benchmark.Name()] = benchmark
}
func GetBenchmarks() map[string]Benchmark {
return benchmarks
}
gluon-0.17.0/benchmarks/gluon_bench/benchmark/run.go 0000664 0000000 0000000 00000007104 14452002130 0022401 0 ustar 00root root 0000000 0000000 package benchmark
import (
"context"
"flag"
"fmt"
"os"
"path/filepath"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/flags"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/reporter"
"golang.org/x/exp/slices"
)
func RunMain() {
flag.Usage = func() {
fmt.Printf("Usage %v [options] benchmark0 benchmark1 ... benchmarkN\n", os.Args[0])
fmt.Printf("\nAvailable Benchmarks:\n")
var benchmarks []string
for k := range GetBenchmarks() {
benchmarks = append(benchmarks, k)
}
slices.Sort(benchmarks)
for _, k := range benchmarks {
fmt.Printf(" * %v\n", k)
}
fmt.Printf("\nOptions:\n")
flag.PrintDefaults()
}
flag.Parse()
var benchmarks []Benchmark
args := flag.Args()
if len(args) == 0 {
flag.Usage()
return
}
for _, arg := range args {
if v, ok := GetBenchmarks()[arg]; ok {
benchmarks = append(benchmarks, v)
}
}
if len(benchmarks) == 0 {
panic("No benchmarks selected")
}
var benchmarkReporter reporter.BenchmarkReporter
if len(*flags.JsonReporter) != 0 {
benchmarkReporter = reporter.NewJSONReporter(*flags.JsonReporter)
} else {
benchmarkReporter = &reporter.StdOutReporter{}
}
var benchDirConfig BenchDirConfig
if len(*flags.BenchPath) != 0 {
benchDirConfig = NewFixedBenchDirConfig(*flags.BenchPath)
} else {
benchDirConfig = &TmpBenchDirConfig{}
}
benchmarkReports := make([]*reporter.BenchmarkReport, 0, len(benchmarks))
for _, v := range benchmarks {
if *flags.Verbose {
fmt.Printf("Begin Benchmark: %v\n", v.Name())
}
numRuns := *flags.BenchmarkRuns
benchmarkStats := make([]*reporter.BenchmarkStatistics, 0, numRuns)
for r := uint(0); r < numRuns; r++ {
if *flags.Verbose {
fmt.Printf("Benchmark Run: %v\n", r)
}
benchStat := measureBenchmark(benchDirConfig, r, v)
benchmarkStats = append(benchmarkStats, benchStat)
}
benchmarkReports = append(benchmarkReports, reporter.NewBenchmarkReport(v.Name(), benchmarkStats...))
if *flags.Verbose {
fmt.Printf("End Benchmark: %v\n", v.Name())
}
}
if benchmarkReporter != nil {
if *flags.Verbose {
fmt.Printf("Generating Report\n")
}
if err := benchmarkReporter.ProduceReport(benchmarkReports); err != nil {
panic(fmt.Sprintf("Failed to produce benchmark report: %v", err))
}
}
if *flags.Verbose {
fmt.Printf("Finished\n")
}
}
func measureBenchmark(dirConfig BenchDirConfig, iteration uint, bench Benchmark) *reporter.BenchmarkStatistics {
benchPath, err := dirConfig.Get()
if err != nil {
panic(fmt.Sprintf("Failed to get server directory: %v", err))
}
benchPath = filepath.Join(benchPath, fmt.Sprintf("%v-%d", bench.Name(), iteration))
if *flags.Verbose {
fmt.Printf("Benchmark Data Path: %v\n", benchPath)
}
if err := os.MkdirAll(benchPath, 0o777); err != nil {
panic(fmt.Sprintf("Failed to create server directory '%v' : %v", benchPath, err))
}
ctx := context.Background()
if err := bench.Setup(ctx, benchPath); err != nil {
panic(fmt.Sprintf("Failed to setup benchmark %v: %v", bench.Name(), err))
}
benchRun, benchErr := bench.Run(ctx)
if benchErr != nil {
panic(fmt.Sprintf("Failed to run benchmark %v: %v", bench.Name(), err))
}
if err := bench.TearDown(ctx); err != nil {
panic(fmt.Sprintf("Failed to teardown benchmark %v: %v", bench.Name(), err))
}
if !*flags.SkipClean {
if *flags.Verbose {
fmt.Printf("Cleaning benchmark dir: %v\n", benchPath)
}
if err := os.RemoveAll(benchPath); err != nil {
fmt.Fprintf(os.Stderr, "Failed to remote benchmark data dir '%v': %v\n", benchPath, err)
}
}
return reporter.NewBenchmarkStatistics(benchRun.Extra, benchRun.Durations...)
}
gluon-0.17.0/benchmarks/gluon_bench/flags/ 0000775 0000000 0000000 00000000000 14452002130 0020406 5 ustar 00root root 0000000 0000000 gluon-0.17.0/benchmarks/gluon_bench/flags/general.go 0000664 0000000 0000000 00000001544 14452002130 0022356 0 ustar 00root root 0000000 0000000 package flags
import "flag"
var (
BenchPath = flag.String("path", "", "Filepath where to write the database data. If not set a temp folder will be used.")
Verbose = flag.Bool("verbose", false, "Enable verbose logging.")
JsonReporter = flag.String("json-reporter", "", "If specified, will generate a json report with the given filename.")
BenchmarkRuns = flag.Uint("bench-runs", 1, "Number of runs per benchmark.")
Connector = flag.String("connector", "dummy", "Key of the connector implementation registered with ConnectorFactory.")
UserName = flag.String("user-name", "user", "Username for the connector user, defaults to 'user'.")
UserPassword = flag.String("user-pwd", "password", "Password for the connector user, defaults to 'password'.")
SkipClean = flag.Bool("skip-clean", false, "Do not cleanup benchmark data directory.")
)
gluon-0.17.0/benchmarks/gluon_bench/flags/imap_benchmarks.go 0000664 0000000 0000000 00000001547 14452002130 0024067 0 ustar 00root root 0000000 0000000 package flags
import "flag"
var (
IMAPRemoteServer = flag.String("imap-remote-server", "", "IP address and port of the remote IMAP server to run against. E.g. 127.0.0.1:1143.")
IMAPMessageCount = flag.Uint("imap-msg-count", 1000, "Number of messages to add to the mailbox before each benchmark")
IMAPRandomSeqSetIntervals = flag.Bool("imap-random-seqset-intervals", false, "When set, generate random sequence intervals rather than single numbers.")
IMAPUIDMode = flag.Bool("imap-uid-mode", false, "When set, will run benchmarks in UID mode if available.")
IMAPParallelClients = flag.Uint("imap-parallel-clients", 1, "Set the number of clients to be run in parallel during the benchmark.")
IMAPMailboxMessageDir = flag.String("imap-mbox-file-dir", "", "Folder path to load *.eml messages instead of builtin selection.")
)
gluon-0.17.0/benchmarks/gluon_bench/flags/store_benchmarks.go 0000664 0000000 0000000 00000001005 14452002130 0024262 0 ustar 00root root 0000000 0000000 package flags
import "flag"
var (
Store = flag.String("store", "default", "Name of the storage implementation to benchmark. Defaults to regular on disk storage by default.")
StoreWorkers = flag.Uint("store-workers", 1, "Number of concurrent workers for store operations.")
StoreItemCount = flag.Uint("store-item-count", 1000, "Number of items to generate in the store benchmarks.")
StoreItemSize = flag.Uint("store-item-size", 15*1024*1024, "Number of items to generate in the store benchmarks.")
)
gluon-0.17.0/benchmarks/gluon_bench/gluon_benchmarks/ 0000775 0000000 0000000 00000000000 14452002130 0022633 5 ustar 00root root 0000000 0000000 gluon-0.17.0/benchmarks/gluon_bench/gluon_benchmarks/sync.go 0000664 0000000 0000000 00000006406 14452002130 0024144 0 ustar 00root root 0000000 0000000 package gluon_benchmarks
import (
"context"
"flag"
"math/rand"
"time"
"github.com/ProtonMail/gluon"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/benchmark"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/flags"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/reporter"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/timing"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/utils"
"github.com/ProtonMail/gluon/imap"
"github.com/ProtonMail/gluon/internal/hash"
"github.com/google/uuid"
_ "github.com/mattn/go-sqlite3"
"github.com/sirupsen/logrus"
)
var (
syncMessageCountFlag = flag.Uint("sync-msg-count", 1000, "Number of messages to sync.")
syncMBoxCountFlag = flag.Uint("sync-mbox-count", 1, "Number of mailboxes to sync.")
)
type Sync struct {
connector utils.ConnectorImpl
server *gluon.Server
mailboxes []imap.MailboxID
}
func NewSync() benchmark.Benchmark {
return &Sync{}
}
func (s *Sync) Name() string {
return "gluon-sync"
}
func (s *Sync) Setup(ctx context.Context, benchmarkDir string) error {
loggerIn := logrus.StandardLogger().WriterLevel(logrus.TraceLevel)
loggerOut := logrus.StandardLogger().WriterLevel(logrus.TraceLevel)
opts := []gluon.Option{
gluon.WithDataDir(benchmarkDir),
gluon.WithDatabaseDir(benchmarkDir),
gluon.WithLogger(loggerIn, loggerOut),
}
server, err := gluon.New(opts...)
if err != nil {
return err
}
s.server = server
connector, err := s.setupConnector(ctx)
if err != nil {
return err
}
s.connector = connector
return nil
}
func (s *Sync) setupConnector(ctx context.Context) (utils.ConnectorImpl, error) {
c, err := utils.NewConnector(*flags.Connector)
if err != nil {
return nil, err
}
mboxIDs := make([]imap.MailboxID, 0, *syncMBoxCountFlag)
for i := uint(0); i < *syncMBoxCountFlag; i++ {
mbox, err := c.Connector().CreateMailbox(ctx, []string{uuid.NewString()})
if err != nil {
return nil, err
}
mboxIDs = append(mboxIDs, mbox.ID)
}
messages := [][]byte{
[]byte(utils.MessageEmbedded),
[]byte(utils.MessageMultiPartMixed),
[]byte(utils.MessageAfterNoonMeeting),
}
flagSet := imap.NewFlagSet("\\Recent", "\\Draft", "\\Foo")
s.mailboxes = make([]imap.MailboxID, 0, len(mboxIDs))
for _, mboxID := range mboxIDs {
for i := uint(0); i < *syncMessageCountFlag; i++ {
randIndex := rand.Intn(len(messages))
if _, _, err := c.Connector().CreateMessage(ctx, mboxID, messages[randIndex], flagSet, time.Now()); err != nil {
return nil, err
}
}
s.mailboxes = append(s.mailboxes, mboxID)
}
if _, err = s.server.AddUser(
ctx,
c.Connector(),
hash.SHA256([]byte(*flags.UserPassword)),
); err != nil {
return nil, err
}
return c, nil
}
func (s *Sync) Run(ctx context.Context) (*reporter.BenchmarkRun, error) {
timer := timing.Timer{}
timer.Start()
err := s.connector.Sync(ctx)
timer.Stop()
if err != nil {
return nil, err
}
return reporter.NewBenchmarkRunSingle(timer.Elapsed(), nil), nil
}
func (s *Sync) TearDown(ctx context.Context) error {
for _, id := range s.mailboxes {
if err := s.connector.Connector().DeleteMailbox(ctx, id); err != nil {
return err
}
}
if s.server != nil {
if err := s.server.Close(ctx); err != nil {
return err
}
}
return nil
}
func init() {
benchmark.RegisterBenchmark(NewSync())
}
gluon-0.17.0/benchmarks/gluon_bench/imap_benchmarks/ 0000775 0000000 0000000 00000000000 14452002130 0022435 5 ustar 00root root 0000000 0000000 gluon-0.17.0/benchmarks/gluon_bench/imap_benchmarks/append.go 0000664 0000000 0000000 00000002311 14452002130 0024230 0 ustar 00root root 0000000 0000000 package imap_benchmarks
import (
"context"
"fmt"
"net"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/benchmark"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/flags"
"github.com/emersion/go-imap/client"
)
type Append struct {
*stateTracker
}
func NewAppend() benchmark.Benchmark {
return NewIMAPBenchmarkRunner(&Append{stateTracker: newStateTracker()})
}
func (a *Append) Name() string {
return "imap-append"
}
func (a *Append) Setup(ctx context.Context, addr net.Addr) error {
if *flags.IMAPMessageCount == 0 {
return fmt.Errorf("invalid message count")
}
return WithClient(addr, func(cl *client.Client) error {
for i := uint(0); i < *flags.IMAPParallelClients; i++ {
if _, err := a.createRandomMBox(cl); err != nil {
return err
}
}
return nil
})
}
func (a *Append) TearDown(ctx context.Context, addr net.Addr) error {
//return a.cleanupWithAddr(addr)
return nil
}
func (a *Append) Run(ctx context.Context, addr net.Addr) error {
RunParallelClients(addr, func(c *client.Client, u uint) {
if err := BuildMailbox(c, a.MBoxes[u], int(*flags.IMAPMessageCount)); err != nil {
panic(err)
}
})
return nil
}
func init() {
benchmark.RegisterBenchmark(NewAppend())
}
gluon-0.17.0/benchmarks/gluon_bench/imap_benchmarks/build_mailbox.go 0000664 0000000 0000000 00000002501 14452002130 0025574 0 ustar 00root root 0000000 0000000 package imap_benchmarks
import (
"fmt"
"time"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/flags"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/utils"
"github.com/emersion/go-imap/client"
"github.com/google/uuid"
)
// BuildMailbox creates a mailbox of name `mailbox` and fills it up with `messageCount` random messages.
func BuildMailbox(cl *client.Client, mailbox string, messageCount int) error {
var messages []string
if len(*flags.IMAPMailboxMessageDir) != 0 {
msgs, err := utils.LoadEMLFilesFromDirectory(*flags.IMAPMailboxMessageDir)
if err != nil {
return err
}
if *flags.Verbose {
fmt.Printf("Loaded %v messages from '%v'\n", len(msgs), *flags.IMAPMailboxMessageDir)
}
messages = msgs
} else {
messages = []string{utils.MessageAfterNoonMeeting, utils.MessageMultiPartMixed, utils.MessageEmbedded}
}
return BuildMailboxWithMessages(cl, mailbox, messageCount, messages)
}
func BuildMailboxWithMessages(cl *client.Client, mailbox string, messageCount int, messages []string) error {
messagesLen := len(messages)
for i := 0; i < messageCount; i++ {
literal := fmt.Sprintf("To: %v@a.com\r\nFrom: %v@a.com\r\n", uuid.NewString(), uuid.NewString()) + messages[i%messagesLen]
if err := AppendToMailbox(cl, mailbox, literal, time.Now()); err != nil {
return err
}
}
return nil
}
gluon-0.17.0/benchmarks/gluon_bench/imap_benchmarks/client.go 0000664 0000000 0000000 00000011534 14452002130 0024246 0 ustar 00root root 0000000 0000000 package imap_benchmarks
import (
"bufio"
"fmt"
"math/rand"
"net"
"os"
"strings"
"sync"
"time"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/flags"
"github.com/bradenaw/juniper/xslices"
"github.com/emersion/go-imap"
"github.com/emersion/go-imap/client"
)
func AppendToMailbox(cl *client.Client, mailboxName string, literal string, time time.Time, flags ...string) error {
return cl.Append(mailboxName, flags, time, strings.NewReader(literal))
}
func FetchMessage(cl *client.Client, sequenceSet *imap.SeqSet, items ...imap.FetchItem) error {
ch := make(chan *imap.Message)
go func() {
for {
if _, ok := <-ch; !ok {
break
}
}
}()
return cl.Fetch(sequenceSet, items, ch)
}
func flagsToInterface(flags ...string) []interface{} {
return xslices.Map(flags, func(f string) interface{} {
return interface{}(f)
})
}
func Store(cl *client.Client, sequenceSet *imap.SeqSet, item string, silent bool, flags ...string) error {
if !silent {
ch := make(chan *imap.Message)
go func() {
for {
if _, ok := <-ch; !ok {
break
}
}
}()
return cl.Store(sequenceSet, imap.StoreItem(item), flagsToInterface(flags...), ch)
} else {
return cl.Store(sequenceSet, imap.StoreItem(item), flagsToInterface(flags...), nil)
}
}
func UIDStore(cl *client.Client, sequenceSet *imap.SeqSet, item string, silent bool, flags ...string) error {
if !silent {
ch := make(chan *imap.Message)
go func() {
for {
if _, ok := <-ch; !ok {
break
}
}
}()
return cl.UidStore(sequenceSet, imap.StoreItem(item), flagsToInterface(flags...), ch)
} else {
return cl.UidStore(sequenceSet, imap.StoreItem(item), flagsToInterface(flags...), nil)
}
}
func UIDFetchMessage(cl *client.Client, sequenceSet *imap.SeqSet, items ...imap.FetchItem) error {
ch := make(chan *imap.Message)
go func() {
for {
if _, ok := <-ch; !ok {
break
}
}
}()
return cl.UidFetch(sequenceSet, items, ch)
}
func SequenceListFromFile(path string) ([]*imap.SeqSet, error) {
result := make([]*imap.SeqSet, 0, 64)
readFile, err := os.Open(path)
if err != nil {
return nil, err
}
defer readFile.Close()
fileScanner := bufio.NewScanner(readFile)
fileScanner.Split(bufio.ScanLines)
for fileScanner.Scan() {
seqSet, err := imap.ParseSeqSet(fileScanner.Text())
if err != nil {
return nil, err
}
result = append(result, seqSet)
}
return result, nil
}
func NewSequenceSetAll() *imap.SeqSet {
seq := &imap.SeqSet{}
seq.AddRange(1, 0)
return seq
}
func RandomSequenceSetNum(max uint32) *imap.SeqSet {
var num uint32
for num == 0 {
num = rand.Uint32() % max
}
r := &imap.SeqSet{}
r.AddNum(num)
return r
}
func RandomSequenceSetRange(max uint32) *imap.SeqSet {
var start uint32
for start == 0 {
start = rand.Uint32() % max
}
stop := start
if start != max-1 {
for stop <= start {
stop = rand.Uint32() % max
}
}
r := &imap.SeqSet{}
r.AddRange(start, stop)
return r
}
type MailboxInfo struct {
Name string
ReadOnly bool
}
func RunParallelClientsWithMailbox(addr net.Addr, mbox string, readOnly bool, fn func(*client.Client, uint)) {
mailboxes := make([]MailboxInfo, *flags.IMAPParallelClients)
for i := uint(0); i < *flags.IMAPParallelClients; i++ {
mailboxes[i] = MailboxInfo{Name: mbox, ReadOnly: readOnly}
}
RunParallelClientsWithMailboxes(addr, mailboxes, fn)
}
func RunParallelClientsWithMailboxes(addr net.Addr, mailboxes []MailboxInfo, fn func(*client.Client, uint)) {
if len(mailboxes) != int(*flags.IMAPParallelClients) {
panic("Mailbox count doesn't match worker count")
}
RunParallelClients(addr, func(cl *client.Client, index uint) {
mbox := mailboxes[index]
if _, err := cl.Select(mbox.Name, mbox.ReadOnly); err != nil {
panic(err)
}
fn(cl, index)
})
}
func RunParallelClients(addr net.Addr, fn func(*client.Client, uint)) {
wg := sync.WaitGroup{}
for i := uint(0); i < *flags.IMAPParallelClients; i++ {
wg.Add(1)
go func(index uint) {
defer wg.Done()
if err := WithClient(addr, func(c *client.Client) error {
fn(c, index)
return nil
}); err != nil {
panic(err)
}
}(i)
}
wg.Wait()
}
func FillMailbox(cl *client.Client, mbox string) error {
if *flags.IMAPMessageCount == 0 {
return fmt.Errorf("message count can't be 0")
}
return BuildMailbox(cl, mbox, int(*flags.IMAPMessageCount))
}
func WithClient(addr net.Addr, fn func(*client.Client) error) error {
cl, err := newClient(addr.String())
if err != nil {
return err
}
defer closeClient(cl)
return fn(cl)
}
func newClient(addr string) (*client.Client, error) {
client, err := client.Dial(addr)
if err != nil {
return nil, err
}
if err := client.Login(*flags.UserName, *flags.UserPassword); err != nil {
return nil, err
}
return client, nil
}
func closeClient(cl *client.Client) {
if err := cl.Logout(); err != nil {
fmt.Fprintf(os.Stderr, "Failed to close client: %v\n", err)
}
}
gluon-0.17.0/benchmarks/gluon_bench/imap_benchmarks/copy.go 0000664 0000000 0000000 00000004412 14452002130 0023737 0 ustar 00root root 0000000 0000000 package imap_benchmarks
import (
"context"
"flag"
"net"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/benchmark"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/flags"
"github.com/emersion/go-imap"
"github.com/emersion/go-imap/client"
)
var (
copyCountFlag = flag.Uint("imap-copy-count", 0, "Total number of messages to copy during copy benchmarks.")
copyListFlag = flag.String("imap-copy-list", "", "Use a list of predefined sequences to copy rather than random generated.")
copyAllFlag = flag.Bool("imap-copy-all", false, "If set, perform a copy of the all messages.")
)
type Copy struct {
*stateTracker
seqSets *ParallelSeqSet
}
func NewCopy() benchmark.Benchmark {
return NewIMAPBenchmarkRunner(&Copy{
stateTracker: newStateTracker(),
})
}
func (*Copy) Name() string {
return "imap-copy"
}
func (c *Copy) Setup(ctx context.Context, addr net.Addr) error {
return WithClient(addr, func(cl *client.Client) error {
if _, err := c.createAndFillRandomMBox(cl); err != nil {
return err
}
if _, err := c.createRandomMBox(cl); err != nil {
return err
}
copyCount := uint32(*copyCountFlag)
if copyCount == 0 {
copyCount = uint32(*flags.IMAPMessageCount)
}
seqSets, err := NewParallelSeqSet(copyCount,
*flags.IMAPParallelClients,
*copyListFlag,
*copyAllFlag,
*flags.IMAPRandomSeqSetIntervals,
false,
*flags.IMAPUIDMode)
if err != nil {
return err
}
c.seqSets = seqSets
return nil
})
}
func (c *Copy) TearDown(ctx context.Context, addr net.Addr) error {
return c.cleanupWithAddr(addr)
}
func (c *Copy) Run(ctx context.Context, addr net.Addr) error {
srcMBox := c.MBoxes[0]
dstMBox := c.MBoxes[1]
RunParallelClientsWithMailbox(addr, srcMBox, true, func(cl *client.Client, index uint) {
var copyFn func(*client.Client, *imap.SeqSet, string) error
if *flags.IMAPUIDMode {
copyFn = func(cl *client.Client, set *imap.SeqSet, mailbox string) error {
return cl.UidCopy(set, mailbox)
}
} else {
copyFn = func(cl *client.Client, set *imap.SeqSet, mailbox string) error {
return cl.Copy(set, mailbox)
}
}
for _, v := range c.seqSets.Get(index) {
if err := copyFn(cl, v, dstMBox); err != nil {
panic(err)
}
}
})
return nil
}
func init() {
benchmark.RegisterBenchmark(NewCopy())
}
gluon-0.17.0/benchmarks/gluon_bench/imap_benchmarks/expunge.go 0000664 0000000 0000000 00000006610 14452002130 0024442 0 ustar 00root root 0000000 0000000 package imap_benchmarks
import (
"context"
"flag"
"fmt"
"net"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/benchmark"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/flags"
"github.com/bradenaw/juniper/xslices"
"github.com/emersion/go-imap"
"github.com/emersion/go-imap/client"
)
var (
expungeCountFlag = flag.Uint("imap-expunge-count", 0, "Total number of messages to expunge during expunge benchmarks.")
expungeSameMBoxFlag = flag.Bool("imap-expunge-same-mbox", false, "When true run all the expunge test on the same inbox rather than separate ones in parallel.")
expungeListFlag = flag.String("imap-expunge-list", "", "Use a list of predefined sequences to expunge rather than random generated. Only works when -expunge-same-mbox is not set.")
expungeAllFlag = flag.Bool("imap-expunge-all", false, "If set, perform a expunge of the all messages. Only works when -expunge-same-mbox is not set.")
)
type Expunge struct {
*stateTracker
seqSets *ParallelSeqSet
mboxInfo []MailboxInfo
}
func NewExpunge() benchmark.Benchmark {
return NewIMAPBenchmarkRunner(&Expunge{stateTracker: newStateTracker()})
}
func (*Expunge) Name() string {
return "imap-expunge"
}
func (e *Expunge) Setup(ctx context.Context, addr net.Addr) error {
return WithClient(addr, func(cl *client.Client) error {
if *expungeSameMBoxFlag {
if _, err := e.createAndFillRandomMBox(cl); err != nil {
return err
}
expungeCount := uint32(*expungeCountFlag)
if expungeCount == 0 {
expungeCount = uint32(*flags.IMAPMessageCount)
}
e.seqSets = NewParallelSeqSetExpunge(expungeCount,
*flags.IMAPParallelClients,
*flags.IMAPRandomSeqSetIntervals,
*flags.IMAPUIDMode,
)
e.mboxInfo = make([]MailboxInfo, *flags.IMAPParallelClients)
for i := 0; i < len(e.mboxInfo); i++ {
e.mboxInfo[i] = MailboxInfo{Name: e.MBoxes[0], ReadOnly: false}
}
} else {
for i := uint(0); i < *flags.IMAPParallelClients; i++ {
if _, err := e.createAndFillRandomMBox(cl); err != nil {
return err
}
}
seqSets, err := NewParallelSeqSet(uint32(*flags.IMAPMessageCount),
*flags.IMAPParallelClients,
*expungeListFlag,
*expungeAllFlag,
*flags.IMAPRandomSeqSetIntervals,
true,
*flags.IMAPUIDMode)
if err != nil {
return err
}
e.seqSets = seqSets
e.mboxInfo = xslices.Map(e.MBoxes, func(m string) MailboxInfo {
return MailboxInfo{Name: m, ReadOnly: false}
})
}
return nil
})
}
func (e *Expunge) TearDown(ctx context.Context, addr net.Addr) error {
return e.cleanupWithAddr(addr)
}
func (e *Expunge) Run(ctx context.Context, addr net.Addr) error {
RunParallelClientsWithMailboxes(addr, e.mboxInfo, func(cl *client.Client, index uint) {
var expungeFn func(*client.Client, *imap.SeqSet) error
if *flags.IMAPUIDMode {
expungeFn = func(cl *client.Client, set *imap.SeqSet) error {
if err := UIDStore(cl, set, "+FLAGS", true, imap.DeletedFlag); err != nil {
return err
}
return cl.Expunge(nil)
}
} else {
expungeFn = func(cl *client.Client, set *imap.SeqSet) error {
if err := Store(cl, set, "+FLAGS", true, imap.DeletedFlag); err != nil {
return err
}
return cl.Expunge(nil)
}
}
for _, v := range e.seqSets.Get(index) {
if err := expungeFn(cl, v); err != nil {
panic(fmt.Sprintf("Seq:%v err:%v", v, err))
}
}
})
return nil
}
func init() {
benchmark.RegisterBenchmark(NewExpunge())
}
gluon-0.17.0/benchmarks/gluon_bench/imap_benchmarks/fetch.go 0000664 0000000 0000000 00000004754 14452002130 0024067 0 ustar 00root root 0000000 0000000 package imap_benchmarks
import (
"context"
"flag"
"net"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/benchmark"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/flags"
"github.com/emersion/go-imap"
"github.com/emersion/go-imap/client"
)
var (
fetchCountFlag = flag.Uint("imap-fetch-count", 0, "Total number of messages to fetch during fetch benchmarks.")
fetchListFlag = flag.String("imap-fetch-list", "", "Use a list of predefined sequences to fetch rather than random generated.")
fetchReadOnly = flag.Bool("imap-fetch-read-only", false, "If set, perform fetches in read-only mode.")
fetchAllFlag = flag.Bool("imap-fetch-all", false, "If set, perform one fetch for all messages.")
)
type Fetch struct {
*stateTracker
seqSets *ParallelSeqSet
}
func NewFetch() benchmark.Benchmark {
return NewIMAPBenchmarkRunner(&Fetch{stateTracker: newStateTracker()})
}
func (*Fetch) Name() string {
return "imap-fetch"
}
func (f *Fetch) Setup(ctx context.Context, addr net.Addr) error {
return WithClient(addr, func(cl *client.Client) error {
if _, err := f.createAndFillRandomMBox(cl); err != nil {
return err
}
fetchCount := uint32(*fetchCountFlag)
if fetchCount == 0 {
fetchCount = uint32(*flags.IMAPMessageCount)
}
seqSets, err := NewParallelSeqSet(fetchCount,
*flags.IMAPParallelClients,
*fetchListFlag,
*fetchAllFlag,
*flags.IMAPRandomSeqSetIntervals,
false,
*flags.IMAPUIDMode)
if err != nil {
return err
}
f.seqSets = seqSets
return nil
})
}
func (f *Fetch) TearDown(ctx context.Context, addr net.Addr) error {
return f.cleanupWithAddr(addr)
}
func (f *Fetch) Run(ctx context.Context, addr net.Addr) error {
attributes := []imap.FetchItem{
imap.FetchFlags,
imap.FetchRFC822Size,
imap.FetchRFC822Header,
imap.FetchInternalDate,
imap.FetchRFC822,
imap.FetchRFC822Text,
imap.FetchBody,
"BODY[]",
"BODY[HEADER]",
}
RunParallelClientsWithMailbox(addr, f.MBoxes[0], *fetchReadOnly, func(cl *client.Client, index uint) {
var fetchFn func(*client.Client, *imap.SeqSet) error
if *flags.IMAPUIDMode {
fetchFn = func(cl *client.Client, set *imap.SeqSet) error {
return UIDFetchMessage(cl, set, attributes...)
}
} else {
fetchFn = func(cl *client.Client, set *imap.SeqSet) error {
return FetchMessage(cl, set, attributes...)
}
}
for _, v := range f.seqSets.Get(index) {
if err := fetchFn(cl, v); err != nil {
panic(err)
}
}
})
return nil
}
func init() {
benchmark.RegisterBenchmark(NewFetch())
}
gluon-0.17.0/benchmarks/gluon_bench/imap_benchmarks/imap_benchmark.go 0000664 0000000 0000000 00000003004 14452002130 0025721 0 ustar 00root root 0000000 0000000 package imap_benchmarks
import (
"context"
"fmt"
"net"
"strings"
"time"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/reporter"
"github.com/ProtonMail/gluon/profiling"
)
// IMAPBenchmark is intended to be used to build benchmarks which bench IMAP commands on a given server.
type IMAPBenchmark interface {
// Name should return the name of the benchmark. It will also be used to match against cli args.
Name() string
// Setup sets up the benchmark state, this is not timed.
Setup(ctx context.Context, addr net.Addr) error
// Run performs the actual benchmark, this is timed.
Run(ctx context.Context, addr net.Addr) error
// TearDown clear the benchmark state, this is not timed.
TearDown(ctx context.Context, addr net.Addr) error
}
type IMAPBenchmarkExtra struct {
CMDStatistic [profiling.CmdTypeTotal]*reporter.BenchmarkStatistics
}
func (i *IMAPBenchmarkExtra) String() string {
builder := strings.Builder{}
for n, v := range i.CMDStatistic {
if v.SampleCount == 0 {
continue
}
builder.WriteString(fmt.Sprintf("[%v] %v\n", profiling.CmdTypeToString(n), v.String()))
}
return builder.String()
}
func NewIMAPBenchmarkRun(duration time.Duration, cmdTimings [profiling.CmdTypeTotal][]time.Duration) *reporter.BenchmarkRun {
var cmdStatistic [profiling.CmdTypeTotal]*reporter.BenchmarkStatistics
for i, v := range cmdTimings {
cmdStatistic[i] = reporter.NewBenchmarkStatistics(nil, v...)
}
return reporter.NewBenchmarkRunSingle(duration, &IMAPBenchmarkExtra{CMDStatistic: cmdStatistic})
}
gluon-0.17.0/benchmarks/gluon_bench/imap_benchmarks/imap_benchmark_runner.go 0000664 0000000 0000000 00000004411 14452002130 0027315 0 ustar 00root root 0000000 0000000 package imap_benchmarks
import (
"context"
"fmt"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/benchmark"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/flags"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/imap_benchmarks/server"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/reporter"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/timing"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/utils"
)
type IMAPBenchmarkRunner struct {
benchmark IMAPBenchmark
cmdProfilerBuilder *utils.DurationCmdProfilerBuilder
server server.Server
}
func (i *IMAPBenchmarkRunner) Name() string {
return i.benchmark.Name()
}
// Setup sets up the benchmark state, this is not timed.
func (i *IMAPBenchmarkRunner) Setup(ctx context.Context, benchmarkDir string) error {
var serverBuilder server.ServerBuilder
if len(*flags.IMAPRemoteServer) != 0 {
builder, err := server.NewRemoteServerBuilder(*flags.IMAPRemoteServer)
if err != nil {
panic(fmt.Sprintf("Invalid Server address: %v", err))
}
serverBuilder = builder
} else {
serverBuilder = &server.LocalServerBuilder{}
}
i.cmdProfilerBuilder.Clear()
server, err := serverBuilder.New(ctx, benchmarkDir, i.cmdProfilerBuilder)
if err != nil {
return err
}
i.server = server
if err := i.benchmark.Setup(ctx, i.server.Address()); err != nil {
return err
}
return nil
}
// Run performs the actual benchmark, this is timed.
func (i *IMAPBenchmarkRunner) Run(ctx context.Context) (*reporter.BenchmarkRun, error) {
scopedTimer := timing.Timer{}
scopedTimer.Start()
err := i.benchmark.Run(ctx, i.server.Address())
scopedTimer.Stop()
if err != nil {
return nil, err
}
return NewIMAPBenchmarkRun(scopedTimer.Elapsed(), i.cmdProfilerBuilder.Merge()), nil
}
// TearDown clear the benchmark state, this is not timed.
func (i *IMAPBenchmarkRunner) TearDown(ctx context.Context) error {
if i.server != nil {
if err := i.benchmark.TearDown(ctx, i.server.Address()); err != nil {
return err
}
if err := i.server.Close(ctx); err != nil {
return err
}
}
return nil
}
func NewIMAPBenchmarkRunner(bench IMAPBenchmark) benchmark.Benchmark {
return &IMAPBenchmarkRunner{
benchmark: bench,
cmdProfilerBuilder: utils.NewDurationCmdProfilerBuilder(),
}
}
gluon-0.17.0/benchmarks/gluon_bench/imap_benchmarks/move.go 0000664 0000000 0000000 00000006267 14452002130 0023745 0 ustar 00root root 0000000 0000000 package imap_benchmarks
import (
"context"
"flag"
"fmt"
"net"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/benchmark"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/flags"
"github.com/bradenaw/juniper/xslices"
"github.com/emersion/go-imap"
"github.com/emersion/go-imap/client"
)
var (
moveListFlag = flag.String("imap-move-list", "", "Use a list of predefined sequences to move rather than random generated.")
moveAllFlag = flag.Bool("imap-move-all", false, "If set, perform a move of the all messages.")
moveIntoSameDstFlag = flag.Bool("imap-move-into-same-dst", false, "If set, rather than moving each unique mailbox into separate unique mailboxes, move all messages into one common destination mailbox.")
)
type Move struct {
*stateTracker
seqSets *ParallelSeqSet
srcMailboxes []string
dstMailboxes []string
}
func NewMove() benchmark.Benchmark {
return NewIMAPBenchmarkRunner(&Move{stateTracker: newStateTracker()})
}
func (*Move) Name() string {
return "imap-move"
}
func (m *Move) Setup(ctx context.Context, addr net.Addr) error {
if *flags.IMAPMessageCount == 0 {
return fmt.Errorf("move benchmark requires a message count > 0")
}
return WithClient(addr, func(cl *client.Client) error {
m.srcMailboxes = make([]string, 0, *flags.IMAPParallelClients)
m.dstMailboxes = make([]string, 0, *flags.IMAPParallelClients)
for i := uint(0); i < *flags.IMAPParallelClients; i++ {
mbox, err := m.createAndFillRandomMBox(cl)
if err != nil {
return err
}
m.srcMailboxes = append(m.srcMailboxes, mbox)
}
var dstMboxCount uint
if *moveIntoSameDstFlag {
dstMboxCount = 1
} else {
dstMboxCount = *flags.IMAPParallelClients
}
for i := uint(0); i < dstMboxCount; i++ {
mbox, err := m.createRandomMBox(cl)
if err != nil {
return err
}
m.dstMailboxes = append(m.dstMailboxes, mbox)
}
seqSets, err := NewParallelSeqSet(uint32(*flags.IMAPMessageCount),
*flags.IMAPParallelClients,
*moveListFlag,
*moveAllFlag,
*flags.IMAPRandomSeqSetIntervals,
true,
*flags.IMAPUIDMode)
if err != nil {
return err
}
m.seqSets = seqSets
return nil
})
}
func (m *Move) TearDown(ctx context.Context, addr net.Addr) error {
return m.cleanupWithAddr(addr)
}
func (m *Move) Run(ctx context.Context, addr net.Addr) error {
mboxInfos := xslices.Map(m.srcMailboxes, func(name string) MailboxInfo {
return MailboxInfo{
Name: name,
ReadOnly: true,
}
})
RunParallelClientsWithMailboxes(addr, mboxInfos, func(cl *client.Client, index uint) {
var moveFn func(*client.Client, *imap.SeqSet, string) error
if *flags.IMAPUIDMode {
moveFn = func(cl *client.Client, set *imap.SeqSet, mailbox string) error {
return cl.UidMove(set, mailbox)
}
} else {
moveFn = func(cl *client.Client, set *imap.SeqSet, mailbox string) error {
return cl.Move(set, mailbox)
}
}
for _, v := range m.seqSets.Get(index) {
if *moveIntoSameDstFlag {
if err := moveFn(cl, v, m.dstMailboxes[0]); err != nil {
panic(err)
}
} else {
if err := moveFn(cl, v, m.dstMailboxes[index]); err != nil {
panic(err)
}
}
}
})
return nil
}
func init() {
benchmark.RegisterBenchmark(NewMove())
}
gluon-0.17.0/benchmarks/gluon_bench/imap_benchmarks/parallel_seqset.go 0000664 0000000 0000000 00000014353 14452002130 0026152 0 ustar 00root root 0000000 0000000 package imap_benchmarks
import (
"fmt"
"math/rand"
"github.com/emersion/go-imap"
)
// ParallelSeqSet contains a list of sequence sets which can be used by the benchmarks. Use one of the several new
// functions to initialize the state. Internally it holds one list per expected worker.
type ParallelSeqSet struct {
seqSets [][]*imap.SeqSet
}
func (p *ParallelSeqSet) Get(i uint) []*imap.SeqSet {
return p.seqSets[i]
}
// NewParallelSeqSetFromFile load the sequence sets from a file. The same sequence set will be assigned to
// all workers.
func NewParallelSeqSetFromFile(path string, numWorkers uint) (*ParallelSeqSet, error) {
list, err := SequenceListFromFile(path)
if err != nil {
return nil, err
}
seqSets := make([][]*imap.SeqSet, numWorkers)
for i := uint(0); i < numWorkers; i++ {
seqSets[i] = list
}
return &ParallelSeqSet{
seqSets: seqSets,
}, nil
}
// NewParallelSeqSetExpunge generates sequence ids or intervals that can be used in cases where the data is expunged
// and/or moved from the original inbox. It also makes sure that concurrent workers can't overlap to avoid operations
// on messages that no longer exist.
func NewParallelSeqSetExpunge(count uint32, numWorkers uint, generateIntervals, uid bool) *ParallelSeqSet {
lists := make([][]*imap.SeqSet, numWorkers)
workerSplit := count / uint32(numWorkers)
available := make([]uint32, count)
for r := uint32(0); r < count; r++ {
available[r] = r + 1
}
for i := uint(0); i < numWorkers; i++ {
available := available[(uint32(i) * workerSplit):(uint32(i+1) * workerSplit)]
list := make([]*imap.SeqSet, 0, workerSplit)
if generateIntervals {
const maxIntervalRange = uint32(40)
for len(available) > 0 {
intervalRange := rand.Uint32() % maxIntervalRange
itemsLeft := uint32(len(available))
index := rand.Uint32() % itemsLeft
if index > intervalRange {
index -= intervalRange
} else {
index = 0
}
if index+intervalRange >= itemsLeft {
intervalRange = itemsLeft - index
}
seqSet := &imap.SeqSet{}
if uid {
seqSet.AddRange(available[index], available[index+intervalRange-1])
} else {
seqSet.AddRange(index+1, index+intervalRange)
}
list = append(list, seqSet)
available = append(available[:index], available[index+intervalRange:]...)
}
} else {
count := uint32(len(available))
for r := uint32(0); r < count; r++ {
index := rand.Uint32() % (count - r)
seqSet := &imap.SeqSet{}
if uid {
tmp := available[index]
available[index] = available[count-r-1]
seqSet.AddNum(tmp)
} else {
seqSet.AddNum(index)
}
list = append(list, seqSet)
}
}
lists[i] = list
}
return &ParallelSeqSet{seqSets: lists}
}
// NewParallelSeqSetRandom generates count random sequence set for each worker. If generateIntervals is set to true,
// it will generate intervals rather than a single number. If randomDrain is set to true it will generate unique
// values that eventually exhaust the problem space.
func NewParallelSeqSetRandom(count uint32, numWorkers uint, generateIntervals, randomDrain, uid bool) *ParallelSeqSet {
lists := make([][]*imap.SeqSet, numWorkers)
for i := uint(0); i < numWorkers; i++ {
list := make([]*imap.SeqSet, 0, count)
if randomDrain {
available := make([]uint32, count)
for r := uint32(0); r < count; r++ {
available[r] = r + 1
}
if generateIntervals {
const maxIntervalRange = uint32(40)
for len(available) > 0 {
intervalRange := rand.Uint32() % maxIntervalRange
itemsLeft := uint32(len(available))
index := rand.Uint32() % itemsLeft
if index >= itemsLeft {
index = 0
}
if index+intervalRange >= itemsLeft {
intervalRange = itemsLeft - index
}
seqSet := &imap.SeqSet{}
if uid {
seqSet.AddRange(available[index], available[index+intervalRange-1])
} else {
endSeq := index + intervalRange + 1
if endSeq > itemsLeft {
endSeq = itemsLeft
}
seqSet.AddRange(index+1, endSeq)
}
list = append(list, seqSet)
cutIndex := index
if index > 0 {
cutIndex--
}
available = append(available[:cutIndex], available[index+intervalRange:]...)
}
} else {
for r := uint32(0); r < count; r++ {
index := rand.Uint32() % (count - r)
seqSet := &imap.SeqSet{}
if uid {
tmp := available[index]
available[index] = available[count-r-1]
seqSet.AddNum(tmp)
} else {
seqSet.AddNum(index)
}
list = append(list, seqSet)
}
}
} else {
for r := uint32(0); r < count; r++ {
var seqSet *imap.SeqSet
if !generateIntervals {
seqSet = RandomSequenceSetNum(count)
} else {
seqSet = RandomSequenceSetRange(count)
}
list = append(list, seqSet)
}
}
lists[i] = list
}
return &ParallelSeqSet{seqSets: lists}
}
// NewParallelSeqSetAll generates once sequence set for each worker which covers everything (1:*).
func NewParallelSeqSetAll(numWorkers uint) *ParallelSeqSet {
lists := make([][]*imap.SeqSet, numWorkers)
for i := uint(0); i < numWorkers; i++ {
lists[i] = []*imap.SeqSet{NewSequenceSetAll()}
}
return &ParallelSeqSet{seqSets: lists}
}
// NewParallelSeqSet generates a parallel SeqSet based on the following conditions:
// - If a listFile is not empty, it will load the sequence sets from that file.
// - If generateAll is set to true, it will call NewParallelSeqSetAll.
// - If none of the above are valid it will generate random collection of sequence sets which can be single or intervals
// based on whether generateIntervals is set to true.
// If randomDrain is set to true, it will generate non repeating sequences. E.g. Useful for move or delete benchmarks.
// If uid is set to true, it will assume the values are UIDs rather than sequence IDs.
func NewParallelSeqSet(count uint32, numWorkers uint, listFile string, generateAll, generateIntervals, randomDrain, uid bool) (*ParallelSeqSet, error) {
if count == 0 {
return nil, fmt.Errorf("count can not be 0")
}
if len(listFile) != 0 {
return NewParallelSeqSetFromFile(listFile, numWorkers)
} else if generateAll {
return NewParallelSeqSetAll(numWorkers), nil
} else {
return NewParallelSeqSetRandom(count, numWorkers, generateIntervals, randomDrain, uid), nil
}
}
gluon-0.17.0/benchmarks/gluon_bench/imap_benchmarks/search.go 0000664 0000000 0000000 00000013646 14452002130 0024243 0 ustar 00root root 0000000 0000000 package imap_benchmarks
import (
"context"
"flag"
"fmt"
"math/rand"
"net"
"strings"
"time"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/benchmark"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/flags"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/utils"
"github.com/bradenaw/juniper/xslices"
"github.com/emersion/go-imap"
"github.com/emersion/go-imap/client"
)
var (
searchCountFlag = flag.Uint("imap-search-count", 0, "Total number of messages to search during search benchmarks.")
searchTextListFlag = flag.String("imap-search-text-list", "", "Use a list of new line separate search queries instead instead of the default list.")
searchSinceListFlag = flag.String("imap-search-since-list", "", "Use a list of new line dates instead of random generated.")
searchCmdQueryFlag = flag.String("imap-search-cmd", "", "Search command to execute e.g.: \"OR BEFORE <> SINCE <>\"")
)
type SearchQuery interface {
Name() string
Setup(context.Context, *client.Client, uint32) error
Run(context.Context, *client.Client, uint) error
TearDown(context.Context, *client.Client) error
}
type Search struct {
*stateTracker
query SearchQuery
}
func NewSearch(query SearchQuery) benchmark.Benchmark {
return NewIMAPBenchmarkRunner(&Search{stateTracker: newStateTracker(), query: query})
}
func (s *Search) Name() string {
return s.query.Name()
}
func (s *Search) Setup(ctx context.Context, addr net.Addr) error {
return WithClient(addr, func(cl *client.Client) error {
if _, err := s.createAndFillRandomMBox(cl); err != nil {
return err
}
searchCount := uint32(*searchCountFlag)
if searchCount == 0 {
searchCount = uint32(*flags.IMAPMessageCount)
}
if err := s.query.Setup(ctx, cl, searchCount); err != nil {
return err
}
return nil
})
}
func (s *Search) TearDown(ctx context.Context, addr net.Addr) error {
return WithClient(addr, func(cl *client.Client) error {
if err := s.query.TearDown(ctx, cl); err != nil {
return err
}
return s.cleanup(cl)
})
}
func (s *Search) Run(ctx context.Context, addr net.Addr) error {
RunParallelClientsWithMailbox(addr, s.MBoxes[0], true, func(cl *client.Client, index uint) {
if err := s.query.Run(ctx, cl, index); err != nil {
panic(err)
}
})
return nil
}
type SearchTextQuery struct {
queries []string
searchCount uint32
}
func (s *SearchTextQuery) Name() string {
return "imap-search-text"
}
func (s *SearchTextQuery) Setup(ctx context.Context, cl *client.Client, searchCount uint32) error {
if len(*searchTextListFlag) != 0 {
queries, err := utils.ReadLinesFromFile(*searchTextListFlag)
if err != nil {
return err
}
s.queries = queries
} else {
s.queries = strings.Split(utils.MessageEmbedded, " ")
}
s.searchCount = searchCount
return nil
}
func (s *SearchTextQuery) TearDown(ctx context.Context, cl *client.Client) error {
return nil
}
func (s *SearchTextQuery) Run(ctx context.Context, cl *client.Client, workerIndex uint) error {
for i := uint32(0); i < s.searchCount; i++ {
keywordIndex := rand.Intn(len(s.queries))
criteria := imap.NewSearchCriteria()
fieldsStr := []string{"TEXT", s.queries[keywordIndex]}
fields := xslices.Map(fieldsStr, func(v string) interface{} {
return interface{}(v)
})
if err := criteria.ParseWithCharset(fields, nil); err != nil {
panic(err)
}
if _, err := cl.Search(criteria); err != nil {
panic(err)
}
}
return nil
}
type SearchSinceQuery struct {
dates []string
}
func (*SearchSinceQuery) Name() string {
return "imap-search-since"
}
func (s *SearchSinceQuery) Setup(ctx context.Context, cl *client.Client, searchCount uint32) error {
if len(*searchSinceListFlag) != 0 {
dates, err := utils.ReadLinesFromFile(*searchSinceListFlag)
if err != nil {
return err
}
// validate date formats
for _, v := range dates {
if _, err := time.Parse("_2-Jan-2006", v); err != nil {
return fmt.Errorf("invalid date format in list file: %v", v)
}
}
s.dates = dates
} else {
s.dates = make([]string, 0, searchCount)
for i := uint32(0); i < searchCount; i++ {
t := time.Date(1980+rand.Intn(40), time.Month(rand.Intn(12)), rand.Intn(28), 0, 0, 0, 0, time.UTC)
s.dates = append(s.dates, t.Format("02-Jan-2006"))
}
}
return nil
}
func (*SearchSinceQuery) TearDown(ctx context.Context, cl *client.Client) error {
return nil
}
func (s *SearchSinceQuery) Run(ctx context.Context, cl *client.Client, workerIndex uint) error {
for _, d := range s.dates {
criteria := imap.NewSearchCriteria()
fieldsStr := []string{"SINCE", d}
fields := xslices.Map(fieldsStr, func(v string) interface{} {
return interface{}(v)
})
if err := criteria.ParseWithCharset(fields, nil); err != nil {
panic(err)
}
if _, err := cl.Search(criteria); err != nil {
panic(err)
}
}
return nil
}
type SearchCmdQuery struct {
criteria *imap.SearchCriteria
searchCount uint32
}
func (*SearchCmdQuery) Name() string {
return "imap-search-cmd"
}
func (s *SearchCmdQuery) Setup(ctx context.Context, cl *client.Client, searchCount uint32) error {
s.criteria = imap.NewSearchCriteria()
if len(*searchCmdQueryFlag) == 0 {
return fmt.Errorf("please provide a query with -imap-search-cmd")
}
queries := strings.Split(*searchCmdQueryFlag, " ")
fields := xslices.Map(queries, func(v string) interface{} {
return interface{}(v)
})
if err := s.criteria.ParseWithCharset(fields, nil); err != nil {
return err
}
s.searchCount = searchCount
return nil
}
func (*SearchCmdQuery) TearDown(ctx context.Context, cl *client.Client) error {
return nil
}
func (s *SearchCmdQuery) Run(ctx context.Context, cl *client.Client, workerIndex uint) error {
for i := uint32(0); i < s.searchCount; i++ {
if _, err := cl.Search(s.criteria); err != nil {
return err
}
}
return nil
}
func init() {
benchmark.RegisterBenchmark(NewSearch(&SearchSinceQuery{}))
benchmark.RegisterBenchmark(NewSearch(&SearchTextQuery{}))
benchmark.RegisterBenchmark(NewSearch(&SearchCmdQuery{}))
}
gluon-0.17.0/benchmarks/gluon_bench/imap_benchmarks/select.go 0000664 0000000 0000000 00000002555 14452002130 0024252 0 ustar 00root root 0000000 0000000 package imap_benchmarks
import (
"context"
"flag"
"net"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/benchmark"
"github.com/emersion/go-imap/client"
)
var (
selectReadOnlyFlag = flag.Bool("imap-select-readonly", false, "If set to true, perform a read only select (examine).")
selectCallCountFlag = flag.Uint("imap-select-count", 1000, "Number of times to call select.")
)
type Select struct {
*stateTracker
}
func NewSelect() benchmark.Benchmark {
return NewIMAPBenchmarkRunner(&Select{stateTracker: newStateTracker()})
}
func (*Select) Name() string {
return "imap-select"
}
func (s *Select) Setup(ctx context.Context, addr net.Addr) error {
return WithClient(addr, func(cl *client.Client) error {
if _, err := s.createAndFillRandomMBox(cl); err != nil {
return err
}
return nil
})
}
func (s *Select) TearDown(ctx context.Context, addr net.Addr) error {
return s.cleanupWithAddr(addr)
}
func (s *Select) Run(ctx context.Context, addr net.Addr) error {
RunParallelClientsWithMailbox(addr, s.MBoxes[0], *fetchReadOnly, func(cl *client.Client, index uint) {
for i := uint(0); i < *selectCallCountFlag; i++ {
_, err := cl.Select(s.MBoxes[0], *selectReadOnlyFlag)
if err != nil {
panic(err)
}
if err := cl.Unselect(); err != nil {
panic(err)
}
}
})
return nil
}
func init() {
benchmark.RegisterBenchmark(NewSelect())
}
gluon-0.17.0/benchmarks/gluon_bench/imap_benchmarks/select_fetch.go 0000664 0000000 0000000 00000003245 14452002130 0025420 0 ustar 00root root 0000000 0000000 package imap_benchmarks
import (
"context"
"flag"
"github.com/emersion/go-imap"
"net"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/benchmark"
"github.com/bradenaw/juniper/xslices"
"github.com/emersion/go-imap/client"
)
var (
selectFetchRepetitionsFlag = flag.Uint("imap-select-fetch-repeat", 50, "Number of times to repeat the request.")
)
type SelectFetch struct {
*stateTracker
mboxInfo []MailboxInfo
}
func NewSelectFetch() benchmark.Benchmark {
return NewIMAPBenchmarkRunner(&SelectFetch{stateTracker: newStateTracker()})
}
func (*SelectFetch) Name() string {
return "imap-select-fetch"
}
func (e *SelectFetch) Setup(ctx context.Context, addr net.Addr) error {
return WithClient(addr, func(cl *client.Client) error {
for i := uint(0); i < *selectFetchRepetitionsFlag; i++ {
if _, err := e.createAndFillRandomMBox(cl); err != nil {
return err
}
}
e.mboxInfo = xslices.Map(e.MBoxes, func(m string) MailboxInfo {
return MailboxInfo{Name: m, ReadOnly: false}
})
return nil
})
}
func (e *SelectFetch) TearDown(ctx context.Context, addr net.Addr) error {
return e.cleanupWithAddr(addr)
}
func (e *SelectFetch) Run(ctx context.Context, addr net.Addr) error {
RunParallelClients(addr, func(cl *client.Client, u uint) {
for i := uint(0); i < *selectFetchRepetitionsFlag; i++ {
if _, err := cl.Select(e.mboxInfo[i].Name, e.mboxInfo[i].ReadOnly); err != nil {
panic(err)
}
if err := FetchMessage(cl, NewSequenceSetAll(), imap.FetchUid, imap.FetchFlags); err != nil {
panic(err)
}
if err := cl.Unselect(); err != nil {
panic(err)
}
}
})
return nil
}
func init() {
benchmark.RegisterBenchmark(NewSelectFetch())
}
gluon-0.17.0/benchmarks/gluon_bench/imap_benchmarks/server/ 0000775 0000000 0000000 00000000000 14452002130 0023743 5 ustar 00root root 0000000 0000000 gluon-0.17.0/benchmarks/gluon_bench/imap_benchmarks/server/local.go 0000664 0000000 0000000 00000004234 14452002130 0025367 0 ustar 00root root 0000000 0000000 package server
import (
"context"
"fmt"
"net"
"github.com/ProtonMail/gluon"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/flags"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/utils"
"github.com/ProtonMail/gluon/internal/hash"
"github.com/ProtonMail/gluon/profiling"
_ "github.com/mattn/go-sqlite3"
"github.com/sirupsen/logrus"
)
// LocalServer runs a gluon server in the same process as the benchmark process.
type LocalServer struct {
server *gluon.Server
listener net.Listener
}
func (l *LocalServer) Address() net.Addr {
return l.listener.Addr()
}
func (l *LocalServer) Close(ctx context.Context) error {
if err := l.server.Close(ctx); err != nil {
return err
}
return l.listener.Close()
}
type LocalServerBuilder struct{}
func (*LocalServerBuilder) New(ctx context.Context, serverPath string, profiler profiling.CmdProfilerBuilder) (Server, error) {
loggerIn := logrus.StandardLogger().WriterLevel(logrus.TraceLevel)
loggerOut := logrus.StandardLogger().WriterLevel(logrus.TraceLevel)
var opts []gluon.Option
opts = append(opts, gluon.WithLogger(loggerIn, loggerOut))
opts = append(opts, gluon.WithCmdProfiler(profiler))
opts = append(opts, gluon.WithDataDir(serverPath))
opts = append(opts, gluon.WithDatabaseDir(serverPath))
server, err := gluon.New(opts...)
if err != nil {
return nil, err
}
if err := addUser(ctx, server); err != nil {
return nil, err
}
listener, err := net.Listen("tcp", "localhost:1143")
if err != nil {
return nil, err
}
if err := server.Serve(ctx, listener); err != nil {
return nil, err
}
go func() {
for err := range server.GetErrorCh() {
logrus.WithError(err).Error("Error while serving")
}
}()
return &LocalServer{
server: server,
listener: listener,
}, nil
}
func addUser(ctx context.Context, server *gluon.Server) error {
c, err := utils.NewConnector(*flags.Connector)
if err != nil {
return err
}
userID, err := server.AddUser(
ctx,
c.Connector(),
hash.SHA256([]byte(*flags.UserPassword)),
)
if err != nil {
return err
}
if *flags.Verbose {
fmt.Printf("Adding user ID=%v\n", userID)
}
if err := c.Sync(ctx); err != nil {
return err
}
return nil
}
gluon-0.17.0/benchmarks/gluon_bench/imap_benchmarks/server/remote.go 0000664 0000000 0000000 00000001540 14452002130 0025565 0 ustar 00root root 0000000 0000000 package server
import (
"context"
"net"
"github.com/ProtonMail/gluon/profiling"
)
// RemoteServer can't control the start or stopping of the server but can still be used to run the benchmarks
// against an existing server.
type RemoteServer struct {
address net.Addr
}
func (*RemoteServer) Close(ctx context.Context) error {
return nil
}
func (r *RemoteServer) Address() net.Addr {
return r.address
}
type RemoteServerBuilder struct {
address net.Addr
}
func NewRemoteServerBuilder(address string) (*RemoteServerBuilder, error) {
addr, err := net.ResolveTCPAddr("tcp", address)
if err != nil {
return nil, err
}
return &RemoteServerBuilder{address: addr}, nil
}
func (r *RemoteServerBuilder) New(ctx context.Context, serverPath string, profiler profiling.CmdProfilerBuilder) (Server, error) {
return &RemoteServer{address: r.address}, nil
}
gluon-0.17.0/benchmarks/gluon_bench/imap_benchmarks/server/server.go 0000664 0000000 0000000 00000001002 14452002130 0025571 0 ustar 00root root 0000000 0000000 package server
import (
"context"
"net"
"github.com/ProtonMail/gluon/profiling"
)
type Server interface {
// Close should close all server connections or shut down the server.
Close(ctx context.Context) error
// Address should return the server address.
Address() net.Addr
}
type ServerBuilder interface {
// New Create new Server instance at a given path and use the command profiler, if possible.
New(ctx context.Context, serverPath string, profiler profiling.CmdProfilerBuilder) (Server, error)
}
gluon-0.17.0/benchmarks/gluon_bench/imap_benchmarks/state_tracker.go 0000664 0000000 0000000 00000002011 14452002130 0025611 0 ustar 00root root 0000000 0000000 package imap_benchmarks
import (
"net"
"github.com/emersion/go-imap/client"
"github.com/google/uuid"
)
type stateTracker struct {
MBoxes []string
}
func newStateTracker() *stateTracker {
return &stateTracker{}
}
func (s *stateTracker) createRandomMBox(cl *client.Client) (string, error) {
mbox := "Folders/" + uuid.NewString()
if err := cl.Create(mbox); err != nil {
return "", err
}
s.MBoxes = append(s.MBoxes, mbox)
return mbox, nil
}
func (s *stateTracker) createAndFillRandomMBox(cl *client.Client) (string, error) {
mbox, err := s.createRandomMBox(cl)
if err != nil {
return "", err
}
if err := FillMailbox(cl, mbox); err != nil {
return "", err
}
return mbox, nil
}
func (s *stateTracker) cleanup(cl *client.Client) error {
for _, v := range s.MBoxes {
if err := cl.Delete(v); err != nil {
return err
}
}
s.MBoxes = nil
return nil
}
func (s *stateTracker) cleanupWithAddr(addr net.Addr) error {
return WithClient(addr, func(c *client.Client) error {
return s.cleanup(c)
})
}
gluon-0.17.0/benchmarks/gluon_bench/imap_benchmarks/status.go 0000664 0000000 0000000 00000002510 14452002130 0024305 0 ustar 00root root 0000000 0000000 package imap_benchmarks
import (
"context"
"flag"
"net"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/benchmark"
"github.com/emersion/go-imap"
"github.com/emersion/go-imap/client"
)
var statusCallCountFlag = flag.Uint("imap-status-count", 1000, "Number of times to call status.")
type Status struct {
*stateTracker
}
func NewStatus() benchmark.Benchmark {
return NewIMAPBenchmarkRunner(&Status{stateTracker: newStateTracker()})
}
func (*Status) Name() string {
return "imap-status"
}
func (s *Status) Setup(ctx context.Context, addr net.Addr) error {
return WithClient(addr, func(cl *client.Client) error {
if _, err := s.createAndFillRandomMBox(cl); err != nil {
return err
}
return nil
})
}
func (s *Status) TearDown(ctx context.Context, addr net.Addr) error {
return s.cleanupWithAddr(addr)
}
func (s *Status) Run(ctx context.Context, addr net.Addr) error {
RunParallelClientsWithMailbox(addr, s.MBoxes[0], *fetchReadOnly, func(cl *client.Client, index uint) {
for i := uint(0); i < *statusCallCountFlag; i++ {
_, err := cl.Status(s.MBoxes[0], []imap.StatusItem{imap.StatusRecent, imap.StatusMessages, imap.StatusRecent, imap.StatusUnseen, imap.StatusUidNext, imap.StatusUidValidity})
if err != nil {
panic(err)
}
}
})
return nil
}
func init() {
benchmark.RegisterBenchmark(NewStatus())
}
gluon-0.17.0/benchmarks/gluon_bench/imap_benchmarks/store.go 0000664 0000000 0000000 00000005126 14452002130 0024124 0 ustar 00root root 0000000 0000000 package imap_benchmarks
import (
"context"
"flag"
"net"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/benchmark"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/flags"
"github.com/emersion/go-imap"
"github.com/emersion/go-imap/client"
)
var (
storeCountFlag = flag.Uint("imap-store-count", 0, "Total number of messages to store during store benchmarks.")
storeListFlag = flag.String("imap-store-list", "", "Use a list of predefined sequences to store rather than random generated.")
storeSilentFlag = flag.Bool("imap-store-silent", false, "When set to true, request silent updates that do not produce any responses")
storeAllFlag = flag.Bool("imap-store-all", false, "If set, perform one store for all messages.")
)
type StoreBench struct {
*stateTracker
seqSets *ParallelSeqSet
}
func NewStore() benchmark.Benchmark {
return NewIMAPBenchmarkRunner(&StoreBench{stateTracker: newStateTracker()})
}
func (*StoreBench) Name() string {
return "imap-store"
}
func (s *StoreBench) Setup(ctx context.Context, addr net.Addr) error {
return WithClient(addr, func(cl *client.Client) error {
if _, err := s.createAndFillRandomMBox(cl); err != nil {
return err
}
storeCount := uint32(*storeCountFlag)
if storeCount == 0 {
storeCount = uint32(*flags.IMAPMessageCount)
}
seqSets, err := NewParallelSeqSet(storeCount,
*flags.IMAPParallelClients,
*storeListFlag,
*storeAllFlag,
*flags.IMAPRandomSeqSetIntervals,
false,
*flags.IMAPUIDMode)
if err != nil {
return err
}
s.seqSets = seqSets
return nil
})
}
func (s *StoreBench) TearDown(ctx context.Context, addr net.Addr) error {
return s.cleanupWithAddr(addr)
}
func (s *StoreBench) Run(ctx context.Context, addr net.Addr) error {
items := []string{"FLAGS", "-FLAGS", "+FLAGS"}
flagList := []string{imap.DeletedFlag, imap.SeenFlag, imap.AnsweredFlag, imap.FlaggedFlag}
RunParallelClientsWithMailbox(addr, s.MBoxes[0], false, func(cl *client.Client, index uint) {
var storeFn func(*client.Client, *imap.SeqSet, int) error
if *flags.IMAPUIDMode {
storeFn = func(cl *client.Client, set *imap.SeqSet, index int) error {
return UIDStore(cl, set, items[index%len(items)], *storeSilentFlag, flagList[index%len(flagList)])
}
} else {
storeFn = func(cl *client.Client, set *imap.SeqSet, index int) error {
return Store(cl, set, items[index%len(items)], *storeSilentFlag, flagList[index%len(flagList)])
}
}
for s, v := range s.seqSets.Get(index) {
if err := storeFn(cl, v, s); err != nil {
panic(err)
}
}
})
return nil
}
func init() {
benchmark.RegisterBenchmark(NewStore())
}
gluon-0.17.0/benchmarks/gluon_bench/main.go 0000664 0000000 0000000 00000000633 14452002130 0020567 0 ustar 00root root 0000000 0000000 package main
import (
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/benchmark"
_ "github.com/ProtonMail/gluon/benchmarks/gluon_bench/gluon_benchmarks"
_ "github.com/ProtonMail/gluon/benchmarks/gluon_bench/imap_benchmarks"
_ "github.com/ProtonMail/gluon/benchmarks/gluon_bench/store_benchmarks"
"github.com/sirupsen/logrus"
)
func main() {
logrus.SetLevel(logrus.ErrorLevel)
benchmark.RunMain()
}
gluon-0.17.0/benchmarks/gluon_bench/reporter/ 0000775 0000000 0000000 00000000000 14452002130 0021154 5 ustar 00root root 0000000 0000000 gluon-0.17.0/benchmarks/gluon_bench/reporter/json_reporter.go 0000664 0000000 0000000 00000000742 14452002130 0024401 0 ustar 00root root 0000000 0000000 package reporter
import (
"encoding/json"
"os"
)
// JSONReporter produces a JSON data file with all the benchmark information.
type JSONReporter struct {
outputPath string
}
func (j *JSONReporter) ProduceReport(reports []*BenchmarkReport) error {
result, err := json.Marshal(reports)
if err != nil {
return err
}
return os.WriteFile(j.outputPath, []byte(result), 0o600)
}
func NewJSONReporter(output string) *JSONReporter {
return &JSONReporter{outputPath: output}
}
gluon-0.17.0/benchmarks/gluon_bench/reporter/reporter.go 0000664 0000000 0000000 00000007436 14452002130 0023357 0 ustar 00root root 0000000 0000000 package reporter
import (
"fmt"
"math"
"sort"
"strings"
"time"
"github.com/bradenaw/juniper/xslices"
)
type BenchmarkStatistics struct {
Total time.Duration
Average time.Duration
Fastest time.Duration
Slowest time.Duration
Median time.Duration
Percentile90 time.Duration
Percentile10 time.Duration
RMS time.Duration
SampleCount int
Extra BenchmarkExtra
}
func (b *BenchmarkStatistics) String() string {
builder := strings.Builder{}
builder.WriteString(fmt.Sprintf("SampleCount:%04d Total:%v Fastest:%v Slowest:%v Average:%v Median:%v 90thPercentile:%v 10thPercentile:%v RMS:%v",
b.SampleCount, b.Total, b.Fastest, b.Slowest, b.Average,
b.Median, b.Percentile90, b.Percentile10, b.RMS,
))
if b.Extra != nil {
builder.WriteString(" Extra:\n")
builder.WriteString(b.Extra.String())
}
return builder.String()
}
func NewBenchmarkStatistics(extra BenchmarkExtra, durations ...time.Duration) *BenchmarkStatistics {
sortedDurations := durations
sort.Slice(sortedDurations, func(i1, i2 int) bool {
return sortedDurations[i1] < sortedDurations[i2]
})
statistics := &BenchmarkStatistics{
Extra: extra,
}
statistics.SampleCount = len(sortedDurations)
if statistics.SampleCount == 1 {
statistics.Fastest = sortedDurations[0]
statistics.Slowest = sortedDurations[0]
statistics.Average = sortedDurations[0]
statistics.Total = sortedDurations[0]
statistics.Median = sortedDurations[0]
statistics.Percentile90 = sortedDurations[0]
statistics.Percentile10 = sortedDurations[0]
statistics.RMS = sortedDurations[0]
} else if statistics.SampleCount > 1 {
statistics.Fastest = sortedDurations[0]
statistics.Slowest = sortedDurations[statistics.SampleCount-1]
statistics.Total = xslices.Reduce(sortedDurations, 0, func(v1 time.Duration, v2 time.Duration) time.Duration {
return v1 + v2
})
statistics.Average = statistics.Total / time.Duration(statistics.SampleCount)
if statistics.Total%2 == 0 {
halfPoint := statistics.SampleCount / 2
statistics.Median = (sortedDurations[halfPoint-1] + sortedDurations[halfPoint]) / 2
} else {
statistics.Median = sortedDurations[((statistics.SampleCount+1)/2)-1]
}
statistics.Percentile90 = sortedDurations[int(math.Floor(float64(statistics.SampleCount)*(90.0/100.0)))]
statistics.Percentile10 = sortedDurations[int(math.Floor(float64(statistics.SampleCount)*(10.0/100.0)))]
var sumSquaredWithDiv float64
for i := 0; i < statistics.SampleCount; i++ {
// Dividing now rather than later or else we will trigger overflow.
f64Duration := float64(sortedDurations[i])
sumSquaredWithDiv += (f64Duration * f64Duration) / float64(statistics.SampleCount)
}
statistics.RMS = time.Duration(math.Round(math.Sqrt(sumSquaredWithDiv)))
}
return statistics
}
type BenchmarkExtra interface {
String() string
}
type BenchmarkRun struct {
Durations []time.Duration
Extra BenchmarkExtra
}
func NewBenchmarkRunSingle(duration time.Duration, extra BenchmarkExtra) *BenchmarkRun {
return &BenchmarkRun{Durations: []time.Duration{duration}, Extra: extra}
}
func NewBenchmarkRun(durations []time.Duration, extra BenchmarkExtra) *BenchmarkRun {
return &BenchmarkRun{Durations: durations, Extra: extra}
}
type BenchmarkReport struct {
Name string
Runs []*BenchmarkStatistics
Statistics *BenchmarkStatistics
}
func NewBenchmarkReport(name string, runs ...*BenchmarkStatistics) *BenchmarkReport {
durations := xslices.Map(runs, func(r *BenchmarkStatistics) time.Duration {
return r.Total
})
return &BenchmarkReport{Name: name, Runs: runs, Statistics: NewBenchmarkStatistics(nil, durations...)}
}
// BenchmarkReporter is the interface that is required to be implemented by any report generation tool.
type BenchmarkReporter interface {
ProduceReport(reports []*BenchmarkReport) error
}
gluon-0.17.0/benchmarks/gluon_bench/reporter/stdout_reporter.go 0000664 0000000 0000000 00000000672 14452002130 0024754 0 ustar 00root root 0000000 0000000 package reporter
import (
"fmt"
)
// StdOutReporter prints the benchmark report to os.Stdout.
type StdOutReporter struct{}
func (*StdOutReporter) ProduceReport(reports []*BenchmarkReport) error {
for i, v := range reports {
fmt.Printf("[%02d] Benchmark %v\n", i, v.Name)
fmt.Printf("[%02d] %v\n", i, v.Statistics.String())
for r, v := range v.Runs {
fmt.Printf("[%02d] Run %02d - %v\n", i, r, v.String())
}
}
return nil
}
gluon-0.17.0/benchmarks/gluon_bench/store_benchmarks/ 0000775 0000000 0000000 00000000000 14452002130 0022643 5 ustar 00root root 0000000 0000000 gluon-0.17.0/benchmarks/gluon_bench/store_benchmarks/create.go 0000664 0000000 0000000 00000002232 14452002130 0024434 0 ustar 00root root 0000000 0000000 package store_benchmarks
import (
"bytes"
"context"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/benchmark"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/flags"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/reporter"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/timing"
"github.com/ProtonMail/gluon/imap"
"github.com/ProtonMail/gluon/store"
)
type Create struct{}
func (*Create) Name() string {
return "store-create"
}
func (*Create) Setup(ctx context.Context, store store.Store) error {
return nil
}
func (*Create) TearDown(ctx context.Context, store store.Store) error {
return nil
}
func (*Create) Run(ctx context.Context, st store.Store) (*reporter.BenchmarkRun, error) {
return RunStoreWorkers(ctx, st, func(ctx context.Context, s store.Store, dc *timing.Collector, u uint) error {
data := make([]byte, *flags.StoreItemSize)
for i := uint(0); i < *flags.StoreItemCount; i++ {
dc.Start()
err := s.Set(imap.NewInternalMessageID(), bytes.NewReader(data))
dc.Stop()
if err != nil {
return err
}
}
return nil
}), nil
}
func init() {
benchmark.RegisterBenchmark(NewStoreBenchmarkRunner(&Create{}))
}
gluon-0.17.0/benchmarks/gluon_bench/store_benchmarks/delete.go 0000664 0000000 0000000 00000002341 14452002130 0024434 0 ustar 00root root 0000000 0000000 package store_benchmarks
import (
"context"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/benchmark"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/flags"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/reporter"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/timing"
"github.com/ProtonMail/gluon/imap"
"github.com/ProtonMail/gluon/store"
)
type Delete struct {
uuids []imap.InternalMessageID
}
func (*Delete) Name() string {
return "store-delete"
}
func (d *Delete) Setup(ctx context.Context, s store.Store) error {
uuids, err := CreateRandomState(s, *flags.StoreItemCount)
if err != nil {
return err
}
d.uuids = uuids
return nil
}
func (*Delete) TearDown(ctx context.Context, store store.Store) error {
return nil
}
func (d *Delete) Run(ctx context.Context, st store.Store) (*reporter.BenchmarkRun, error) {
return RunStoreWorkersSplitRange(ctx, st, uint(len(d.uuids)), func(ctx context.Context, s store.Store, dc *timing.Collector, start, end uint) error {
for i := start; i < end; i++ {
dc.Start()
err := s.Delete(d.uuids[i])
dc.Stop()
if err != nil {
panic(err)
}
}
return nil
}), nil
}
func init() {
benchmark.RegisterBenchmark(NewStoreBenchmarkRunner(&Delete{}))
}
gluon-0.17.0/benchmarks/gluon_bench/store_benchmarks/get.go 0000664 0000000 0000000 00000002361 14452002130 0023753 0 ustar 00root root 0000000 0000000 package store_benchmarks
import (
"context"
"math/rand"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/benchmark"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/flags"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/reporter"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/timing"
"github.com/ProtonMail/gluon/imap"
"github.com/ProtonMail/gluon/store"
)
type Get struct {
uuids []imap.InternalMessageID
}
func (*Get) Name() string {
return "store-get"
}
func (g *Get) Setup(ctx context.Context, s store.Store) error {
uuids, err := CreateRandomState(s, *flags.StoreItemCount)
if err != nil {
return err
}
g.uuids = uuids
return nil
}
func (*Get) TearDown(ctx context.Context, store store.Store) error {
return nil
}
func (g *Get) Run(ctx context.Context, st store.Store) (*reporter.BenchmarkRun, error) {
uuidLen := len(g.uuids)
return RunStoreWorkers(ctx, st, func(ctx context.Context, s store.Store, dc *timing.Collector, u uint) error {
for i := 0; i < uuidLen; i++ {
index := rand.Intn(uuidLen)
dc.Start()
_, err := s.Get(g.uuids[index])
dc.Stop()
if err != nil {
panic(err)
}
}
return nil
}), nil
}
func init() {
benchmark.RegisterBenchmark(NewStoreBenchmarkRunner(&Get{}))
}
gluon-0.17.0/benchmarks/gluon_bench/store_benchmarks/store.go 0000664 0000000 0000000 00000000704 14452002130 0024327 0 ustar 00root root 0000000 0000000 package store_benchmarks
import (
"path/filepath"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/flags"
"github.com/ProtonMail/gluon/store"
"github.com/google/uuid"
)
type OnDiskStoreBuilder struct{}
func (*OnDiskStoreBuilder) New(path string) (store.Store, error) {
return store.NewOnDiskStore(filepath.Join(path, uuid.NewString()), []byte(*flags.UserPassword))
}
func init() {
RegisterStoreBuilder("default", &OnDiskStoreBuilder{})
}
gluon-0.17.0/benchmarks/gluon_bench/store_benchmarks/store_benchmark.go 0000664 0000000 0000000 00000003361 14452002130 0026343 0 ustar 00root root 0000000 0000000 package store_benchmarks
import (
"context"
"os"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/benchmark"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/flags"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/reporter"
"github.com/ProtonMail/gluon/store"
)
type StoreBenchmark interface {
// Name returns benchmark's name.
Name() string
// Setup should prepare the benchmark.
Setup(ctx context.Context, store store.Store) error
// TearDown should clean the benchmark.
TearDown(ctx context.Context, store store.Store) error
// Run the benchmark.
Run(ctx context.Context, store store.Store) (*reporter.BenchmarkRun, error)
}
type StoreBenchmarkRunner struct {
benchmark StoreBenchmark
benchmarkDir string
store store.Store
}
func (s *StoreBenchmarkRunner) Name() string {
return s.benchmark.Name()
}
func (s *StoreBenchmarkRunner) Setup(ctx context.Context, benchmarkDir string) error {
store, err := NewStore(*flags.Store, benchmarkDir)
if err != nil {
return err
}
s.store = store
s.benchmarkDir = benchmarkDir
if err := s.benchmark.Setup(ctx, s.store); err != nil {
return err
}
return nil
}
func (s *StoreBenchmarkRunner) Run(ctx context.Context) (*reporter.BenchmarkRun, error) {
benchRuns, err := s.benchmark.Run(ctx, s.store)
if err != nil {
return nil, err
}
return benchRuns, nil
}
func (s *StoreBenchmarkRunner) TearDown(ctx context.Context) error {
if err := s.benchmark.TearDown(ctx, s.store); err != nil {
return err
}
if err := s.store.Close(); err != nil {
return err
}
if err := os.RemoveAll(s.benchmarkDir); err != nil {
return err
}
return nil
}
func NewStoreBenchmarkRunner(bench StoreBenchmark) benchmark.Benchmark {
return &StoreBenchmarkRunner{benchmark: bench}
}
gluon-0.17.0/benchmarks/gluon_bench/store_benchmarks/store_factory.go 0000664 0000000 0000000 00000002022 14452002130 0026051 0 ustar 00root root 0000000 0000000 package store_benchmarks
import (
"fmt"
"github.com/ProtonMail/gluon/store"
)
type StoreBuilder interface {
New(path string) (store.Store, error)
}
type storeFactory struct {
builders map[string]StoreBuilder
}
func newStoreFactory() *storeFactory {
return &storeFactory{builders: make(map[string]StoreBuilder)}
}
func (sf *storeFactory) Register(name string, builder StoreBuilder) error {
if _, ok := sf.builders[name]; ok {
return fmt.Errorf("builder already exists")
}
sf.builders[name] = builder
return nil
}
func (sf *storeFactory) New(name, path string) (store.Store, error) {
builder, ok := sf.builders[name]
if !ok {
return nil, fmt.Errorf("no such builder exists")
}
return builder.New(path)
}
var storeFactoryInstance = newStoreFactory()
func RegisterStoreBuilder(name string, storeBuilder StoreBuilder) {
if err := storeFactoryInstance.Register(name, storeBuilder); err != nil {
panic(err)
}
}
func NewStore(name, path string) (store.Store, error) {
return storeFactoryInstance.New(name, path)
}
gluon-0.17.0/benchmarks/gluon_bench/store_benchmarks/utils.go 0000664 0000000 0000000 00000003703 14452002130 0024335 0 ustar 00root root 0000000 0000000 package store_benchmarks
import (
"bytes"
"context"
"sync"
"time"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/flags"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/reporter"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/timing"
"github.com/ProtonMail/gluon/imap"
"github.com/ProtonMail/gluon/store"
)
func CreateRandomState(st store.Store, count uint) ([]imap.InternalMessageID, error) {
uuids := make([]imap.InternalMessageID, 0, count)
data := make([]byte, *flags.StoreItemSize)
for i := uint(0); i < count; i++ {
uuid := imap.NewInternalMessageID()
if err := st.Set(uuid, bytes.NewReader(data)); err != nil {
return nil, err
}
uuids = append(uuids, uuid)
}
return uuids, nil
}
func RunStoreWorkers(ctx context.Context, st store.Store, fn func(context.Context, store.Store, *timing.Collector, uint) error) *reporter.BenchmarkRun {
wg := sync.WaitGroup{}
durations := make([]time.Duration, 0, *flags.StoreWorkers**flags.StoreItemCount)
collectors := make([]*timing.Collector, *flags.StoreWorkers)
for i := uint(0); i < *flags.StoreWorkers; i++ {
wg.Add(1)
go func(index uint) {
defer wg.Done()
collector := timing.NewDurationCollector(int(*flags.StoreItemCount))
if err := fn(ctx, st, collector, index); err != nil {
panic(err)
}
collectors[index] = collector
}(i)
}
wg.Wait()
for _, v := range collectors {
durations = append(durations, v.Durations()...)
}
return reporter.NewBenchmarkRun(durations, nil)
}
func RunStoreWorkersSplitRange(ctx context.Context, st store.Store, length uint, fn func(context.Context, store.Store, *timing.Collector, uint, uint) error) *reporter.BenchmarkRun {
workDivision := length / *flags.StoreWorkers
return RunStoreWorkers(ctx, st, func(ctx context.Context, s store.Store, collector *timing.Collector, u uint) error {
end := workDivision * (u + 1)
if end > length {
end = length
}
return fn(ctx, st, collector, u*workDivision, end)
})
}
gluon-0.17.0/benchmarks/gluon_bench/timing/ 0000775 0000000 0000000 00000000000 14452002130 0020601 5 ustar 00root root 0000000 0000000 gluon-0.17.0/benchmarks/gluon_bench/timing/timing.go 0000664 0000000 0000000 00000001356 14452002130 0022424 0 ustar 00root root 0000000 0000000 package timing
import "time"
// Timer tracks the duration between invocations to Start and Stop.
type Timer struct {
start time.Time
end time.Time
}
func (s *Timer) Start() {
s.start = time.Now()
}
func (s *Timer) Stop() {
s.end = time.Now()
}
func (s *Timer) Elapsed() time.Duration {
return s.end.Sub(s.start)
}
type Collector struct {
durations []time.Duration
timer Timer
}
func NewDurationCollector(capacity int) *Collector {
return &Collector{
durations: make([]time.Duration, 0, capacity),
}
}
func (d *Collector) Start() {
d.timer.Start()
}
func (d *Collector) Stop() {
d.timer.Stop()
d.durations = append(d.durations, d.timer.Elapsed())
}
func (d *Collector) Durations() []time.Duration {
return d.durations
}
gluon-0.17.0/benchmarks/gluon_bench/tools/ 0000775 0000000 0000000 00000000000 14452002130 0020452 5 ustar 00root root 0000000 0000000 gluon-0.17.0/benchmarks/gluon_bench/tools/compare_bench_output.go 0000664 0000000 0000000 00000005041 14452002130 0025206 0 ustar 00root root 0000000 0000000 package main
import (
"encoding/json"
"fmt"
"os"
"time"
)
// we redefine the json statistic types here since we don't care about the extra information
type JSONBenchmarkStatistics struct {
Total time.Duration
Average time.Duration
Fastest time.Duration
Slowest time.Duration
Median time.Duration
Percentile90 time.Duration
Percentile10 time.Duration
RMS time.Duration
SampleCount int
}
type JSONBenchmarkReport struct {
Name string
Runs []*JSONBenchmarkStatistics
Statistics *JSONBenchmarkStatistics
}
func loadReportFromFile(path string) ([]*JSONBenchmarkReport, error) {
contents, err := os.ReadFile(path)
if err != nil {
return nil, err
}
var reports []*JSONBenchmarkReport
if err := json.Unmarshal(contents, &reports); err != nil {
return nil, err
}
return reports, nil
}
type BenchmarkRun struct {
fileIndex int
statistics *JSONBenchmarkStatistics
}
func main() {
if len(os.Args) < 3 {
fmt.Fprintf(os.Stderr, "Usage %v json_report0 json_report1... json_report N\n", os.Args[0])
return
}
// load reports
reportFiles := os.Args[1:]
reports := make([][]*JSONBenchmarkReport, 0, len(reportFiles))
for _, v := range reportFiles {
report, err := loadReportFromFile(v)
if err != nil {
panic(fmt.Errorf("failed to load report: %w", err))
}
reports = append(reports, report)
}
benchMap := map[string][]BenchmarkRun{}
for idx, report := range reports {
for _, run := range report {
b := BenchmarkRun{fileIndex: idx, statistics: run.Statistics}
v, ok := benchMap[run.Name]
if ok {
v = append(v, b)
} else {
v = []BenchmarkRun{b}
}
benchMap[run.Name] = v
}
}
for k, v := range benchMap {
if len(v) == 1 {
fmt.Printf("Benchmark %v: Only has one run\n", k)
continue
}
// check if all benchmarks have the same benchmark runs
{
expectedRuns := v[0].statistics.SampleCount
for i := 1; i < len(v); i++ {
if v[i].statistics.SampleCount != expectedRuns {
fmt.Fprintf(os.Stderr, "Benchmark %v: File '%v' does not have the expected sample count (%v)\n",
k, reportFiles[v[i].fileIndex], expectedRuns)
continue
}
}
}
// Check which run has the best 90th percentile
{
fastest := v[0].statistics.Percentile90
fastestIndex := 0
for i := 1; i < len(v); i++ {
if v[i].statistics.Percentile90 < fastest {
fastestIndex = i
fastest = v[i].statistics.Percentile90
}
}
fmt.Printf("Benchmark %v: Fastest (90th Percentile=%v) %v\n", k, fastest, reportFiles[fastestIndex])
}
}
}
gluon-0.17.0/benchmarks/gluon_bench/utils/ 0000775 0000000 0000000 00000000000 14452002130 0020452 5 ustar 00root root 0000000 0000000 gluon-0.17.0/benchmarks/gluon_bench/utils/cmd_profiler.go 0000664 0000000 0000000 00000003602 14452002130 0023447 0 ustar 00root root 0000000 0000000 package utils
import (
"sync"
"time"
"github.com/ProtonMail/gluon/profiling"
)
// DurationCmdProfiler records the duration of the duration between invocations of IMAP Commands.
type DurationCmdProfiler struct {
durations [profiling.CmdTypeTotal][]time.Duration
start [profiling.CmdTypeTotal]time.Time
}
func (c *DurationCmdProfiler) Start(cmdType int) {
// We can use time since Go 1.9 they have switch to monotonic clocks.
c.start[cmdType] = time.Now()
}
func (c *DurationCmdProfiler) Stop(cmdType int) {
elapsed := time.Since(c.start[cmdType])
c.durations[cmdType] = append(c.durations[cmdType], elapsed)
}
func NewDurationCmdProfiler() *DurationCmdProfiler {
profiler := &DurationCmdProfiler{}
for i := 0; i < len(profiler.durations); i++ {
profiler.durations[i] = make([]time.Duration, 0, 128)
}
return profiler
}
type DurationCmdProfilerBuilder struct {
mutex sync.Mutex
profilers []*DurationCmdProfiler
}
func (c *DurationCmdProfilerBuilder) New() profiling.CmdProfiler {
return NewDurationCmdProfiler()
}
func (c *DurationCmdProfilerBuilder) Collect(profiler profiling.CmdProfiler) {
switch v := profiler.(type) {
case *DurationCmdProfiler:
c.mutex.Lock()
defer c.mutex.Unlock()
c.profilers = append(c.profilers, v)
}
}
// Merge merges all collected command profilers into a single timing Calculator for each IMAP command.
func (c *DurationCmdProfilerBuilder) Merge() [profiling.CmdTypeTotal][]time.Duration {
c.mutex.Lock()
defer c.mutex.Unlock()
var result [profiling.CmdTypeTotal][]time.Duration
for _, v := range c.profilers {
for i := 0; i < len(result); i++ {
result[i] = append(result[i], v.durations[i]...)
}
}
return result
}
func (c *DurationCmdProfilerBuilder) Clear() {
c.mutex.Lock()
defer c.mutex.Unlock()
c.profilers = nil
}
func NewDurationCmdProfilerBuilder() *DurationCmdProfilerBuilder {
return &DurationCmdProfilerBuilder{}
}
gluon-0.17.0/benchmarks/gluon_bench/utils/connector_factory.go 0000664 0000000 0000000 00000004061 14452002130 0024523 0 ustar 00root root 0000000 0000000 package utils
import (
"context"
"fmt"
"time"
"github.com/ProtonMail/gluon/benchmarks/gluon_bench/flags"
"github.com/ProtonMail/gluon/connector"
"github.com/ProtonMail/gluon/imap"
)
type ConnectorImpl interface {
Connector() connector.Connector
Sync(ctx context.Context) error
}
type ConnectorBuilder interface {
New() (ConnectorImpl, error)
}
type connectorFactory struct {
connectors map[string]ConnectorBuilder
}
func newConnectorFactory() *connectorFactory {
return &connectorFactory{
connectors: make(map[string]ConnectorBuilder),
}
}
func (c *connectorFactory) register(name string, builder ConnectorBuilder) error {
if _, ok := c.connectors[name]; ok {
return fmt.Errorf("connector '%v' already exists", name)
}
c.connectors[name] = builder
return nil
}
func (c *connectorFactory) new(name string) (ConnectorImpl, error) {
builder, ok := c.connectors[name]
if !ok {
return nil, fmt.Errorf("no such connector available: '%v'", name)
}
return builder.New()
}
var connectorFactoryInstance = newConnectorFactory()
func RegisterConnector(name string, builder ConnectorBuilder) error {
return connectorFactoryInstance.register(name, builder)
}
func NewConnector(name string) (ConnectorImpl, error) {
return connectorFactoryInstance.new(name)
}
type DummyConnectorBuilder struct{}
type DummyConnectorImpl struct {
dummy *connector.Dummy
}
func (d *DummyConnectorImpl) Connector() connector.Connector {
return d.dummy
}
func (d *DummyConnectorImpl) Sync(ctx context.Context) error {
d.dummy.ClearUpdates()
return d.dummy.Sync(ctx)
}
func (*DummyConnectorBuilder) New() (ConnectorImpl, error) {
addresses := []string{*flags.UserName}
connector := connector.NewDummy(
addresses,
[]byte(*flags.UserPassword),
time.Second,
imap.NewFlagSet(`\Answered`, `\Seen`, `\Flagged`, `\Deleted`),
imap.NewFlagSet(`\Answered`, `\Seen`, `\Flagged`, `\Deleted`),
imap.NewFlagSet(),
)
return &DummyConnectorImpl{dummy: connector}, nil
}
func init() {
if err := RegisterConnector("dummy", &DummyConnectorBuilder{}); err != nil {
panic(err)
}
}
gluon-0.17.0/benchmarks/gluon_bench/utils/messages.go 0000664 0000000 0000000 00000006151 14452002130 0022613 0 ustar 00root root 0000000 0000000 package utils
// Hardcoded messages used to generate mailboxes
const MessageMultiPartMixed = `Return-Path:
Received: from [10.1.1.121] ([185.159.157.131])
by smtp.gmail.com with ESMTPSA id t8sm14889112wrr.10.2021.03.26.12.01.23
for
(version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
Fri, 26 Mar 2021 12:01:24 -0700 (PDT)
To: somebody@gmail.com
From: BQA
Subject: Simple test mail
Message-ID:
Date: Fri, 26 Mar 2021 20:01:23 +0100
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0)
Gecko/20100101 Thunderbird/78.8.1
MIME-Version: 1.0
Content-Type: multipart/mixed;
boundary="------------4AC5F36D876D5EED478B5FF9"
Content-Language: en-US
This is a multi-part message in MIME format.
--------------4AC5F36D876D5EED478B5FF9
Content-Type: multipart/alternative;
boundary="------------62DCF50B21CF279F489F0184"
--------------62DCF50B21CF279F489F0184
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
*this */is**/_html_
**
--------------62DCF50B21CF279F489F0184
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 7bit
this is html
--------------62DCF50B21CF279F489F0184--
--------------4AC5F36D876D5EED478B5FF9
Content-Type: text/plain; charset=UTF-8; x-mac-type="0"; x-mac-creator="0";
name="thing.txt"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
filename="thing.txt"
dGhpcyBpcyBteSBhdHRhY2htZW50Cg==
--------------4AC5F36D876D5EED478B5FF9--
`
const MessageAfterNoonMeeting = `Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST)
From: Fred Foobar
Subject: afternoon meeting
Date: Fri, 26 Mar 2021 20:01:23 +0100
To: mooch@owatagu.siam.edu
Message-Id:
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
Hello Joe, do you think we can meet at 3:30 tomorrow?
`
const MessageEmbedded = `From: Nathaniel Borenstein
To: Ned Freed
Subject: Sample message
MIME-Version: 1.0
Date: Fri, 26 Mar 2021 20:01:23 +0100
Content-type: multipart/mixed; boundary="simple boundary"
This is the preamble. It is to be ignored, though it
is a handy place for mail composers to include an
explanatory note to non-MIME compliant readers.
--simple boundary
Content-type: text/plain; charset=us-ascii
This part does not end with a linebreak.
--simple boundary
Content-Disposition: attachment; filename=test.eml
Content-Type: message/rfc822; name=test.eml
X-Pm-Content-Encryption: on-import
To: someone
Subject: Fwd: embedded
Content-type: multipart/mixed; boundary="embedded-boundary"
--embedded-boundary
Content-type: text/plain; charset=us-ascii
This part is embedded
--
From me
--embedded-boundary
Content-type: text/plain; charset=us-ascii
This part is also embedded
--embedded-boundary--
--simple boundary--
This is the epilogue. It is also to be ignored.`
gluon-0.17.0/benchmarks/gluon_bench/utils/utils.go 0000664 0000000 0000000 00000002136 14452002130 0022143 0 ustar 00root root 0000000 0000000 package utils
import (
"bufio"
"io/fs"
"os"
"path/filepath"
"strings"
)
func ReadLinesFromFile(path string) ([]string, error) {
readFile, err := os.Open(path)
if err != nil {
return nil, err
}
defer readFile.Close()
fileScanner := bufio.NewScanner(readFile)
fileScanner.Split(bufio.ScanLines)
lines := make([]string, 0, 16)
for fileScanner.Scan() {
lines = append(lines, fileScanner.Text())
}
return lines, nil
}
func LoadFilesFromDirectory(path string, filter func(string, fs.FileInfo) bool) ([]string, error) {
var files []string
if err := filepath.Walk(path, func(path string, info fs.FileInfo, err error) error {
if err != nil {
return err
}
if !filter(path, info) {
return nil
}
bytes, err := os.ReadFile(path)
if err != nil {
return err
}
files = append(files, string(bytes))
return nil
}); err != nil {
return nil, err
}
return files, nil
}
func LoadEMLFilesFromDirectory(path string) ([]string, error) {
return LoadFilesFromDirectory(path, func(s string, info fs.FileInfo) bool {
return !info.IsDir() && strings.HasSuffix(s, ".eml")
})
}
gluon-0.17.0/benchmarks/imaptest/ 0000775 0000000 0000000 00000000000 14452002130 0016655 5 ustar 00root root 0000000 0000000 gluon-0.17.0/benchmarks/imaptest/README.md 0000664 0000000 0000000 00000003343 14452002130 0020137 0 ustar 00root root 0000000 0000000 # ImapTest - Profiling & Compliance
This "benchmark" uses [Dovecot's ImapTest tool](https://imapwiki.org/ImapTest) to profile the performance of Gluon. The
information present here can also be used to verify the compliance of Gluon with the IMAP protocol.
Build gluon demo in project root folder:
```bash
go build -o gluon-demo ./demo/demo.go
```
## Installation of ImapTest
Follow the instructions outlined in [the tools' installation page](https://imapwiki.org/ImapTest/Installation)
to build the binary. The test mailbox is already present in this folder.
## Simple ImapTest run
Assuming gluon demo is running, the bare minimum required for running ImapTest is a username and a password:
```bash
imaptest host=127.0.0.1 port=1143 user=user1@example.com pass=password1
```
## Advance testing
The multiple scenario coverage can be run by
```
go test
```
The test cases are defined in `benchmark.yml`. Each case defines a number of
clients and users to be used by ImapTest. One case can have multiple
settings defined by name. The options are specified in `settings` section. The
settings reflects ImapTest options as described in
[here](https://imapwiki.org/ImapTest/Running).
The ImapTest states are described in
[here](https://imapwiki.org/ImapTest/States).
We don't use for now the ImapTest scriptable scenarios but it is
possible by defining the new test settings in file `./benchmark.yml` and
creating separate definition file like example
[here](https://github.com/dovecot/imaptest/tree/main/src/tests)
## Note about this tool
The execution of this tool is non-deterministic, this means it can't be used to
compare profile runs of two different versions.
It should be only be used to profile and/or stress the codebase in an
isolation.
gluon-0.17.0/benchmarks/imaptest/benchmark.yml 0000664 0000000 0000000 00000001621 14452002130 0021332 0 ustar 00root root 0000000 0000000
---
cases:
- users: 1
clients: 1
settings:
- simple
- full
- users: 1
clients: 10
settings:
- simple
- users: 1
clients: 100
settings:
- simple
- users: 10
clients: 10
settings:
- simple
- users: 10
clients: 100
settings:
- simple
- users: 100
clients: 100
settings:
- simple
settings:
simple:
mbox: dovecot-crlf
secs: 10
no_pipelining: true
simple-with-checks:
mbox: dovecot-crlf
secs: 10
checkpoint: 3
no_pipelining: true
own_msgs: true
own_flags: true
full:
mbox: dovecot-crlf
no_pipelining: false
secs: 60
mcreate: 50
mdelete: 50
uidf: 50
search: 30
noop: 15
fetch: 50
login: 100
logout: 100
list: 50
select: 100
fet2: 100,30
copy: 30,5
store: 50
delete: 100
expunge: 100
append: 100,5
gluon-0.17.0/benchmarks/imaptest/dovecot-crlf 0000664 0000000 0000000 00050764077 14452002130 0021221 0 ustar 00root root 0000000 0000000 From cras@irccrew.org Tue Jul 23 19:39:23 2002
Received: with ECARTIS (v1.0.0; list dovecot); Tue, 23 Jul 2002 19:39:23 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from shodan.irccrew.org (shodan.irccrew.org [80.83.4.2])
by danu.procontrol.fi (Postfix) with ESMTP id 434B423848
for ; Tue, 23 Jul 2002 19:39:23 +0300 (EEST)
Received: by shodan.irccrew.org (Postfix, from userid 6976)
id 175FA4C0A0; Tue, 23 Jul 2002 19:39:23 +0300 (EEST)
Date: Tue, 23 Jul 2002 19:39:23 +0300
From: Timo Sirainen
To: dovecot@procontrol.fi
Subject: [dovecot] first test mail
Message-ID: <20020723193923.J22431@irccrew.org>
Mime-Version: 1.0
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
Content-Type: text/plain; charset=us-ascii
X-archive-position: 1
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-IMAPbase: 1096038620 0000010517
X-UID: 1
Status: O
lets see if it works
From cras@irccrew.org Mon Jul 29 02:17:12 2002
Received: with ECARTIS (v1.0.0; list dovecot); Mon, 29 Jul 2002 02:17:12 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from shodan.irccrew.org (shodan.irccrew.org [80.83.4.2])
by danu.procontrol.fi (Postfix) with ESMTP id 8D21723848
for ; Mon, 29 Jul 2002 02:17:12 +0300 (EEST)
Received: by shodan.irccrew.org (Postfix, from userid 6976)
id 8BAD24C0A0; Mon, 29 Jul 2002 02:17:11 +0300 (EEST)
Date: Mon, 29 Jul 2002 02:17:11 +0300
From: Timo Sirainen
To: dovecot@procontrol.fi
Subject: [dovecot] Dovecot 0.93 released
Message-ID: <20020729021711.W22431@irccrew.org>
Mime-Version: 1.0
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
Content-Type: text/plain; charset=us-ascii
X-archive-position: 2
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 2
Status: O
First alpha quality release, everything critical is now implemented. From
now on it's mostly stabilization and optimization. Features that can't break
existing code could still be added, especially SSL and authentication stuff.
From cras@irccrew.org Wed Jul 31 22:48:41 2002
Received: with ECARTIS (v1.0.0; list dovecot); Wed, 31 Jul 2002 22:48:41 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from shodan.irccrew.org (shodan.irccrew.org [80.83.4.2])
by danu.procontrol.fi (Postfix) with ESMTP id F141123829
for ; Wed, 31 Jul 2002 22:48:40 +0300 (EEST)
Received: by shodan.irccrew.org (Postfix, from userid 6976)
id 42ED44C0A0; Wed, 31 Jul 2002 22:48:40 +0300 (EEST)
Date: Wed, 31 Jul 2002 22:48:39 +0300
From: Timo Sirainen
To: dovecot@procontrol.fi
Subject: [dovecot] v0.95 released
Message-ID: <20020731224839.H22431@irccrew.org>
Mime-Version: 1.0
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
Content-Type: text/plain; charset=us-ascii
X-archive-position: 3
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 3
Status: O
v0.95 2002-07-31 Timo Sirainen
+ Initial SSL support using GNU TLS, tested with v0.5.1.
TLS support is still missing.
+ Digest-MD5 authentication method
+ passwd-file authentication backend
+ Code cleanups
- Found several bugs from mempool and ioloop code, now we should
be stable? :)
- A few corrections for long header field handling
From return@trafficmagnet.com Mon Aug 5 19:26:52 2002
Received: with ECARTIS (v1.0.0; list dovecot); Mon, 05 Aug 2002 19:26:52 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from ns5.trafficmagnet.net (unknown [211.157.101.52])
by danu.procontrol.fi (Postfix) with ESMTP id 48C2C23831
for ; Mon, 5 Aug 2002 19:26:51 +0300 (EEST)
Received: from 181-Dispatcher ([211.101.236.181])
by ns5.trafficmagnet.net (8.11.6/8.11.6) with SMTP id g765MXt31378
for ; Tue, 6 Aug 2002 00:22:34 -0500
Message-Id: <200208060522.g765MXt31378@ns5.trafficmagnet.net>
From: Sarah Williams
To: "dovecot@procontrol.fi"
Subject: [dovecot] DOVECOT.PROCONTROL.FI
Date: Tue, 6 Aug 2002 0:29:18 +0800
X-Mailer: CSMTPConnection v2.17
MIME-Version: 1.0
Content-Type: multipart/related; boundary="956bff02-8aec-485e-a58c-40fda617ecbe"
Content-Transfer-Encoding: quoted-printable
Reply-To: Sarah Williams
X-archive-position: 4
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: return@trafficmagnet.com
Precedence: bulk
X-list: dovecot
X-UID: 4
Status: O
Content-Length: 2421
This is a multi-part message in MIME format
--956bff02-8aec-485e-a58c-40fda617ecbe
Content-Type: text/html; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
Hi
I visited DOVECOT.PROCONTROL.FI, =
and
noticed that you're not listed on some search engines! I think we can offer
=
you a service which can help you increase traffic and the number of visitors =
to your website.
I would like to introduce you to TrafficMagnet.com. We =
offer a unique technology
that will submit your website to over 300,000 search engines and directories =
every month.
You'll be surprised by the low cost, and by how effective this website =
promotion
method can be.
To find out more about TrafficMagnet and the cost for submitting your =
website
to over 300,000 search engines and directories, visit www.TrafficMagnet.com.
I would love to hear from you.
Best Regards,
Sarah Williams
Sales and Marketing
E-mail: sarah_williams@trafficmagnet.com
http://www.TrafficMagnet.com=
|
--956bff02-8aec-485e-a58c-40fda617ecbe--
From reply@seekercenter.net Tue Aug 6 13:01:17 2002
Received: with ECARTIS (v1.0.0; list dovecot); Tue, 06 Aug 2002 13:01:17 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from XXXXXX (unknown [211.101.236.162])
by danu.procontrol.fi (Postfix) with ESMTP id C6D0823832
for ; Tue, 6 Aug 2002 13:01:15 +0300 (EEST)
From: "Vanessa Lintner"
Subject: [dovecot] I have visited DOVECOT.PROCONTROL.FI and noticed that ...
To: dovecot@procontrol.fi
Content-Type: text/html;
Reply-To: "Vanessa Lintner"
Date: Tue, 6 Aug 2002 18:05:01 +0800
X-Priority: 3
X-Library: Business Promotion
Message-Id: <20020806100115.C6D0823832@danu.procontrol.fi>
X-archive-position: 5
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: reply@seekercenter.net
Precedence: bulk
X-list: dovecot
X-UID: 5
Status: O
Content-Length: 5137
|
Hello,
I have visited dovecot.procontrol.fi and noticed that your website is not listed on some search engines.
I am sure that through our service the number of people who visit your website will definitely increase. SeekerCenter
is a unique technology that instantly submits your website
to over 500,000 search engines and directories
-- a really low-cost and effective way to advertise your site.
For more details please go to SeekerCenter.net.
Give your website maximum exposure today!
Looking forward to hearing from you.
|
|
|
|
From cras@irccrew.org Tue Aug 6 13:34:34 2002
Received: with ECARTIS (v1.0.0; list dovecot); Tue, 06 Aug 2002 13:34:34 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from shodan.irccrew.org (shodan.irccrew.org [80.83.4.2])
by danu.procontrol.fi (Postfix) with ESMTP id 1EC3C23831
for ; Tue, 6 Aug 2002 13:34:34 +0300 (EEST)
Received: by shodan.irccrew.org (Postfix, from userid 6976)
id E37C74C0A0; Tue, 6 Aug 2002 13:34:33 +0300 (EEST)
Date: Tue, 6 Aug 2002 13:34:33 +0300
From: Timo Sirainen
To: dovecot@procontrol.fi
Subject: [dovecot] spam / updates
Message-ID: <20020806133433.T22431@irccrew.org>
Mime-Version: 1.0
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
Content-Type: text/plain; charset=us-ascii
X-archive-position: 6
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 6
Status: O
I guess I underestimated the spammers :) This list is subscribers-only now.
As for dovecot, I've been cleaning the code and probably will continue with
that some time. I was also thinking about moving the authentication code
into separate (SASL) library which could then be plugged into other
software, such as postfix.
From darix@linux.taugt.net Tue Aug 6 20:54:39 2002
Received: with ECARTIS (v1.0.0; list dovecot); Tue, 06 Aug 2002 20:54:39 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from linux.taugt.net (wh5035.stw.uni-rostock.de [139.30.105.35])
by danu.procontrol.fi (Postfix) with ESMTP id 99FC923829
for ; Tue, 6 Aug 2002 20:54:39 +0300 (EEST)
Received: by linux.taugt.net (none of your biz, from userid 500)
id 28A8D29F49; Tue, 6 Aug 2002 19:54:41 +0200 (CEST)
Date: Tue, 6 Aug 2002 19:54:41 +0200
From: Marcus Rueckert
To: dovecot mailing list
Subject: [dovecot] mbox support
Message-ID: <20020806175441.GA7148@linux.taugt.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
User-Agent: Mutt/1.4i
X-archive-position: 7
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: rueckert@informatik.uni-rostock.de
Precedence: bulk
X-list: dovecot
X-UID: 7
Status: O
hi
could you explain the following sentence from the readme.txt:
"mbox support is available but currently it relies a little bit on good
luck, ..."
what kind of luck do i need?
why do you think the mbox support isnt this reliable?
Marcus Rückert
--
irssi - the client of the smart and beautiful people
From marcelo@carpa.ciagri.usp.br Wed Aug 7 02:39:26 2002
Received: with ECARTIS (v1.0.0; list dovecot); Wed, 07 Aug 2002 02:39:26 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from carpa.ciagri.usp.br (carpa.ciagri.usp.br [143.107.209.25])
by danu.procontrol.fi (Postfix) with SMTP id 42D5323829
for ; Wed, 7 Aug 2002 02:39:25 +0300 (EEST)
Received: (qmail 32442 invoked by uid 1000); 6 Aug 2002 23:40:54 -0000
From: marcelo@carpa.ciagri.usp.br
Date: Tue, 6 Aug 2002 20:40:54 -0300
To: dovecot@procontrol.fi
Subject: [dovecot] starting
Message-ID: <20020806234054.GA30820@carpa.ciagri.usp.br>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.3.28i
X-archive-position: 8
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: marcelo@carpa.ciagri.usp.br
Precedence: bulk
X-list: dovecot
X-UID: 8
Status: O
Hi. Sorry for the basic question but... how do I
start the server? From inetd or in standalone mode
(which binary)?
Thanks
From cras@irccrew.org Wed Aug 7 06:54:12 2002
Received: with ECARTIS (v1.0.0; list dovecot); Wed, 07 Aug 2002 06:54:12 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from shodan.irccrew.org (shodan.irccrew.org [80.83.4.2])
by danu.procontrol.fi (Postfix) with ESMTP id EA9D623829
for ; Wed, 7 Aug 2002 06:54:11 +0300 (EEST)
Received: by shodan.irccrew.org (Postfix, from userid 6976)
id D125E4C0A0; Wed, 7 Aug 2002 06:54:11 +0300 (EEST)
Date: Wed, 7 Aug 2002 06:54:11 +0300
From: Timo Sirainen
To: dovecot mailing list
Subject: [dovecot] Re: mbox support
Message-ID: <20020807065411.A16470@irccrew.org>
References: <20020806175441.GA7148@linux.taugt.net>
Mime-Version: 1.0
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <20020806175441.GA7148@linux.taugt.net>; from rueckert@informatik.uni-rostock.de on Tue, Aug 06, 2002 at 07:54:41PM +0200
Content-Type: text/plain; charset=us-ascii
X-archive-position: 9
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 9
Status: O
On Tue, Aug 06, 2002 at 07:54:41PM +0200, Marcus Rueckert wrote:
> could you explain the following sentence from the readme.txt:
> "mbox support is available but currently it relies a little bit on good
> luck, ..."
>
> what kind of luck do i need?
> why do you think the mbox support isnt this reliable?
mbox files are locked only while they're being modified. If only dovecot was
accessing the mailbox I guess there wouldn't be any problem since it
properly locks the index files as well. But if any other MUA was just
modifying the mbox file (deleting messages) while dovecot was reading it,
it's possible that you end up having data from some random message. Storing
MD5 sum of all messages and making sure they match would fix this.
From cras@irccrew.org Wed Aug 7 06:58:27 2002
Received: with ECARTIS (v1.0.0; list dovecot); Wed, 07 Aug 2002 06:58:27 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from shodan.irccrew.org (shodan.irccrew.org [80.83.4.2])
by danu.procontrol.fi (Postfix) with ESMTP id B90D723829
for ; Wed, 7 Aug 2002 06:58:27 +0300 (EEST)
Received: by shodan.irccrew.org (Postfix, from userid 6976)
id 443C64C0A0; Wed, 7 Aug 2002 06:58:24 +0300 (EEST)
Date: Wed, 7 Aug 2002 06:58:24 +0300
From: Timo Sirainen
To: dovecot@procontrol.fi
Subject: [dovecot] Re: starting
Message-ID: <20020807065824.C16470@irccrew.org>
References: <20020806234054.GA30820@carpa.ciagri.usp.br>
Mime-Version: 1.0
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <20020806234054.GA30820@carpa.ciagri.usp.br>; from marcelo@carpa.ciagri.usp.br on Tue, Aug 06, 2002 at 08:40:54PM -0300
Content-Type: text/plain; charset=us-ascii
X-archive-position: 10
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 10
Status: O
On Tue, Aug 06, 2002 at 08:40:54PM -0300, marcelo@carpa.ciagri.usp.br wrote:
> Hi. Sorry for the basic question but... how do I
> start the server? From inetd or in standalone mode
> (which binary)?
Run imap-master
From cras@irccrew.org Wed Aug 7 06:59:01 2002
Received: with ECARTIS (v1.0.0; list dovecot); Wed, 07 Aug 2002 06:59:01 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from shodan.irccrew.org (shodan.irccrew.org [80.83.4.2])
by danu.procontrol.fi (Postfix) with ESMTP id 05B9423829
for ; Wed, 7 Aug 2002 06:59:01 +0300 (EEST)
Received: by shodan.irccrew.org (Postfix, from userid 6976)
id E84874C0A0; Wed, 7 Aug 2002 06:59:00 +0300 (EEST)
Date: Wed, 7 Aug 2002 06:59:00 +0300
From: Timo Sirainen
To: dovecot@procontrol.fi
Subject: [dovecot] Re: starting
Message-ID: <20020807065900.D16470@irccrew.org>
References: <20020806234054.GA30820@carpa.ciagri.usp.br>
Mime-Version: 1.0
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <20020806234054.GA30820@carpa.ciagri.usp.br>; from marcelo@carpa.ciagri.usp.br on Tue, Aug 06, 2002 at 08:40:54PM -0300
Content-Type: text/plain; charset=us-ascii
X-archive-position: 11
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 11
Status: O
On Tue, Aug 06, 2002 at 08:40:54PM -0300, marcelo@carpa.ciagri.usp.br wrote:
> Hi. Sorry for the basic question but... how do I
> start the server? From inetd or in standalone mode
> (which binary)?
Oh, and standalone :) Maybe I should support inetd too later.
From marcelo@carpa.ciagri.usp.br Thu Aug 8 02:18:19 2002
Received: with ECARTIS (v1.0.0; list dovecot); Thu, 08 Aug 2002 02:18:19 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from carpa.ciagri.usp.br (carpa.ciagri.usp.br [143.107.209.25])
by danu.procontrol.fi (Postfix) with SMTP id 348EE23829
for ; Thu, 8 Aug 2002 02:18:18 +0300 (EEST)
Received: (qmail 18910 invoked by uid 1000); 7 Aug 2002 23:19:56 -0000
From: marcelo@carpa.ciagri.usp.br
Date: Wed, 7 Aug 2002 20:19:56 -0300
To: dovecot@procontrol.fi
Subject: [dovecot] SELECT
Message-ID: <20020807231956.GA11240@carpa.ciagri.usp.br>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.3.28i
X-archive-position: 12
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: marcelo@carpa.ciagri.usp.br
Precedence: bulk
X-list: dovecot
X-UID: 12
Status: O
Content-Length: 1024
Hi, all.
Thanks Timo for answering my previous post. It's starting
fine now :)
I must be doing something wrong again but I can't connect from
any imap client. If I telnet to the server I get:
$ telnet carpa 11143
Trying 143.107.209.25...
Connected to carpa.ciagri.usp.br.
Escape character is '^]'.
* OK dovecot ready.
a001 login marcelo #####
a001 OK Logged in.
a002 select INBOX
a002 NO Broken INBOX: Permission denied
a003 select ""
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft \Recent)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft)] Flags
permitted.
* 246 EXISTS
* 246 RECENT
* OK [UNSEEN 1] First unseen.
* OK [UIDVALIDITY 1028760751] UIDs valid
a003 OK [READ-WRITE] Select completed.
a004 close
a004 OK Close completed.
a005 logout
* BYE Logging out
a005 OK Logout completed.
Connection closed by foreign host.
"INBOX" seems not be a valid mailbox name to Dovecot but
it works with others servers (courier, at least).
The server uses Maildirs and runs qmail and courier (by now).
Thanks.
From cras@irccrew.org Thu Aug 8 07:13:04 2002
Received: with ECARTIS (v1.0.0; list dovecot); Thu, 08 Aug 2002 07:13:04 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from shodan.irccrew.org (shodan.irccrew.org [80.83.4.2])
by danu.procontrol.fi (Postfix) with ESMTP id 65BF823831
for ; Thu, 8 Aug 2002 07:13:04 +0300 (EEST)
Received: by shodan.irccrew.org (Postfix, from userid 6976)
id 3BB134C0A0; Thu, 8 Aug 2002 07:13:03 +0300 (EEST)
Date: Thu, 8 Aug 2002 07:13:03 +0300
From: Timo Sirainen
To: dovecot@procontrol.fi
Subject: [dovecot] Re: SELECT
Message-ID: <20020808071303.F16470@irccrew.org>
References: <20020807231956.GA11240@carpa.ciagri.usp.br>
Mime-Version: 1.0
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <20020807231956.GA11240@carpa.ciagri.usp.br>; from marcelo@carpa.ciagri.usp.br on Wed, Aug 07, 2002 at 08:19:56PM -0300
Content-Type: text/plain; charset=us-ascii
X-archive-position: 13
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 13
Status: O
On Wed, Aug 07, 2002 at 08:19:56PM -0300, marcelo@carpa.ciagri.usp.br wrote:
> a002 NO Broken INBOX: Permission denied
You use normal passwd/shadow/pam authentication? Where does the home
directory point to, to your real home dir under which Maildir/ is? Anyway,
one of the following failed (under Maildir/):
mkdir .INBOX
ln -s ../cur .INBOX/cur
ln -s ../new .INBOX/new
ln -s ../tmp .INBOX/tmp
Next release will tell exactly which of them failed :)
> a003 select ""
I don't think this should be allowed :)
From marcelo@carpa.ciagri.usp.br Thu Aug 8 16:11:45 2002
Received: with ECARTIS (v1.0.0; list dovecot); Thu, 08 Aug 2002 16:11:45 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from carpa.ciagri.usp.br (carpa.ciagri.usp.br [143.107.209.25])
by danu.procontrol.fi (Postfix) with SMTP id 88CF423831
for ; Thu, 8 Aug 2002 16:11:44 +0300 (EEST)
Received: (qmail 9606 invoked by uid 1000); 8 Aug 2002 13:13:30 -0000
From: marcelo@carpa.ciagri.usp.br
Date: Thu, 8 Aug 2002 10:13:30 -0300
To: dovecot@procontrol.fi
Subject: [dovecot] Re: SELECT
Message-ID: <20020808131329.GA30775@carpa.ciagri.usp.br>
References: <20020807231956.GA11240@carpa.ciagri.usp.br> <20020808071303.F16470@irccrew.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20020808071303.F16470@irccrew.org>
User-Agent: Mutt/1.3.28i
X-archive-position: 14
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: marcelo@carpa.ciagri.usp.br
Precedence: bulk
X-list: dovecot
X-UID: 14
Status: O
On Thu, Aug 08, 2002 at 07:13:03AM +0300, Timo Sirainen wrote:
> On Wed, Aug 07, 2002 at 08:19:56PM -0300, marcelo@carpa.ciagri.usp.br wrote:
>
> > a002 NO Broken INBOX: Permission denied
>
> You use normal passwd/shadow/pam authentication?
Well, in dovecot.conf I have:
auth = default
auth_methods = plain
auth_userinfo = shadow
auth_user = root
> Where does the home
> directory point to, to your real home dir under which Maildir/ is?
Maildir is in the real users' home (/home/user/Maildir)
> Anyway,
> one of the following failed (under Maildir/):
>
> mkdir .INBOX
> ln -s ../cur .INBOX/cur
> ln -s ../new .INBOX/new
> ln -s ../tmp .INBOX/tmp
>
> Next release will tell exactly which of them failed :)
>
Shouldn't it be:
mkdir .INBOX
ln -s cur .INBOX/
ln -s new .INBOX/
ln -s tmp .INBOX/
?
Anyway, "mkdir .INBOX" succeeds but it is created with
perms 070 and trying to create the links inside it
causes the permission errors.
From cras@irccrew.org Thu Aug 8 18:04:54 2002
Received: with ECARTIS (v1.0.0; list dovecot); Thu, 08 Aug 2002 18:04:54 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from shodan.irccrew.org (shodan.irccrew.org [80.83.4.2])
by danu.procontrol.fi (Postfix) with ESMTP id C053623831
for ; Thu, 8 Aug 2002 18:04:54 +0300 (EEST)
Received: by shodan.irccrew.org (Postfix, from userid 6976)
id 6E2674C0A0; Thu, 8 Aug 2002 18:04:53 +0300 (EEST)
Date: Thu, 8 Aug 2002 18:04:53 +0300
From: Timo Sirainen
To: dovecot@procontrol.fi
Subject: [dovecot] Re: SELECT
Message-ID: <20020808180453.J16470@irccrew.org>
References: <20020807231956.GA11240@carpa.ciagri.usp.br> <20020808071303.F16470@irccrew.org> <20020808131329.GA30775@carpa.ciagri.usp.br>
Mime-Version: 1.0
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <20020808131329.GA30775@carpa.ciagri.usp.br>; from marcelo@carpa.ciagri.usp.br on Thu, Aug 08, 2002 at 10:13:30AM -0300
Content-Type: text/plain; charset=us-ascii
X-archive-position: 15
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 15
Status: O
On Thu, Aug 08, 2002 at 10:13:30AM -0300, marcelo@carpa.ciagri.usp.br wrote:
> Shouldn't it be:
>
> mkdir .INBOX
> ln -s cur .INBOX/
> ln -s new .INBOX/
> ln -s tmp .INBOX/
> ?
No, if that was done the symlinks would point to themselves. ln -s behaviour
looks a bit weird if you're not doing it from the destination directory :)
> Anyway, "mkdir .INBOX" succeeds but it is created with
> perms 070 and trying to create the links inside it
> causes the permission errors.
Ah, and this was because I was stupid and set the default umask to 0700
instead of 0077 :) You could fix this by uncommenting the umask-line in
dovecot.conf.
btw. 0.95 also has problems with bad network connections or large mailboxes,
I'll probably release 0.96 soon which fixes it.
From cras@irccrew.org Thu Aug 8 19:00:00 2002
Received: with ECARTIS (v1.0.0; list dovecot); Thu, 08 Aug 2002 19:00:00 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from shodan.irccrew.org (shodan.irccrew.org [80.83.4.2])
by danu.procontrol.fi (Postfix) with ESMTP id 5A8E723829
for ; Thu, 8 Aug 2002 19:00:00 +0300 (EEST)
Received: by shodan.irccrew.org (Postfix, from userid 6976)
id 36C274C0A0; Thu, 8 Aug 2002 19:00:00 +0300 (EEST)
Date: Thu, 8 Aug 2002 19:00:00 +0300
From: Timo Sirainen
To: dovecot@procontrol.fi
Subject: [dovecot] v0.96 released
Message-ID: <20020808190000.K16470@irccrew.org>
Mime-Version: 1.0
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
Content-Type: text/plain; charset=us-ascii
X-archive-position: 16
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 16
Status: O
Content-Length: 1072
This release was actually tested with reading my inbox (500+ messages, mbox,
Solaris 8) using Outlook and Evolution
v0.96 2002-08-08 Timo Sirainen
* Changed to LGPL v2.1 license
+ STARTTLS support and optional disabling of plaintext authentication
(LOGINDISABLED capability)
+ Support for custom message flags, each folder can have 26 different.
+ New configuration file options: imap_listen, max_logging_users,
max_imap_processes
+ You can specify config file location to imap-master with -c
+ All IMAP processes can now write to specified log file instead of
syslog. Either do this by setting IMAP_LOGFILE environment, or
give -l parameter to imap-master.
+ Some cleanups to remove warnings with BSDs
+ Changed all %s .. strerror(errno) -> %m
+ Rewritten memory pool code
- imap-master didn't close all the fds for executed processes
- iobuffer code was buggy and caused the connection to terminate
sometimes
- make install overwrote the existing dovecot.conf file, so it's now
named as dovecot-example.conf
From marcelo@carpa.ciagri.usp.br Thu Aug 8 22:33:47 2002
Received: with ECARTIS (v1.0.0; list dovecot); Thu, 08 Aug 2002 22:33:47 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from carpa.ciagri.usp.br (carpa.ciagri.usp.br [143.107.209.25])
by danu.procontrol.fi (Postfix) with SMTP id 18B9C23829
for ; Thu, 8 Aug 2002 22:33:45 +0300 (EEST)
Received: (qmail 23496 invoked by uid 1000); 8 Aug 2002 19:35:33 -0000
From: marcelo@carpa.ciagri.usp.br
Date: Thu, 8 Aug 2002 16:35:33 -0300
To: dovecot@procontrol.fi
Subject: [dovecot] Disk quotas (was: SELECT)
Message-ID: <20020808193533.GA28619@carpa.ciagri.usp.br>
References: <20020807231956.GA11240@carpa.ciagri.usp.br> <20020808071303.F16470@irccrew.org> <20020808131329.GA30775@carpa.ciagri.usp.br> <20020808180453.J16470@irccrew.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20020808180453.J16470@irccrew.org>
User-Agent: Mutt/1.3.28i
X-archive-position: 17
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: marcelo@carpa.ciagri.usp.br
Precedence: bulk
X-list: dovecot
X-UID: 17
Status: O
Content-Length: 1136
On Thu, Aug 08, 2002 at 06:04:53PM +0300, Timo Sirainen wrote:
> > Anyway, "mkdir .INBOX" succeeds but it is created with
> > perms 070 and trying to create the links inside it
> > causes the permission errors.
>
> Ah, and this was because I was stupid and set the default umask to 0700
> instead of 0077 :) You could fix this by uncommenting the umask-line in
> dovecot.conf.
Hey, it works! :)
>
> btw. 0.95 also has problems with bad network connections or large mailboxes,
> I'll probably release 0.96 soon which fixes it.
>
Yes, confirmed. Using 0.96 my inbox shows up with all my ~200 messages now.
Now, one last question (for a while): would be possible/desirable make
Dovecot work when an user is with her disk quota completely full (hard
quota)? The main reason I'm looking for an alternative imap server is that
neither WU-Imapd nor Courier work "properly" (from an user point of view) in
that situation (the user can not login or delete her messages to clean up
the mailbox). If Dovecot could handle this problem (creating its scratch
files in /tmp, perhaps) it'd be big plus, IMHO.
Thanks for the great support :)
From cras@irccrew.org Fri Aug 9 07:03:02 2002
Received: with ECARTIS (v1.0.0; list dovecot); Fri, 09 Aug 2002 07:03:02 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from shodan.irccrew.org (shodan.irccrew.org [80.83.4.2])
by danu.procontrol.fi (Postfix) with ESMTP id A2CE123829
for ; Fri, 9 Aug 2002 07:03:02 +0300 (EEST)
Received: by shodan.irccrew.org (Postfix, from userid 6976)
id 30A774C0A0; Fri, 9 Aug 2002 07:03:01 +0300 (EEST)
Date: Fri, 9 Aug 2002 07:03:01 +0300
From: Timo Sirainen
To: dovecot@procontrol.fi
Subject: [dovecot] Re: Disk quotas (was: SELECT)
Message-ID: <20020809070301.L16470@irccrew.org>
References: <20020807231956.GA11240@carpa.ciagri.usp.br> <20020808071303.F16470@irccrew.org> <20020808131329.GA30775@carpa.ciagri.usp.br> <20020808180453.J16470@irccrew.org> <20020808193533.GA28619@carpa.ciagri.usp.br>
Mime-Version: 1.0
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <20020808193533.GA28619@carpa.ciagri.usp.br>; from marcelo@carpa.ciagri.usp.br on Thu, Aug 08, 2002 at 04:35:33PM -0300
Content-Type: text/plain; charset=us-ascii
X-archive-position: 18
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 18
Status: O
Content-Length: 1031
On Thu, Aug 08, 2002 at 04:35:33PM -0300, marcelo@carpa.ciagri.usp.br wrote:
> Now, one last question (for a while): would be possible/desirable make
> Dovecot work when an user is with her disk quota completely full (hard
> quota)? The main reason I'm looking for an alternative imap server is that
> neither WU-Imapd nor Courier work "properly" (from an user point of view) in
> that situation (the user can not login or delete her messages to clean up
> the mailbox). If Dovecot could handle this problem (creating its scratch
> files in /tmp, perhaps) it'd be big plus, IMHO.
Well, that is a bit difficult to handle.. It's mostly the index files that
dovecot has to create/grow. I think I should support keeping them only in
memory when disk quota is exceeded.
But what about clients, I think some of them just copy the mail to trash
folder instead of deleting the mail? But maybe with
maildir_copy_with_hardlinks=yes that'd be possible even with quota full.
I guess I'll have to install quota support and start fixing.
From marcelo@carpa.ciagri.usp.br Fri Aug 9 22:05:26 2002
Received: with ECARTIS (v1.0.0; list dovecot); Fri, 09 Aug 2002 22:05:26 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from carpa.ciagri.usp.br (carpa.ciagri.usp.br [143.107.209.25])
by danu.procontrol.fi (Postfix) with SMTP id 41E5B23831
for ; Fri, 9 Aug 2002 22:03:46 +0300 (EEST)
Received: (qmail 27100 invoked by uid 1000); 9 Aug 2002 19:03:39 -0000
From: marcelo@carpa.ciagri.usp.br
Date: Fri, 9 Aug 2002 16:03:39 -0300
To: dovecot@procontrol.fi
Subject: [dovecot] Re: Disk quotas
Message-ID: <20020809190339.GA2063@carpa.ciagri.usp.br>
References: <20020807231956.GA11240@carpa.ciagri.usp.br> <20020808071303.F16470@irccrew.org> <20020808131329.GA30775@carpa.ciagri.usp.br> <20020808180453.J16470@irccrew.org> <20020808193533.GA28619@carpa.ciagri.usp.br> <20020809070301.L16470@irccrew.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20020809070301.L16470@irccrew.org>
User-Agent: Mutt/1.3.28i
X-archive-position: 19
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: marcelo@carpa.ciagri.usp.br
Precedence: bulk
X-list: dovecot
X-UID: 19
Status: O
On Fri, Aug 09, 2002 at 07:03:01AM +0300, Timo Sirainen wrote:
>
> Well, that is a bit difficult to handle.. It's mostly the index files that
> dovecot has to create/grow. I think I should support keeping them only in
> memory when disk quota is exceeded.
>
I see...
> But what about clients, I think some of them just copy the mail to trash
> folder instead of deleting the mail? But maybe with
> maildir_copy_with_hardlinks=yes that'd be possible even with quota full.
>
When a user or his imap client asks for a "write" operation (like saving in
Trash) I think it's fair for an imap server just fail the operation and
return an error (he's overquota :).
But using "good" clients that offer the option to just delete the messages
(not copy/move them) it would be nice if they could have the chance to at
least login and delete the spam.
> I guess I'll have to install quota support and start fixing.
>
I'm by no means a decent C programmer but if I could help somehow, just ask.
Thanks,
Marcelo.
From ev@kernel.ping-viini.org Mon Aug 12 01:09:37 2002
Received: with ECARTIS (v1.0.0; list dovecot); Mon, 12 Aug 2002 01:09:37 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from ping-viini.at.home.telekarelia.fi (ping-viini.at.home.telekarelia.fi [195.197.199.46])
by danu.procontrol.fi (Postfix) with SMTP id C8B3423833
for ; Mon, 12 Aug 2002 01:09:37 +0300 (EEST)
Received: (qmail 17429 invoked from network); 11 Aug 2002 21:32:33 -0000
Received: from unknown (HELO eero) (192.168.0.2)
by 0 with SMTP; 11 Aug 2002 21:32:33 -0000
Message-ID: <006701c24183$c7f10d60$0200a8c0@eero>
From: "Eero Volotinen"
To:
Subject: [dovecot] vpopmail authentication support
Date: Mon, 12 Aug 2002 01:09:37 +0300
MIME-Version: 1.0
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2600.0000
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
X-archive-position: 20
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: ev@kernel.ping-viini.org
Precedence: bulk
X-list: dovecot
X-UID: 20
Status: O
is nice feature to get soon if possible?
--
Eero
From cras@irccrew.org Thu Aug 22 16:25:52 2002
Received: with ECARTIS (v1.0.0; list dovecot); Thu, 22 Aug 2002 16:25:52 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from shodan.irccrew.org (shodan.irccrew.org [80.83.4.2])
by danu.procontrol.fi (Postfix) with ESMTP id 926902386D
for ; Thu, 22 Aug 2002 16:25:52 +0300 (EEST)
Received: by shodan.irccrew.org (Postfix, from userid 6976)
id C96234C0A0; Thu, 22 Aug 2002 16:25:51 +0300 (EEST)
Date: Thu, 22 Aug 2002 16:25:51 +0300
From: Timo Sirainen
To: dovecot@procontrol.fi
Subject: [dovecot] status update / cvs repository
Message-ID: <20020822132551.GD12341@irccrew.org>
Mime-Version: 1.0
Content-Disposition: inline
User-Agent: Mutt/1.4i
Content-Type: text/plain; charset=us-ascii
X-archive-position: 21
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 21
Status: O
Past few weeks went mostly while rewriting small part of code. It really
wouldn't have needed to take that long, but I always got really tired after
just looking at that code for a few seconds :)
Anyway, now messages are parsed in 256kB blocks (maybe I should make it
configurable), so large messages don't eat all memory anymore. Maildir
support seems to work with the new code, though I haven't tested it too well
yet. mbox is currently broken.
If you want to test it, dovecot's CVS is available with rsync. I don't think
I'll bother setting up pserver since that CVS has also other sources that
aren't public.
With rsync you'd do it something like:
# initialize new cvs repository (do it just once)
cvs -d ~/cvs init
# update cvs repository
rsync -avz --delete dovecot.procontrol.fi::dovecotcvs ~/cvs/dovecot
# checkout the sources from your cvs repository
cd ~/src
cvs -d ~/cvs checkout dovecot
From cras@irccrew.org Mon Aug 26 18:23:16 2002
Received: with ECARTIS (v1.0.0; list dovecot); Mon, 26 Aug 2002 18:23:16 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from shodan.irccrew.org (shodan.irccrew.org [80.83.4.2])
by danu.procontrol.fi (Postfix) with ESMTP id AB6572386D
for ; Mon, 26 Aug 2002 18:23:16 +0300 (EEST)
Received: by shodan.irccrew.org (Postfix, from userid 6976)
id ADE354C0A0; Mon, 26 Aug 2002 18:23:12 +0300 (EEST)
Date: Mon, 26 Aug 2002 18:23:12 +0300
From: Timo Sirainen
To: dovecot@procontrol.fi
Subject: [dovecot] Re: vpopmail authentication support
Message-ID: <20020826152312.GI7103@irccrew.org>
References: <006701c24183$c7f10d60$0200a8c0@eero>
Mime-Version: 1.0
Content-Disposition: inline
In-Reply-To: <006701c24183$c7f10d60$0200a8c0@eero>
User-Agent: Mutt/1.4i
Content-Type: text/plain; charset=us-ascii
X-archive-position: 22
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 22
Status: O
On Mon, Aug 12, 2002 at 01:09:37AM +0300, Eero Volotinen wrote:
> is nice feature to get soon if possible?
OK, I finally remembered to put this in TODO :) I also looked at vpopmail
before but it had at least one very weird thing which I thought about asking
about it's authors (gid field actually contains just flags).
From cras@irccrew.org Thu Aug 29 03:17:57 2002
Received: with ECARTIS (v1.0.0; list dovecot); Thu, 29 Aug 2002 03:17:57 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from shodan.irccrew.org (shodan.irccrew.org [80.83.4.2])
by danu.procontrol.fi (Postfix) with ESMTP id 56E0223831
for ; Thu, 29 Aug 2002 03:17:57 +0300 (EEST)
Received: by shodan.irccrew.org (Postfix, from userid 6976)
id A9A3B4C0A0; Thu, 29 Aug 2002 03:17:56 +0300 (EEST)
Date: Thu, 29 Aug 2002 03:17:56 +0300
From: Timo Sirainen
To: dovecot@procontrol.fi
Subject: [dovecot] v0.97 released
Message-ID: <20020829001756.GU7103@irccrew.org>
Mime-Version: 1.0
Content-Disposition: inline
User-Agent: Mutt/1.4i
Content-Type: text/plain; charset=us-ascii
X-archive-position: 23
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 23
Status: O
Content-Length: 2242
This is a major cleanup release. My near future goal is to make it
impossible to crash Dovecot no matter what you did. This release also adds
support for files larger than 2GB with Linux and Solaris. Also mbox support
seems to work quite well now, except EXPUNGE and STORE are still not
implemented.
The "impossible to crash" should already be quite true, at least when it
comes to index files. You can try to corrupt indexes in any way you want,
and Dovecot should not crash. There's one exception to this: modifying the
files while dovecot is reading them may crash it, this is too difficult to
fix as long as shared memory maps are used, and I don't see much point in it
anyway. I mostly want dovecot to survive corrupted files, not intentional
crashing :)
This release is the first I'd almost like to call beta-quality. But not
quite. It still lacks testing and several features, but it's very close :)
Changelog:
v0.97 2002-08-29 Timo Sirainen
+ Large mails are handled in 256kB blocks, so mail size no longer
has hardly any effect on memory usage
+ 64bit file offsets are used if supported by system. This means
Dovecot is fully capable of handling >2G mails in those systems.
With 32bit offsets >2G mails may not behave too well, but should
not crash either.
+ I fixed lots of potential integer overflows. This should make us
fully crash-free no matter what happens (index file corruption
mostly). I didn't verify everything too carefully yet, so more
auditing is still needed before we fully reach that goal.
+ Implemented several missing tasks / optimizations to index handling.
It should now stay fast after longer usage periods.
+ New configuration file options: log_path, log_timestamp, imaps_listen
+ "Critical errors" are now hidden from users, ie. any error message
that is not a direct reply to user error is written into log file
and user gets only "Internal error [timestamp]".
+ Nonblocking SSL handshaking
+ Lots of code cleanups
- Lots of mbox fixes, it seems to be somewhat reliable now
- Year in Date-field was parsed wrong
- Appending mail to mbox didn't work right
- Always verify that mailbox names are valid (especially they shouldn't
contain "../")
From svrmarty@gmx.net Sat Sep 14 23:50:41 2002
Received: with ECARTIS (v1.0.0; list dovecot); Sat, 14 Sep 2002 23:50:41 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from svrmarty.gnome.at (unknown [213.225.45.83])
by danu.procontrol.fi (Postfix) with ESMTP id 0964B23831
for ; Sat, 14 Sep 2002 23:50:40 +0300 (EEST)
Received: from silent (silent.svrmarty.gnome.at [192.168.1.8])
by svrmarty.gnome.at (8.12.3/8.12.3/Debian -4) with SMTP id g8EM00he027169
for ; Sun, 15 Sep 2002 00:00:01 +0200
Message-ID: <059101c25c30$da8c2b40$0801a8c0@svrmarty.gnome.at>
From: "SvR Marty"
To:
Subject: [dovecot] Problems with it
Date: Sat, 14 Sep 2002 22:54:00 +0200
MIME-Version: 1.0
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 5.50.4807.1700
X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-archive-position: 24
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: svrmarty@gmx.net
Precedence: bulk
X-list: dovecot
X-UID: 24
Status: O
i've install it, used the example config and started
tryed to access my mailbox via Outlook Express
here's the syslog:
Sep 14 22:42:17 p133 imap-login: Logged in. [192.168.1.8]
Sep 15 00:42:17 p133 imap(svrmarty): MAIL environment missing and
autodetection failed (home /home/svrmarty)
Sep 15 00:42:17 p133 imap-master: child 20960 (imap) returned error 98
Sep 14 22:42:19 p133 imap-login: Logged in. [192.168.1.8]
Sep 15 00:42:19 p133 imap(svrmarty): MAIL environment missing and
autodetection failed (home /home/svrmarty)
Sep 15 00:42:19 p133 imap-master: child 20961 (imap) returned error 98
it fails and i don't see any message
From cras@irccrew.org Sun Sep 15 00:50:10 2002
Received: with ECARTIS (v1.0.0; list dovecot); Sun, 15 Sep 2002 00:50:10 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from shodan.irccrew.org (shodan.irccrew.org [80.83.4.2])
by danu.procontrol.fi (Postfix) with ESMTP id 75F9423837
for ; Sun, 15 Sep 2002 00:50:10 +0300 (EEST)
Received: by shodan.irccrew.org (Postfix, from userid 6976)
id 4AA2C4C0A5; Sun, 15 Sep 2002 00:50:09 +0300 (EEST)
Date: Sun, 15 Sep 2002 00:50:08 +0300
From: Timo Sirainen
To: dovecot@procontrol.fi
Subject: [dovecot] Re: Problems with it
Message-ID: <20020914215008.GU9842@irccrew.org>
References: <059101c25c30$da8c2b40$0801a8c0@svrmarty.gnome.at>
Mime-Version: 1.0
Content-Disposition: inline
In-Reply-To: <059101c25c30$da8c2b40$0801a8c0@svrmarty.gnome.at>
User-Agent: Mutt/1.4i
Content-Type: text/plain; charset=us-ascii
X-archive-position: 25
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 25
Status: O
On Sat, Sep 14, 2002 at 10:54:00PM +0200, SvR Marty wrote:
> Sep 15 00:42:17 p133 imap(svrmarty): MAIL environment missing and
> autodetection failed (home /home/svrmarty)
Are you using mbox or maildir? It looks only for ~/Maildir with maildir.
With mbox it looks for ~/Mail and ~/mail directories, and they must contain
either "inbox" or "mbox" file.
If your mail is in /var/mail, that's not supported yet, and I'm not sure if
it will be. You could create symlink from it to ~/mail though.
From svrmarty@gmx.net Sun Sep 15 15:46:31 2002
Received: with ECARTIS (v1.0.0; list dovecot); Sun, 15 Sep 2002 15:46:31 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from svrmarty.gnome.at (unknown [213.225.45.83])
by danu.procontrol.fi (Postfix) with ESMTP id B6BA023831
for ; Sun, 15 Sep 2002 15:46:28 +0300 (EEST)
Received: from silent (silent.svrmarty.gnome.at [192.168.1.8])
by svrmarty.gnome.at (8.12.3/8.12.3/Debian -4) with SMTP id g8FDuEhe013158
for ; Sun, 15 Sep 2002 15:56:15 +0200
Message-ID: <009601c25cb6$b1e567c0$0801a8c0@svrmarty.gnome.at>
From: "SvR Marty"
To:
References: <059101c25c30$da8c2b40$0801a8c0@svrmarty.gnome.at> <20020914215008.GU9842@irccrew.org>
Subject: [dovecot] Re: Problems with it
Date: Sun, 15 Sep 2002 14:52:03 +0200
MIME-Version: 1.0
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 5.50.4807.1700
X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
X-archive-position: 26
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: svrmarty@gmx.net
Precedence: bulk
X-list: dovecot
X-UID: 26
Status: O
Content-Length: 1045
i can't find a good howto doc.
can you put ls -laR of the special folders into a mail an send it ?
i don't know how this should work
seems i'm using maildir
Internal error [2002-09-15 16:42:35]
Sep 15 16:42:35 p133 imap(svrmarty): Can't open index data
/home/svrmarty/Maildir/.INBOX/.imap.index.data: No such file or directory
dunno what to do
----- Original Message -----
From: "Timo Sirainen"
To:
Sent: Saturday, September 14, 2002 11:50 PM
Subject: [dovecot] Re: Problems with it
> On Sat, Sep 14, 2002 at 10:54:00PM +0200, SvR Marty wrote:
>
> > Sep 15 00:42:17 p133 imap(svrmarty): MAIL environment missing and
> > autodetection failed (home /home/svrmarty)
>
> Are you using mbox or maildir? It looks only for ~/Maildir with maildir.
> With mbox it looks for ~/Mail and ~/mail directories, and they must
contain
> either "inbox" or "mbox" file.
>
> If your mail is in /var/mail, that's not supported yet, and I'm not sure
if
> it will be. You could create symlink from it to ~/mail though.
>
>
From cras@irccrew.org Mon Sep 16 07:41:59 2002
Received: with ECARTIS (v1.0.0; list dovecot); Mon, 16 Sep 2002 07:41:59 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from shodan.irccrew.org (shodan.irccrew.org [80.83.4.2])
by danu.procontrol.fi (Postfix) with ESMTP id A05B423831
for ; Mon, 16 Sep 2002 07:41:59 +0300 (EEST)
Received: by shodan.irccrew.org (Postfix, from userid 6976)
id 4D5EF4C0A5; Mon, 16 Sep 2002 07:41:56 +0300 (EEST)
Date: Mon, 16 Sep 2002 07:41:56 +0300
From: Timo Sirainen
To: dovecot@procontrol.fi
Subject: [dovecot] Re: Problems with it
Message-ID: <20020916044155.GY9842@irccrew.org>
References: <059101c25c30$da8c2b40$0801a8c0@svrmarty.gnome.at> <20020914215008.GU9842@irccrew.org> <009601c25cb6$b1e567c0$0801a8c0@svrmarty.gnome.at>
Mime-Version: 1.0
Content-Disposition: inline
In-Reply-To: <009601c25cb6$b1e567c0$0801a8c0@svrmarty.gnome.at>
User-Agent: Mutt/1.4i
Content-Type: text/plain; charset=us-ascii
X-archive-position: 27
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 27
Status: O
On Sun, Sep 15, 2002 at 02:52:03PM +0200, SvR Marty wrote:
> seems i'm using maildir
> Internal error [2002-09-15 16:42:35]
> Sep 15 16:42:35 p133 imap(svrmarty): Can't open index data
> /home/svrmarty/Maildir/.INBOX/.imap.index.data: No such file or directory
Looks like some bug .. rm -f /home/svrmarty/Maildir/.INBOX/.imap.index*
might help.
Anyway, the 0.97 version isn't very usable. I think I'll release new version
today or tomorrow, which has had quite a lot of testing. I'll just have to
see if all those changes I did last weekend broke everything or fixed the
last few problems :)
From cras@irccrew.org Fri Sep 20 14:30:45 2002
Received: with ECARTIS (v1.0.0; list dovecot); Fri, 20 Sep 2002 14:30:45 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from shodan.irccrew.org (shodan.irccrew.org [80.83.4.2])
by danu.procontrol.fi (Postfix) with ESMTP id 8FDD82383F
for ; Fri, 20 Sep 2002 14:30:45 +0300 (EEST)
Received: by shodan.irccrew.org (Postfix, from userid 6976)
id 63BF44C0A5; Fri, 20 Sep 2002 14:30:45 +0300 (EEST)
Date: Fri, 20 Sep 2002 14:30:45 +0300
From: Timo Sirainen
To: dovecot@procontrol.fi
Subject: [dovecot] vpopmail authentication
Message-ID: <20020920113045.GC8225@irccrew.org>
Mime-Version: 1.0
Content-Disposition: inline
User-Agent: Mutt/1.4i
Content-Type: text/plain; charset=us-ascii
X-archive-position: 28
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 28
Status: O
CVS has now support for vpopmail authentication. .c file compiles, but I'm
not sure if I did the other checks properly and if it actually works :)
If someone here is using it, could you test it? At least to see if configure
detects it and if it compiles/links.
BTW. I think maildir support is now working quite well, 0.98 isn't far away.
From cras@irccrew.org Fri Sep 20 15:19:11 2002
Received: with ECARTIS (v1.0.0; list dovecot); Fri, 20 Sep 2002 15:19:11 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from shodan.irccrew.org (shodan.irccrew.org [80.83.4.2])
by danu.procontrol.fi (Postfix) with ESMTP id 202B223831
for ; Fri, 20 Sep 2002 15:19:11 +0300 (EEST)
Received: by shodan.irccrew.org (Postfix, from userid 6976)
id F03784C0A5; Fri, 20 Sep 2002 15:19:10 +0300 (EEST)
Date: Fri, 20 Sep 2002 15:19:10 +0300
From: Timo Sirainen
To: dovecot@procontrol.fi
Subject: [dovecot] Re: vpopmail authentication
Message-ID: <20020920121910.GA27866@irccrew.org>
References: <20020920113045.GC8225@irccrew.org>
Mime-Version: 1.0
Content-Disposition: inline
In-Reply-To: <20020920113045.GC8225@irccrew.org>
User-Agent: Mutt/1.4i
Content-Type: text/plain; charset=us-ascii
X-archive-position: 29
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 29
Status: O
On Fri, Sep 20, 2002 at 02:30:45PM +0300, Timo Sirainen wrote:
> CVS has now support for vpopmail authentication. .c file compiles, but I'm
> not sure if I did the other checks properly and if it actually works :)
Yes, it works.
From tss@iki.fi Mon Sep 23 20:42:43 2002
Received: with ECARTIS (v1.0.0; list dovecot); Mon, 23 Sep 2002 20:42:44 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from hurina (ip213-185-36-189.laajakaista.mtv3.fi [213.185.36.189])
by danu.procontrol.fi (Postfix) with ESMTP id D7B9D2383F
for ; Mon, 23 Sep 2002 20:42:43 +0300 (EEST)
Received: by hurina (Postfix, from userid 1000)
id 9D32C5E01F4F; Mon, 23 Sep 2002 20:42:43 +0300 (EEST)
Subject: [dovecot] 0.98 released
From: Timo Sirainen
To: dovecot@procontrol.fi
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
X-Mailer: Ximian Evolution 1.0.8
Date: 23 Sep 2002 20:42:43 +0300
Message-Id: <1032802963.15743.2.camel@hurina>
Mime-Version: 1.0
X-archive-position: 30
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 30
Status: O
Content-Length: 1061
0.98 released. This release includes LOTS of bugfixes after almost 4
weeks of actual use. We've mostly tested with Outlook 2000, Outlook
Express 6 and Evolution 1.0.8. I haven't heard of a single bug in
maildir for several days, and mbox worked quite well first time today :)
New features include vpopmail authentication, ability to run properly
when there's no disk space left to allow user to delete mail, and full
support for mbox.
v0.98 2002-09-23 Timo Sirainen
+ mbox support is finally working. There's still some reliability
fixes left but overall it should be quite usable.
+ vpopmail authentication support
+ We should be able to deal with "out of diskspace/quota" conditions
properly, by keeping the indexes in memory and allowing user to
delete mails to get more space.
+ Several speed enhancements
+ New configuration file option: overwrite_incompatible_index to force
using ".imap.index" file, overwriting it if it isn't compatible
- Handle invalid message headers reliably
- Tons of bugfixes and code cleanups everywhere
From tss@iki.fi Tue Sep 24 20:04:30 2002
Received: with ECARTIS (v1.0.0; list dovecot); Tue, 24 Sep 2002 20:04:30 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from hurina (ip213-185-36-189.laajakaista.mtv3.fi [213.185.36.189])
by danu.procontrol.fi (Postfix) with ESMTP id 6B45C23831
for ; Tue, 24 Sep 2002 20:04:30 +0300 (EEST)
Received: by hurina (Postfix, from userid 1000)
id 2AD7C5E01F42; Tue, 24 Sep 2002 20:04:30 +0300 (EEST)
Subject: [dovecot] 0.98.1 released
From: Timo Sirainen
To: dovecot@procontrol.fi
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
X-Mailer: Ximian Evolution 1.0.8
Date: 24 Sep 2002 20:04:30 +0300
Message-Id: <1032887070.27868.0.camel@hurina>
Mime-Version: 1.0
X-archive-position: 31
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 31
Status: O
Guess I should have waited one more day before releasing 0.98 :) This
fixes a few mbox problems and should finally make it safe to use. Also
fixes a bug of not allowing to save mail larger than 8kB.
From tss@iki.fi Mon Sep 30 23:53:10 2002
Received: with ECARTIS (v1.0.0; list dovecot); Mon, 30 Sep 2002 23:53:10 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from hurina (ip213-185-36-189.laajakaista.mtv3.fi [213.185.36.189])
by danu.procontrol.fi (Postfix) with ESMTP id 6FE262383F
for ; Mon, 30 Sep 2002 23:53:10 +0300 (EEST)
Received: by hurina (Postfix, from userid 1000)
id 69C545E03E5C; Mon, 30 Sep 2002 23:53:09 +0300 (EEST)
Subject: [dovecot] 0.98.2 released
From: Timo Sirainen
To: dovecot@procontrol.fi
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Organization:
Message-Id: <1033419189.14835.2.camel@hurina>
Mime-Version: 1.0
X-Mailer: Ximian Evolution 1.1.1.99 (Preview Release)
Date: 30 Sep 2002 23:53:09 +0300
X-archive-position: 32
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 32
Status: O
Content-Length: 1080
Lets see if I fixed all the nasty bugs now.
v0.98.2 2002-09-30 Timo Sirainen
+ --with-file-offset-size=32 can now be used to select 32bit file
offsets. Using them should be a bit faster and take a bit less
disk and memory (also needed to compile Dovecot successfully with
TinyCC).
+ maildir_copy_with_hardlinks option works now
+ Check new mail and notify about it to client also after
commands which don't allow full syncing (FETCH, STORE, SEARCH).
Also always send RECENT after EXISTS notify.
+ If we're out of disk space while opening mailbox, notify about it
with ALERT.
- STORE and SEARCH didn't handle properly message sequence numbers
when some in the middle were externally deleted
- SEARCH: Only first search condition was checked.
- mbox: Message flags given to APPEND were ignored.
- mbox: index was corrupted when changing flags for multipart MIME
messages
- Out of disk space-handling wasn't working properly with .customflags
file
- if auth processes were killed, login processes weren't reconnecting
to them
From tss@iki.fi Tue Oct 1 00:28:31 2002
Received: with ECARTIS (v1.0.0; list dovecot); Tue, 01 Oct 2002 00:28:31 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from hurina (ip213-185-36-189.laajakaista.mtv3.fi [213.185.36.189])
by danu.procontrol.fi (Postfix) with ESMTP id 3D1E123831
for ; Tue, 1 Oct 2002 00:28:31 +0300 (EEST)
Received: by hurina (Postfix, from userid 1000)
id D03AB5E03E5C; Tue, 1 Oct 2002 00:28:30 +0300 (EEST)
Subject: [dovecot] 0.98.3 released.
From: Timo Sirainen
To: dovecot@procontrol.fi
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Organization:
Message-Id: <1033421310.19032.0.camel@hurina>
Mime-Version: 1.0
X-Mailer: Ximian Evolution 1.1.1.99 (Preview Release)
Date: 01 Oct 2002 00:28:30 +0300
X-archive-position: 33
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 33
Status: O
v0.98.3 2002-10-01 Timo Sirainen
* Sorry, just noticed a very stupid bug which caused evolution 1.2
beta to crash. I always thought it was just evolution's fault :)
- Several fields in BODY / BODYSTRUCTURE replies weren't quoted
From tss@iki.fi Sun Oct 6 00:39:51 2002
Received: with ECARTIS (v1.0.0; list dovecot); Sun, 06 Oct 2002 00:39:51 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from hurina (ip213-185-36-189.laajakaista.mtv3.fi [213.185.36.189])
by danu.procontrol.fi (Postfix) with ESMTP id 14CF32382D
for ; Sun, 6 Oct 2002 00:39:51 +0300 (EEST)
Received: by hurina (Postfix, from userid 1000)
id BDD015E03E5C; Sun, 6 Oct 2002 00:39:50 +0300 (EEST)
Subject: [dovecot] 0.98.4 released
From: Timo Sirainen
To: dovecot@procontrol.fi
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Organization:
Message-Id: <1033853990.9860.17.camel@hurina>
Mime-Version: 1.0
X-Mailer: Ximian Evolution 1.1.1.99 (Preview Release)
Date: 06 Oct 2002 00:39:50 +0300
X-archive-position: 34
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 34
Status: O
v0.99 will be next, having the new great binary tree code which should
make huge mailboxes possible. With smaller mailboxes I think it still
makes it better than the old code, even while it is slower in some cases
(hash lookups are faster than btree search in optimal cases).
v0.98.4 2002-10-06 Timo Sirainen
* Just a final release before replacing hash file with a binary tree.
- When fetching messages larger than 256k, sometimes Dovecot missed
to send CR causing corrupted data at end of message and possibly
complete failure depending on IMAP client.
- Fetching BODY or BODYSTRUCTURE for message having content-type of
message/rfc822 didn't correctly add () around the envelope data.
- Several fixes to make it compile with HP/UX ANSI C compiler.
Also fixed several warnings it showed up.
From tss@iki.fi Tue Oct 8 00:43:16 2002
Received: with ECARTIS (v1.0.0; list dovecot); Tue, 08 Oct 2002 00:43:16 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from hurina (ip213-185-36-189.laajakaista.mtv3.fi [213.185.36.189])
by danu.procontrol.fi (Postfix) with ESMTP id 786252382B
for ; Tue, 8 Oct 2002 00:43:16 +0300 (EEST)
Received: by hurina (Postfix, from userid 1000)
id E5EDA5E015A0; Tue, 8 Oct 2002 00:43:15 +0300 (EEST)
Subject: [dovecot] Benchmarks
From: Timo Sirainen
To: dovecot@procontrol.fi
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Organization:
Message-Id: <1034026995.782.15.camel@hurina>
Mime-Version: 1.0
X-Mailer: Ximian Evolution 1.1.1.99 (Preview Release)
Date: 08 Oct 2002 00:43:15 +0300
X-archive-position: 35
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 35
Status: O
Just tried Dovecot with 85k mails from Linux kernel mailing list, total
of 357MB. I mostly wanted to see if the new binary tree file works well.
And it does :) Nothing gets slowed down after deleting messages all
around the mailbox.
I tried several things with Dovecot, UW-IMAPd and Courier. You'll see
that Dovecot is faster in everything else except raw I/O which is a bit
strange, have to look why.
I think the most important thing anyway is the time spent opening
mailbox. Clients are doing "STATUS mailbox (UNSEEN)" very often and that
has to be fast. With Dovecot it's so small that I didn't even bother
adding it to the benchmarks, with UW-IMAPd it's 16 seconds, with courier
2 seconds.
Full results can be found at
http://dovecot.procontrol.fi/dovecot-benchmark.txt
From tss@iki.fi Tue Oct 8 03:33:27 2002
Received: with ECARTIS (v1.0.0; list dovecot); Tue, 08 Oct 2002 03:33:27 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from hurina (ip213-185-36-189.laajakaista.mtv3.fi [213.185.36.189])
by danu.procontrol.fi (Postfix) with ESMTP id 1C9832382D
for ; Tue, 8 Oct 2002 03:33:27 +0300 (EEST)
Received: by hurina (Postfix, from userid 1000)
id 469EE5E015A0; Tue, 8 Oct 2002 03:33:26 +0300 (EEST)
Subject: [dovecot] Re: Benchmarks - updated for Cyrus
From: Timo Sirainen
To: dovecot@procontrol.fi
In-Reply-To: <1034026995.782.15.camel@hurina>
References: <1034026995.782.15.camel@hurina>
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Organization:
Message-Id: <1034037205.770.6.camel@hurina>
Mime-Version: 1.0
X-Mailer: Ximian Evolution 1.1.1.99 (Preview Release)
Date: 08 Oct 2002 03:33:26 +0300
X-archive-position: 36
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 36
Status: O
On Tue, 2002-10-08 at 00:43, Timo Sirainen wrote:
> Full results can be found at
> http://dovecot.procontrol.fi/dovecot-benchmark.txt
Updated to contain Cyrus benchmarks as well. Cyrus' BODY and ENVELOPE
fetches are equilevant to Dovecot's "BODY and ENVELOPE cached"
benchmarks.
So, we slightly beat Cyrus in many things, EXPUNGE most notably. BODY[]
fetching is slower probably because of our I/O problems, and possibly
also because we didn't store messages with CR+LFs (in which case we
would use sendfile(), and that should be _fast_).
We're also slow with message flag changes because maildir requires
rename()ing the mail files, we could later add optimization not to do it
but rely on flags specified in index file.
From tss@iki.fi Tue Oct 8 06:23:31 2002
Received: with ECARTIS (v1.0.0; list dovecot); Tue, 08 Oct 2002 06:23:31 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from hurina (ip213-185-36-189.laajakaista.mtv3.fi [213.185.36.189])
by danu.procontrol.fi (Postfix) with ESMTP id 6DFF12382B
for ; Tue, 8 Oct 2002 06:23:31 +0300 (EEST)
Received: by hurina (Postfix, from userid 1000)
id 466A25E015A0; Tue, 8 Oct 2002 06:23:31 +0300 (EEST)
Subject: [dovecot] Re: Benchmarks - updated for Cyrus
From: Timo Sirainen
To: dovecot@procontrol.fi
In-Reply-To: <1034037205.770.6.camel@hurina>
References: <1034026995.782.15.camel@hurina> <1034037205.770.6.camel@hurina>
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Organization:
Message-Id: <1034047411.780.5.camel@hurina>
Mime-Version: 1.0
X-Mailer: Ximian Evolution 1.1.1.99 (Preview Release)
Date: 08 Oct 2002 06:23:31 +0300
X-archive-position: 37
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 37
Status: O
On Tue, 2002-10-08 at 03:33, Timo Sirainen wrote:
> So, we slightly beat Cyrus in many things, EXPUNGE most notably. BODY[]
> fetching is slower probably because of our I/O problems, and possibly
> also because we didn't store messages with CR+LFs (in which case we
> would use sendfile(), and that should be _fast_).
Actually, after trying again a few more times (rebooting between),
Cyrus' BODY[] got much slower. Updated the benchmark once more, now
added Dovecot with CR+LF and Dovecot with 32bit file offsets (default is
64bit).
From tss@iki.fi Thu Oct 10 04:40:09 2002
Received: with ECARTIS (v1.0.0; list dovecot); Thu, 10 Oct 2002 04:40:09 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from hurina (ip213-185-36-189.laajakaista.mtv3.fi [213.185.36.189])
by danu.procontrol.fi (Postfix) with ESMTP id 745FB2382B
for ; Thu, 10 Oct 2002 04:40:09 +0300 (EEST)
Received: by hurina (Postfix, from userid 1000)
id F269E5E03E5C; Thu, 10 Oct 2002 04:40:08 +0300 (EEST)
Subject: [dovecot] Re: Benchmarks - updated for Cyrus
From: Timo Sirainen
To: dovecot@procontrol.fi
In-Reply-To: <1034047411.780.5.camel@hurina>
References: <1034026995.782.15.camel@hurina> <1034037205.770.6.camel@hurina>
<1034047411.780.5.camel@hurina>
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Organization:
Message-Id: <1034214008.770.7.camel@hurina>
Mime-Version: 1.0
X-Mailer: Ximian Evolution 1.1.1.99 (Preview Release)
Date: 10 Oct 2002 04:40:08 +0300
X-archive-position: 38
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 38
Status: O
On Tue, 2002-10-08 at 06:23, Timo Sirainen wrote:
> Actually, after trying again a few more times (rebooting between),
> Cyrus' BODY[] got much slower. Updated the benchmark once more, now
> added Dovecot with CR+LF and Dovecot with 32bit file offsets (default is
> 64bit).
After 3 hours of tweaking which didn't make much difference, I finally
tried if filesystem had any effect on the speed. It had. A lot. Using
the same mail files as Cyrus, I got the time going from 2:32 down to
1:41. Cyrus used 2:25 to do the same, meaning we easily beat Cyrus.
Now, uw-imapd is still faster with mbox handling..
From faulerhund@faulerhund.net Sat Oct 12 18:26:54 2002
Received: with ECARTIS (v1.0.0; list dovecot); Sat, 12 Oct 2002 18:26:55 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from moutvdom.kundenserver.de (moutvdom.kundenserver.de [195.20.224.200])
by danu.procontrol.fi (Postfix) with ESMTP id D6D972382D
for ; Sat, 12 Oct 2002 18:26:54 +0300 (EEST)
Received: from [195.20.224.206] (helo=mrvdomng.kundenserver.de)
by moutvdom.kundenserver.de with esmtp (Exim 3.35 #1)
id 180OAI-0001FX-00
for dovecot@procontrol.fi; Sat, 12 Oct 2002 17:26:54 +0200
Received: from [217.84.28.114] (helo=mobilo)
by mrvdomng.kundenserver.de with smtp (Exim 3.35 #1)
id 180OAH-0002wH-00
for dovecot@procontrol.fi; Sat, 12 Oct 2002 17:26:53 +0200
Message-ID: <000e01c27203$e947a450$1964a8c0@mobilo>
From: "Korbinian Riedhammer"
To:
Subject: [dovecot] auth problem
Date: Sat, 12 Oct 2002 17:27:43 +0200
MIME-Version: 1.0
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2600.0000
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
X-archive-position: 39
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: faulerhund@faulerhund.net
Precedence: bulk
X-list: dovecot
X-UID: 39
Status: O
hi, im facing this problem:
Fatal: can't listen in UNIX socket /var/rin/dovecot/login/
/usr/lib/dovecot/imap-login: No such file or directory
Errror: child xxxxx (auth) returned error 98
any idea?
- bini
From sean@tcob1.net Sat Oct 12 19:18:50 2002
Received: with ECARTIS (v1.0.0; list dovecot); Sat, 12 Oct 2002 19:18:50 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from s2.uklinux.net (smtp2.uklinux.net [80.84.64.30])
by danu.procontrol.fi (Postfix) with ESMTP id 17E9D2382B
for ; Sat, 12 Oct 2002 19:18:50 +0300 (EEST)
Received: from tcob1.net (y-airlock008.esatclear.ie [213.202.165.8])
(authenticated)
by s2.uklinux.net (8.11.6/8.11.6) with ESMTP id g9CGIm916110
for ; Sat, 12 Oct 2002 17:18:48 +0100
Envelope-To:
Received: from [192.168.0.1] (helo=tcob1.net.tcob1.net ident=sean)
by tcob1.net with asmtp (Exim 4.10)
id 180OyZ-00020q-00
for dovecot@procontrol.fi; Sat, 12 Oct 2002 17:18:51 +0100
Date: Sat, 12 Oct 2002 17:18:38 +0100
Message-ID:
From: Sean Rima
To: dovecot@procontrol.fi
Subject: [dovecot] replacing Courier imapd
User-Agent: Wanderlust/2.9.15 (Unchained Melody) SEMI/1.14.3 (Ushinoya)
FLIM/1.14.3 (=?ISO-8859-4?Q?Unebigory=F2mae?=) APEL/10.3 Emacs/21.2
(i586-pc-linux-gnu) MULE/5.0 (SAKAKI)
Organization: There Can Only Be 1
X-Home-Page: http://www.tcob1.net
X-GPG-Key-FingerPrint: AB0A 3748 9565 1BFD 4E77 D9D5 1CC9 D25A 7DA7 0294
X-GPG-Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x7DA70294
X-OS: Linux 2.4.19
MIME-Version: 1.0 (generated by SEMI 1.14.3 - "Ushinoya")
Content-Type: text/plain; charset=US-ASCII
X-archive-position: 40
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: sean@tcob1.net
Precedence: bulk
X-list: dovecot
X-UID: 40
Status: O
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi folks,
I currently use courier imapd, exim and ldap on my mail box. There are
no users on this box, all info is taken from the ldap server. I note
that dovecot does not use ldap as a backend, but curious if this in on
the cards, or even mysql backend.
Sean
- --
Sean Rima http://www.tcob1.net
Linux User: 231986 Jabber: tcobone@jabber.org
THE VIEWS EXPRESSED HERE ARE NOT NECESSARILY THOSE OF MY WIFE.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.0 (GNU/Linux)
Comment: Use GPG for Secure Mail
iD8DBQE9qEs4HMnSWn2nApQRAn22AJ4xdGWfpxAuY8dLDuHM/XUAXms0mwCfZJzk
ZD8cTK98j7f7CGcu1OENJXI=
=BS5M
-----END PGP SIGNATURE-----
From tss@iki.fi Sat Oct 12 20:42:41 2002
Received: with ECARTIS (v1.0.0; list dovecot); Sat, 12 Oct 2002 20:42:41 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from hurina (ip213-185-36-189.laajakaista.mtv3.fi [213.185.36.189])
by danu.procontrol.fi (Postfix) with ESMTP id B9BC12382D
for ; Sat, 12 Oct 2002 20:42:41 +0300 (EEST)
Received: by hurina (Postfix, from userid 1000)
id 7E8C95E03E5C; Sat, 12 Oct 2002 20:42:41 +0300 (EEST)
Subject: [dovecot] Re: auth problem
From: Timo Sirainen
To: dovecot@procontrol.fi
In-Reply-To: <000e01c27203$e947a450$1964a8c0@mobilo>
References: <000e01c27203$e947a450$1964a8c0@mobilo>
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Organization:
Message-Id: <1034444559.30856.52.camel@hurina>
Mime-Version: 1.0
X-Mailer: Ximian Evolution 1.1.1.99 (Preview Release)
Date: 12 Oct 2002 20:42:41 +0300
X-archive-position: 41
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 41
Status: O
On Sat, 2002-10-12 at 18:27, Korbinian Riedhammer wrote:
> hi, im facing this problem:
> Fatal: can't listen in UNIX socket /var/rin/dovecot/login/
> /usr/lib/dovecot/imap-login: No such file or directory
> Errror: child xxxxx (auth) returned error 98
Well, that error message at least looks a bit strange. Is that one or
two error messages? The first one should continue but it instead shows
two paths .. plus typo in /var/rin.
Anyway, probably something wrong in your config file, maybe you've tried
to add some path to "auth = xxx"? If not, send the whole file to me and
I'll see what's wrong.
Does /var/run/dovecot/login exist? And /usr/lib/dovecot/imap-login?
You did do make install, right?
From tss@iki.fi Sat Oct 12 20:46:54 2002
Received: with ECARTIS (v1.0.0; list dovecot); Sat, 12 Oct 2002 20:46:54 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from hurina (ip213-185-36-189.laajakaista.mtv3.fi [213.185.36.189])
by danu.procontrol.fi (Postfix) with ESMTP id 770472382D
for ; Sat, 12 Oct 2002 20:46:54 +0300 (EEST)
Received: by hurina (Postfix, from userid 1000)
id 42A2D5E03E5C; Sat, 12 Oct 2002 20:46:54 +0300 (EEST)
Subject: [dovecot] Re: replacing Courier imapd
From: Timo Sirainen
To: dovecot@procontrol.fi
In-Reply-To:
References:
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Organization:
Message-Id: <1034444814.30856.58.camel@hurina>
Mime-Version: 1.0
X-Mailer: Ximian Evolution 1.1.1.99 (Preview Release)
Date: 12 Oct 2002 20:46:54 +0300
X-archive-position: 42
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 42
Status: O
On Sat, 2002-10-12 at 19:18, Sean Rima wrote:
> I currently use courier imapd, exim and ldap on my mail box. There are
> no users on this box, all info is taken from the ldap server. I note
> that dovecot does not use ldap as a backend, but curious if this in on
> the cards, or even mysql backend.
It wouldn't be hard to implement them, but they're not high on my
TODO-list yet. If anyone else doesn't write them, I eventually will.
They would be possible through vpopmail though, since it supports them
and Dovecot supports vpopmail. I wouldn't really encourage using
vpopmail though, doesn't look very secure.
From tss@iki.fi Tue Oct 15 04:16:57 2002
Received: with ECARTIS (v1.0.0; list dovecot); Tue, 15 Oct 2002 04:16:57 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from hurina (ip213-185-36-189.laajakaista.mtv3.fi [213.185.36.189])
by danu.procontrol.fi (Postfix) with ESMTP id 13D9A2382D
for ; Tue, 15 Oct 2002 04:16:57 +0300 (EEST)
Received: by hurina (Postfix, from userid 1000)
id D9E845E015A0; Tue, 15 Oct 2002 04:16:56 +0300 (EEST)
Subject: [dovecot] More documentation
From: Timo Sirainen
To: dovecot@procontrol.fi
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Organization:
Message-Id: <1034644616.24679.4.camel@hurina>
Mime-Version: 1.0
X-Mailer: Ximian Evolution 1.1.1.99 (Preview Release)
Date: 15 Oct 2002 04:16:56 +0300
X-archive-position: 43
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 43
Status: O
I just wrote a bit of documentation. Probably not very well organized,
improvement suggestions welcome :)
http://dovecot.procontrol.fi/doc/INSTALL
http://dovecot.procontrol.fi/doc/configuration.txt
http://dovecot.procontrol.fi/doc/mail-storages.txt
Those are also in the main dovecot.procontrol.fi web page. They also
talk about a few features not yet in CVS..
From faulerhund@faulerhund.net Tue Oct 15 22:47:16 2002
Received: with ECARTIS (v1.0.0; list dovecot); Tue, 15 Oct 2002 22:47:16 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from moutvdom.kundenserver.de (moutvdom.kundenserver.de [195.20.224.131])
by danu.procontrol.fi (Postfix) with ESMTP id 76DA02382D
for ; Tue, 15 Oct 2002 22:47:16 +0300 (EEST)
Received: from [195.20.224.206] (helo=mrvdomng.kundenserver.de)
by moutvdom.kundenserver.de with esmtp (Exim 3.35 #1)
id 181Xes-0007dC-00
for dovecot@procontrol.fi; Tue, 15 Oct 2002 21:47:14 +0200
Received: from [217.235.92.239] (helo=Bini)
by mrvdomng.kundenserver.de with smtp (Exim 3.35 #1)
id 181Xer-0001oK-00
for dovecot@procontrol.fi; Tue, 15 Oct 2002 21:47:13 +0200
Message-ID: <000b01c27484$608af620$0164a8c0@Bini>
From: "Korbinian Riedhammer"
To:
Subject: [dovecot] still problems getting it to work
Date: Tue, 15 Oct 2002 21:52:21 +0200
MIME-Version: 1.0
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2600.0000
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
X-archive-position: 44
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: faulerhund@faulerhund.net
Precedence: bulk
X-list: dovecot
X-UID: 44
Status: O
Content-Length: 1645
hi all,
after havong some other trouble with my server i finally managed to
recompile and install dovecot. installed it in /usr/local (the dafault).
below u see a dump of my config file. i want to use "normal" shadow
passwords for authentification. i adjustet the pathes from /usr/ to
/usr/local/. when i login with outlook express 4 example i get the error:
unsupported authentification method. (my system isn't configured for shadow
md5). in the error log i get the the /dev/urandom is needed but not found,
well but i have it :)
maybe a chroot setting?
thx bini
imap_port = 143
#imaps_port = 993
imap_listen =
#imaps_listen =
#ssl_cert_file = /etc/ssl/certs/imapd.pem
#ssl_key_file = /etc/ssl/private/imapd.pem
disable_plaintext_auth = no
log_path = /var/log/imapd.log
log_timestamp = %d %H:%M:%S
login_executable = /usr/local/lib/dovecot/imap-login
login_user = imapd
login_dir = /var/run/dovecot/login
login_chroot = yes
login_processes_count = 1
max_logging_users = 256
imap_executable = /usr/local/lib/dovecot/imap
max_imap_processes = 1024
first_valid_uid = 1000
#last_valid_uid = 0
first_valid_gid = 101
last_valid_gid = 101
valid_chroot_dirs = /var/run/dovecot
maildir_copy_with_hardlinks = no
maildir_check_content_changes = no
overwrite_incompatible_index = yes
umask = 0077
auth = default
auth_methods = plain
#auth_realms =
auth_userinfo = shadow
auth_executable = /usr/local/lib/dovecot/imap-auth
auth_user = root
auth_chroot = /var/run/dovecot
auth_count = 1
#auth = digest_md5
#auth_methods = digest-md5
#auth_realms =
#auth_userinfo = passwd-file /etc/passwd.imap
#auth_user = imapauth
#auth_chroot = /var/run/dovecot/auth
From tss@iki.fi Tue Oct 15 23:38:37 2002
Received: with ECARTIS (v1.0.0; list dovecot); Tue, 15 Oct 2002 23:38:37 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from hurina (ip213-185-36-189.laajakaista.mtv3.fi [213.185.36.189])
by danu.procontrol.fi (Postfix) with ESMTP id D3448238A3
for ; Tue, 15 Oct 2002 23:38:37 +0300 (EEST)
Received: by hurina (Postfix, from userid 1000)
id 4EC045E107F9; Tue, 15 Oct 2002 23:38:34 +0300 (EEST)
Subject: [dovecot] Re: still problems getting it to work
From: Timo Sirainen
To: dovecot@procontrol.fi
In-Reply-To: <000b01c27484$608af620$0164a8c0@Bini>
References: <000b01c27484$608af620$0164a8c0@Bini>
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Organization:
Message-Id: <1034714314.24679.17.camel@hurina>
Mime-Version: 1.0
X-Mailer: Ximian Evolution 1.1.1.99 (Preview Release)
Date: 15 Oct 2002 23:38:34 +0300
X-archive-position: 45
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 45
Status: O
On Tue, 2002-10-15 at 22:52, Korbinian Riedhammer wrote:
> hi all,
> after havong some other trouble with my server i finally managed to
> recompile and install dovecot. installed it in /usr/local (the dafault).
> below u see a dump of my config file. i want to use "normal" shadow
> passwords for authentification. i adjustet the pathes from /usr/ to
> /usr/local/. when i login with outlook express 4 example i get the error:
> unsupported authentification method. (my system isn't configured for shadow
> md5). in the error log i get the the /dev/urandom is needed but not found,
> well but i have it :)
> maybe a chroot setting?
You're right, it's the auth_chroot which breaks it. I fixed it in CVS
now. But your auth process is still running as root, which makes
chrooting pretty useless, so just disable it.
From tss@iki.fi Tue Oct 15 23:57:35 2002
Received: with ECARTIS (v1.0.0; list dovecot); Tue, 15 Oct 2002 23:57:35 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from hurina (ip213-185-36-189.laajakaista.mtv3.fi [213.185.36.189])
by danu.procontrol.fi (Postfix) with ESMTP id BD343238A3
for ; Tue, 15 Oct 2002 23:57:35 +0300 (EEST)
Received: by hurina (Postfix, from userid 1000)
id 801965E107F9; Tue, 15 Oct 2002 23:57:35 +0300 (EEST)
Subject: [dovecot] Re: still problems getting it to work
From: Timo Sirainen
To: dovecot@procontrol.fi
In-Reply-To: <000b01c27484$608af620$0164a8c0@Bini>
References: <000b01c27484$608af620$0164a8c0@Bini>
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Organization:
Message-Id: <1034715455.24679.22.camel@hurina>
Mime-Version: 1.0
X-Mailer: Ximian Evolution 1.1.1.99 (Preview Release)
Date: 15 Oct 2002 23:57:35 +0300
X-archive-position: 46
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 46
Status: O
On Tue, 2002-10-15 at 22:52, Korbinian Riedhammer wrote:
> hi all,
> after havong some other trouble with my server i finally managed to
> recompile and install dovecot. installed it in /usr/local (the dafault).
> below u see a dump of my config file. i want to use "normal" shadow
> passwords for authentification. i adjustet the pathes from /usr/ to
> /usr/local/.
Path changing isn't needed btw. Dovecot uses automatically the paths
given with --prefix if they're not set in config file.
> when i login with outlook express 4 example i get the error:
> unsupported authentification method. (my system isn't configured for shadow
> md5). in the error log i get the the /dev/urandom is needed but not found,
> well but i have it :)
> maybe a chroot setting?
Also chrooting wouldn't work anyway with shadow authentication since it
can't open /etc/shadow. I also just tried to see if opening shadow
database before chrooting would work, but it doesn't, probably because
it wants to reopen the file if it has changed.
From thomas@xs4all.net Sat Oct 19 15:39:57 2002
Received: with ECARTIS (v1.0.0; list dovecot); Sat, 19 Oct 2002 15:39:57 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from smtpzilla3.xs4all.nl (smtpzilla3.xs4all.nl [194.109.127.139])
by danu.procontrol.fi (Postfix) with ESMTP id 163C1238A7
for ; Sat, 19 Oct 2002 15:39:57 +0300 (EEST)
Received: from centurion.xs4all.nl (centurion.xs4all.nl [194.109.0.100])
by smtpzilla3.xs4all.nl (8.12.0/8.12.0) with ESMTP id g9JCdus7052057
for ; Sat, 19 Oct 2002 14:39:56 +0200 (CEST)
Received: by centurion.xs4all.nl (Postfix, from userid 1000)
id D6EDC2268; Sat, 19 Oct 2002 14:38:46 +0200 (CEST)
Date: Sat, 19 Oct 2002 14:38:46 +0200
From: Thomas Wouters
To: dovecot@procontrol.fi
Subject: [dovecot] Architectural questions
Message-ID: <20021019123846.GK543@xs4all.nl>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.4i
X-message-flag: Danger Will Robinson!
X-archive-position: 47
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: thomas@xs4all.net
Precedence: bulk
X-list: dovecot
X-UID: 47
Status: O
Content-Length: 8111
Greetings,
I have some architectural questions regarding dovecot, and though I've half
answered them by looking at the source, I'm also interested in hearing
whether my (our) wishes and suggestions are already being considered (or can
be considered, once built) for inclusion in dovecot itself.
Let me first explain why I'm doing this. I work for XS4ALL, a fairly large
ISP in the Netherlands. We provide a wide variety of services, including
shell access, pop3, webmail, et cetera. We use Sendmail on several clusters
of FreeBSD machines (loadbalanced using layer-4 ethernet switches) and
several NetApp Filers (dedicated NFS servers with fail-safe disk-arrays and
such) for backend. Several years ago (when we were a lot smaller) we noticed
the typical use of the mailboxes included leaving much old email on the
server, at least for a while, and that this is a bothersome thing when using
mbox mailboxes. (The mboxes basically have to be copied over whenever the
status of an email changes, leading to a lot of I/O.)
We briefly played with modifying sendmail and the pop server to avoid the
full copy in the common case (only status changes) by doing in-place edits
of a pre-generated Status line, as well as avoid full scanning of the mbox
file by creating special headers to mark the 'real' length of an email. It
worked, for a while, but it wasn't going to scale very well. So we switched
to maildir mailboxes for the mail spool. A modified mail.local (which we
need for other reasons as well) delivers in /var/spool/mail/u/s/username,
and mutt, uqwk, a modified pine and qmail's pop3 daemon read it from there.
Until last week our clients could choose to have mbox inboxes, to use with
'elm' or 'mail', but we decided to discontinue that support. Our new shell
servers, which are in test, don't have elm installed anymore anyway.
We still have support for mbox mailboxes in a user's homedirectory though,
by using procmail and such. So when we needed an IMAP server for use with
our webmail (based on SquirrelMail), we were forced to go with the UW-IMAP
server, with the maildir patch that's been scattered around the 'net. This
worked, for a while; we also use the maildir patch with pine after all.
However, the maildir patch is not very good. Not at all, even, and it only
seems to work by pure chance. Pine works for the average user who does not
get a lot of new mail while his pine is open or does not use procmail, and
fortunately a lot of the people that do get a lot of email use mutt, which
does work properly. The UW-IMAP server worked fine because SquirrelMail only
uses (used) a small subset of the available functionality. But that's
changing, as SquirrelMail gets actively developed, and we're also
considering other IMAP-based services. But we can't switch to Courier or
Cyrus because we need mbox support. And while looking for mbox patches for
either of those two, I ran across dovecot. Yay! :)
Dovecot is not everything we'd want, but it comes very close, and contrary
to UW-IMAP both the design and the actual source code are clean, readable
and logical, which means we can add the features we need and support them.
What we need and want to add is fairly simple, but I've only been looking at
dovecot since yesterday so I'd be happy to hear if any of it is possible,
feasible, unwise or unacceptable.
- First off, we need the maildir support to be 'correct' in that it does
not rely on the naming of the files in the mailbox, other than the very
loose specification DJB gives (doesn't contain a colon or slash and
doesn't start with a dot.) The pine/UW-imap patch breaks here because it
depends on the first part of the filename being time() or something else
that, when sorted alphanumerically, puts new mail at the end. Our
LDA does this, but procmail does not, and it shouldn't have to.
- Second, we need the maildir support to be 'correct' in that it does not
rely on the directory order being persistant. The NetApp Filers use
btree-indexed directories, so the order of readdir() can change
completely whenever a file is added or removed. The pine/uw-imap patch
relies on the '.uidvalidity' file being modified whenever the maildir sort
order changed, and this isn't happening.
I *think*, from reading the sources, both of those are correct already. If
they aren't, I'd strongly urge you to fix it, as #1 is a problem for anyone
using procmail and #2 is a problem for anyone with 'indexed' directories
(including such new filesystems as reiserfs, and I assume FreeBSD's new
hashed directories.)
- We need to avoid using fcntl(). The Netapps support it, but file-locking
over NFS is very, very poorly designed and we've had too much problems of
various kinds before, with fcntl. We also don't like the idea of having
thousands of fcntl locks at the same time ;P Instead, we've switched to
the locking method described in the Linux open(2) manpage under O_EXCL.
(We call it 'dot-locking', I'm not sure where the name came from.)
The actual implementation of that method is pretty simple, and I have a C
version and a Python version hanging around here somewhere (the Python
version is being used by GNU Mailman, last I looked.) If we're going to
use dovecot, we will replace most, if not all, fcntl()s with dot-locking,
the question is whether you want it contributed to dovecot :)
- Every user's incoming mailbox is /var/spool/u/s/username. Other mailboxes
are in /home/u/username/mail or /home/u/username/Mail (the second if the
first does not exist.) We are not yet certain whether we want the inbox
to be able to have subdir-mailboxes, as /var/spool and /home have
different quotas and we urge people not to store their mail on
/var/spool. (for one thing, it doesn't get backed-up.) We want these
things to work without magical symlinks or empty files, because people
_will_ delete them and cause unnecessary helpdesk calls :) Again, the
question is mostly whether this is desirable in dovecot (or something
enough like it to reduce local changes.)
- We have over 300k mailboxes at the moment. We expect that number to keep
growing. The indexer process (as described by design.txt) does not sound
as a good idea in our case :) How necessary is it, really ? Especially
since we do not expect more than 10% of those mailboxes to be actually
used by IMAP, not even once. If disabling the indexer completely just
means longer startup times for IMAP sessions, we can live with that.
- The UW-IMAP maildir patch stores UID's in the indiviual filenames, using
a 'U' flag. Will this interfere with dovecot ? We don't really need
dovecot and UW-IMAP to share UIDs, but we would like to have an as
painless transition as possible, without having to rename millions of
files to remove the U flag and other flags :P It would also be nice to
keep pine using the existing maildir patch, even though very few
IMAP-users would use pine.
- Would dovecot scale, architecturally speaking, to 500k+ active mailboxes ?
The amount of hardware is not really an issue, we can add a lot of
machines (off-the-shelve intel hadware) to each cluster, but if each
dovecot process has to load in an index of all possible mailboxes... that
would be a problem. Doing an inordinate number of file-accesses over NFS
would also be a problem, but I haven't seen any indication of that in the
source, yet.
In case it wasn't clear yet, I'm very happy to have found dovecot. The lack
of a decent mbox IMAP server has always dismayed me, let alone an
mbox+maildir one :) I should also point out that even though XS4ALL is a
commercial company, we would contribute our changes even if the licence
didn't require it, and we want to contribute them back the way you want
them, not necessarily the way it's easiest for us. We have a lot of
experience with opensource software, as a simple google on my name should
indicate ;P
Regards,
--
Thomas Wouters
Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
From tss@iki.fi Sat Oct 19 17:01:56 2002
Received: with ECARTIS (v1.0.0; list dovecot); Sat, 19 Oct 2002 17:01:56 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from hurina (ip213-185-36-189.laajakaista.mtv3.fi [213.185.36.189])
by danu.procontrol.fi (Postfix) with ESMTP id B8F982382D
for ; Sat, 19 Oct 2002 17:01:55 +0300 (EEST)
Received: by hurina (Postfix, from userid 1000)
id 822415E03E5C; Sat, 19 Oct 2002 17:01:55 +0300 (EEST)
Subject: [dovecot] Re: Architectural questions
From: Timo Sirainen
To: dovecot@procontrol.fi
In-Reply-To: <20021019123846.GK543@xs4all.nl>
References: <20021019123846.GK543@xs4all.nl>
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Organization:
Message-Id: <1035036115.1674.81.camel@hurina>
Mime-Version: 1.0
X-Mailer: Ximian Evolution 1.1.1.99 (Preview Release)
Date: 19 Oct 2002 17:01:55 +0300
X-archive-position: 48
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 48
Status: O
Content-Length: 8886
On Sat, 2002-10-19 at 15:38, Thomas Wouters wrote:
> We briefly played with modifying sendmail and the pop server to avoid the
> full copy in the common case (only status changes) by doing in-place edits
> of a pre-generated Status line,
UW-imapd does this as well, creating "X-Keywords: " line for
each mail. I had thought about this first with dovecot too, but since
mutt rewrote the whole mailbox always I figured I might as well. But
with larger mailboxes this is really slow, so I think I'll support the
X-keywords trick myself too.
> as well as avoid full scanning of the mbox
> file by creating special headers to mark the 'real' length of an email.
For each mail? Content-Length? With my tests that didn't seem to help
much, rather made it just slower.. Could be that I just did something
badly, have to look into it more when I begin optimizing mbox handling
more. Have to get it at least as fast as UW-imapd :)
> We still have support for mbox mailboxes in a user's homedirectory though,
> by using procmail and such. So when we needed an IMAP server for use with
> our webmail (based on SquirrelMail), we were forced to go with the UW-IMAP
> server, with the maildir patch that's been scattered around the 'net. This
Hm. Squirrelmail requires SORT extension which Dovecot doesn't support
yet. Notes about SORT from CVS's TODO:
- sort (draft-ietf-imapext-sort)
- basically sorted SEARCH, requiring CHARSET support for
UTF-8 and ASCII
- we could create alternative binary tree file(s) for different sort
conditions, ".tree-sort" or something. or if we decide to just
keep it in memory, btree could still be best choice.
- required by squirrelmail (webmail)
> - First off, we need the maildir support to be 'correct' in that it does
> not rely on the naming of the files in the mailbox, other than the very
> loose specification DJB gives (doesn't contain a colon or slash and
> doesn't start with a dot.) The pine/UW-imap patch breaks here because it
> depends on the first part of the filename being time() or something else
> that, when sorted alphanumerically, puts new mail at the end. Our
> LDA does this, but procmail does not, and it shouldn't have to.
Dovecot doesn't care as long as the file name stays same before the ':'
character.
> - Second, we need the maildir support to be 'correct' in that it does not
> rely on the directory order being persistant. The NetApp Filers use
> btree-indexed directories, so the order of readdir() can change
> completely whenever a file is added or removed. The pine/uw-imap patch
> relies on the '.uidvalidity' file being modified whenever the maildir sort
> order changed, and this isn't happening.
Dovecot reads them into hash so it doesn't depend on readdir()
behaviour.
> - We need to avoid using fcntl(). The Netapps support it, but file-locking
> over NFS is very, very poorly designed and we've had too much problems of
> various kinds before, with fcntl. We also don't like the idea of having
> thousands of fcntl locks at the same time ;P Instead, we've switched to
> the locking method described in the Linux open(2) manpage under O_EXCL.
> (We call it 'dot-locking', I'm not sure where the name came from.)
Hmm. The dot-lock means the "mbox.lock" file which gets created when
someone wants it exclusively locked. Dovecot supports it, and maildir
itself doesn't need locking at all. Dovecot's index files currently use
fcntl()-locking, but it would be possible to replace them with lock
files.
Then there's modify log file. Dovecot uses fcntl() locking for it as a
way to figure out if it's the only one using the log file. Like make
everyone read-lock the file, then if someone wants to know if it's the
only one using it it tries to set write-lock on, if it fails it knows
someone else it using it as well. I'm not sure if there's any good way
to replace that by using files, I had pretty complicated (desperate)
plans before figuring out fcntl() could be used to do it easily.
It would be possible to just assume that there's always someone else
using the modify log, but each flag change or expunge would always write
a few bytes to it then, and when log file is switched (there's .log and
.log.2) it wouldn't be truncated after last process is finished with it
which is not too bad since after the next switch it will be truncated.
Also it would be possible not to use index files at all but just keep
them in memory. I've been fixing code to make this possible and somewhat
fast.
> If we're going to
> use dovecot, we will replace most, if not all, fcntl()s with dot-locking,
> the question is whether you want it contributed to dovecot :)
All locking goes through file_*_lock() or mbox_lock_*() functions. mbox
locking supports it already, and file_*_lock() could be made to support
it. It doesn't get currently file name but that could be done.
> - Every user's incoming mailbox is /var/spool/u/s/username. Other mailboxes
> are in /home/u/username/mail or /home/u/username/Mail (the second if the
> first does not exist.) We are not yet certain whether we want the inbox
> to be able to have subdir-mailboxes, as /var/spool and /home have
> different quotas and we urge people not to store their mail on
> /var/spool. (for one thing, it doesn't get backed-up.) We want these
> things to work without magical symlinks or empty files, because people
> _will_ delete them and cause unnecessary helpdesk calls :) Again, the
> question is mostly whether this is desirable in dovecot (or something
> enough like it to reduce local changes.)
Are maildir inboxes also in /var/spool? With mbox sub-inboxes wouldn't
be even possible because dir structure == mailbox structure, and since
inbox file exists there can't be inbox-dir (except maybe with different
case but that's kludgy).
I've also thought I might as well make it possible to read the mbox
inbox from /var/mail or whereever it is. Pretty easy to do, but .lock
file is problematic if new files can't be added to the /var/mail
directory.
> - We have over 300k mailboxes at the moment. We expect that number to keep
> growing. The indexer process (as described by design.txt) does not sound
> as a good idea in our case :) How necessary is it, really ? Especially
> since we do not expect more than 10% of those mailboxes to be actually
> used by IMAP, not even once. If disabling the indexer completely just
> means longer startup times for IMAP sessions, we can live with that.
Indexer doesn't exist yet, and wouldn't be really needed even. I still
think it could be somewhat nice idea, the system load is probably less
during night so we could use the extra time to make mailboxes perform
faster next day.
It'd be difficult to know when exactly there is "extra time" which is
why I haven't yet done the indexer. Probably needs some external program
(script) which tells it by maybe looking at some I/O statistics from
/proc or doing a few file operations and checking the latency.
Am I right in that CPU usage still isn't any problem but rather the I/O?
> - The UW-IMAP maildir patch stores UID's in the indiviual filenames, using
> a 'U' flag. Will this interfere with dovecot ? We don't really need
> dovecot and UW-IMAP to share UIDs, but we would like to have an as
> painless transition as possible, without having to rename millions of
> files to remove the U flag and other flags :P It would also be nice to
> keep pine using the existing maildir patch, even though very few
> IMAP-users would use pine.
How exactly does the U flag work? I hope it's before the ':' character
like Courier's S=filesize? Otherwise U=1234 would be thought of as 6
different flags which isn't very good since Dovecot reorders them as
1234=U.
> - Would dovecot scale, architecturally speaking, to 500k+ active mailboxes ?
> The amount of hardware is not really an issue, we can add a lot of
> machines (off-the-shelve intel hadware) to each cluster, but if each
> dovecot process has to load in an index of all possible mailboxes... that
> would be a problem. Doing an inordinate number of file-accesses over NFS
> would also be a problem, but I haven't seen any indication of that in the
> source, yet.
Dovecot opens the index when opening mailbox. It doesn't open other
mailboxes indexes. Also the indexes should make the file accesses less
than otherwise, especially with mbox since it wouldn't need to read and
parse the whole mbox file. In general I've tried to keep the file I/O as
little as possible.
If your clusters access the files through NFS, there should be no
problem. Except I've never tried Dovecot through NFS, and I'm not sure
how well mmap()ing works through NFS. I know there's been problems
before but hopefully they've been fixed already.
From thomas@xs4all.net Sat Oct 19 18:11:38 2002
Received: with ECARTIS (v1.0.0; list dovecot); Sat, 19 Oct 2002 18:11:38 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from smtpzilla3.xs4all.nl (smtpzilla3.xs4all.nl [194.109.127.139])
by danu.procontrol.fi (Postfix) with ESMTP id F2204238A7
for ; Sat, 19 Oct 2002 18:11:37 +0300 (EEST)
Received: from centurion.xs4all.nl (centurion.xs4all.nl [194.109.0.100])
by smtpzilla3.xs4all.nl (8.12.0/8.12.0) with ESMTP id g9JFBbW0074539;
Sat, 19 Oct 2002 17:11:37 +0200 (CEST)
Received: by centurion.xs4all.nl (Postfix, from userid 1000)
id 9191F2268; Sat, 19 Oct 2002 17:10:26 +0200 (CEST)
Date: Sat, 19 Oct 2002 17:10:26 +0200
From: Thomas Wouters
To: Timo Sirainen
Cc: dovecot@procontrol.fi
Subject: [dovecot] Re: Architectural questions
Message-ID: <20021019151026.GM543@xs4all.nl>
References: <20021019123846.GK543@xs4all.nl> <1035036115.1674.81.camel@hurina>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1035036115.1674.81.camel@hurina>
User-Agent: Mutt/1.4i
X-message-flag: Danger Will Robinson!
X-archive-position: 49
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: thomas@xs4all.net
Precedence: bulk
X-list: dovecot
X-UID: 49
Status: O
Content-Length: 7406
On Sat, Oct 19, 2002 at 05:01:55PM +0300, Timo Sirainen wrote:
> On Sat, 2002-10-19 at 15:38, Thomas Wouters wrote:
> > We briefly played with modifying sendmail and the pop server to avoid the
> > full copy in the common case (only status changes) by doing in-place edits
> > of a pre-generated Status line,
> UW-imapd does this as well, creating "X-Keywords: " line for
> each mail. I had thought about this first with dovecot too, but since
> mutt rewrote the whole mailbox always I figured I might as well. But
> with larger mailboxes this is really slow, so I think I'll support the
> X-keywords trick myself too.
Well, for POP3 servers the story is a bit different than IMAP. The typical
use we were seeing was "user", "pass", "list", "retr ", "quit".
Sometimes (for some users) every few minutes. In that case, having to write
a 'RO' at a specific location in a large mbox is oodles more efficient than
copying the whole thing to local disk and back again (which is what the
popserver would do.) I'm not sure if it matters much with typical IMAP
usage.
> > as well as avoid full scanning of the mbox file by creating special
> > headers to mark the 'real' length of an email.
> For each mail? Content-Length? With my tests that didn't seem to help
> much, rather made it just slower.. Could be that I just did something
> badly, have to look into it more when I begin optimizing mbox handling
> more. Have to get it at least as fast as UW-imapd :)
Well, if I recall correctly, we added an 'X-Offset' header which pointed to
the exact (relative) byte offset for the next 'From ' line. It made our
pop3d (a modified qpopper 2.3 by the way) a much happier puppy. I'm not sure
what the difference with Content-Length was. I could find the sources, I
suppose; since we disabled mbox-inbox support we aren't using that code
anymore.
> > We still have support for mbox mailboxes in a user's homedirectory though,
> > by using procmail and such. So when we needed an IMAP server for use with
> > our webmail (based on SquirrelMail), we were forced to go with the UW-IMAP
> > server, with the maildir patch that's been scattered around the 'net. This
> Hm. Squirrelmail requires SORT extension which Dovecot doesn't support
> yet.
Ah, that's a shame. It means we can't use dovecot for our internal
SquirrelMail+IMAP testing yet :) We likely wouldn't start using dovecot for
production SquirrelMail anytime soon anyway, so it's not a big issue right
now... We'll have to see if our other uses of IMAP require it or not.
> Dovecot doesn't care [about maildir-message filenames] as long as the file
> name stays same before the ':' character.
They do.
> It would be possible to just assume that there's always someone else
> using the modify log, but each flag change or expunge would always write
> a few bytes to it then, and when log file is switched (there's .log and
> .log.2) it wouldn't be truncated after last process is finished with it
> which is not too bad since after the next switch it will be truncated.
> Also it would be possible not to use index files at all but just keep
> them in memory. I've been fixing code to make this possible and somewhat
> fast.
Hmm. I'd have to look at the code to say for sure, but I think we could live
with keeping them in memory. Accessing the same mailbox from two different
clients at the same time is not something we're too worried about, at the
moment.
> > - Every user's incoming mailbox is /var/spool/u/s/username.
> Are maildir inboxes also in /var/spool?
Yes. We don't use the ~/Maildir structure at all. We've always simply used
maildir mailboxes as a directly replacement of mbox mailboxes; a directory
instead of a file, and no sub-boxes :) I guess it's a philosphical
difference. To me, and to my colleagues, everything can be a mailbox, not
just something stored in an arbitrary directory somewhere. I guess we could
change that position, if necessary, but so far it hasn't proven to be.
> With mbox sub-inboxes wouldn't be even possible because dir structure ==
> mailbox structure, and since inbox file exists there can't be inbox-dir
> (except maybe with different case but that's kludgy).
Yes... don't worry, we don't even want to consider mbox-subboxes :)
> I've also thought I might as well make it possible to read the mbox
> inbox from /var/mail or whereever it is. Pretty easy to do, but .lock
> file is problematic if new files can't be added to the /var/mail
> directory.
Our /var/spool/mail subdirectories are mode 01733 (drwx-wx-wt) owned by
root, so creating files and removing them is not an issue, but reading the
directory is. You can of course still check for existance of specific
filenames.
> Am I right in that CPU usage still isn't any problem but rather the I/O?
Yes. As I said, we use several netapp filers (currently two for /home and
two for /var/spool/mail, with several hundred gigabytes filespace each) and
though they're great boxes, their performance does tend to drop off when it
gets flooded with I/O requests :) And they're used by a lot of machines, so
if they are slow to respond, a lot of our services do too.
> > - The UW-IMAP maildir patch stores UID's in the indiviual filenames, using
> > a 'U' flag. Will this interfere with dovecot ? We don't really need
> > dovecot and UW-IMAP to share UIDs, but we would like to have an as
> > painless transition as possible, without having to rename millions of
> > files to remove the U flag and other flags :P It would also be nice to
> > keep pine using the existing maildir patch, even though very few
> > IMAP-users would use pine.
> How exactly does the U flag work? I hope it's before the ':' character
> like Courier's S=filesize? Otherwise U=1234 would be thought of as 6
> different flags which isn't very good since Dovecot reorders them as
> 1234=U.
No, it can't be before the :, because the UID is generated by UW-IMAP, and
the maildir spec says you can't change the uniqe part of the name, just the
info :) Here are some examples. The ',U*' is the UID.
_k2,6NtZ9.maildrop4.xs4all.nl:2,S,U1030712092
_fmT,O63l8.maildrop8.xs4all.nl:2,RS,U1026644784
990612135.16312.000000002.maildrop2.xs4all.nl:2,S,U991994304
993058841.maildrop7.49267:2,S,U993058888
(In case you're wondering, the first two files were created by standard
procmail, the third by our modified procmail which tries to allow for the
pine/uw-imap maildir patch, and the last is our mail.local's format.)
As long as dovecot doesn't read a different meaning into those flags
(ignoring them is just fine) we should be fine. I don't think we'll have
many customers switching back and forth between dovecot and UW-IMAP, just
people switching from UW-IMAP to dovecot.
> If your clusters access the files through NFS, there should be no
> problem. Except I've never tried Dovecot through NFS, and I'm not sure
> how well mmap()ing works through NFS. I know there's been problems
> before but hopefully they've been fixed already.
I'm not too worried about bugs. I've yet to see a piece of software that we
don't find oodles of small and large bugs in just by installing and trying
to run on our clientbase. That's what testing is for :) But I wouldn't mind
being happily suprised by dovecot, we'll see :)
--
Thomas Wouters
Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
From tss@iki.fi Sat Oct 19 18:53:44 2002
Received: with ECARTIS (v1.0.0; list dovecot); Sat, 19 Oct 2002 18:53:44 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from hurina (ip213-185-36-189.laajakaista.mtv3.fi [213.185.36.189])
by danu.procontrol.fi (Postfix) with ESMTP id C511F2382D
for ; Sat, 19 Oct 2002 18:53:43 +0300 (EEST)
Received: by hurina (Postfix, from userid 1000)
id 941B65E03E5C; Sat, 19 Oct 2002 18:53:43 +0300 (EEST)
Subject: [dovecot] Re: Architectural questions
From: Timo Sirainen
To: dovecot@procontrol.fi
In-Reply-To: <20021019151026.GM543@xs4all.nl>
References: <20021019123846.GK543@xs4all.nl>
<1035036115.1674.81.camel@hurina> <20021019151026.GM543@xs4all.nl>
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Organization:
Message-Id: <1035042823.1679.127.camel@hurina>
Mime-Version: 1.0
X-Mailer: Ximian Evolution 1.1.1.99 (Preview Release)
Date: 19 Oct 2002 18:53:43 +0300
X-archive-position: 50
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 50
Status: O
Content-Length: 4750
On Sat, 2002-10-19 at 18:10, Thomas Wouters wrote:
> Well, if I recall correctly, we added an 'X-Offset' header which pointed to
> the exact (relative) byte offset for the next 'From ' line. It made our
> pop3d (a modified qpopper 2.3 by the way) a much happier puppy. I'm not sure
> what the difference with Content-Length was. I could find the sources, I
> suppose; since we disabled mbox-inbox support we aren't using that code
> anymore.
Content-Length saves just the size of mail body, so it can be skipped
over. I implemented it mostly because mutt doesn't escape the "From "
lines when saving mails so it was a bit difficult sometimes to figure
out if the From-line means a new mail or if it was just written into the
mail body.
I'm not sure what I should do with Dovecot, both From-line escaping and
Content-Length writing makes it annoyingly slower then now with more
code..
> > Also it would be possible not to use index files at all but just keep
> > them in memory. I've been fixing code to make this possible and somewhat
> > fast.
>
> Hmm. I'd have to look at the code to say for sure, but I think we could live
> with keeping them in memory. Accessing the same mailbox from two different
> clients at the same time is not something we're too worried about, at the
> moment.
Well, Outlook (and OE I think) opens two simultaneous connections
sometimes to fetch mails.
Not having index files doesn't affect the possibility to have multiple
connections, but it affects the overall performance because it needs to
do more I/O, especially with mbox.
> > I've also thought I might as well make it possible to read the mbox
> > inbox from /var/mail or whereever it is. Pretty easy to do, but .lock
> > file is problematic if new files can't be added to the /var/mail
> > directory.
>
> Our /var/spool/mail subdirectories are mode 01733 (drwx-wx-wt) owned by
> root, so creating files and removing them is not an issue, but reading the
> directory is. You can of course still check for existance of specific
> filenames.
OK, no problem then.
> > How exactly does the U flag work? I hope it's before the ':' character
> > like Courier's S=filesize? Otherwise U=1234 would be thought of as 6
> > different flags which isn't very good since Dovecot reorders them as
> > 1234=U.
>
> No, it can't be before the :, because the UID is generated by UW-IMAP, and
> the maildir spec says you can't change the uniqe part of the name, just the
> info :) Here are some examples. The ',U*' is the UID.
>
> _k2,6NtZ9.maildrop4.xs4all.nl:2,S,U1030712092
> _fmT,O63l8.maildrop8.xs4all.nl:2,RS,U1026644784
> 990612135.16312.000000002.maildrop2.xs4all.nl:2,S,U991994304
> 993058841.maildrop7.49267:2,S,U993058888
Well, maildir spec also doesn't say you can add flags with parameters
using comma separators :) But I think that's good enough extension that
Dovecot could support as well.
Programs supporting Courier's Maildir++ quota writes the mails
immediately like "something,S=size". Something like that could have been
done by UID-capable mailers too, since UID won't change.
> As long as dovecot doesn't read a different meaning into those flags
> (ignoring them is just fine) we should be fine. I don't think we'll have
> many customers switching back and forth between dovecot and UW-IMAP, just
> people switching from UW-IMAP to dovecot.
Keeping the UIDs untouched when changing could be important to some
people whose mail clients can save some extra information related to
specific messages and use UID to identify the mails. I think Evolution
does this with it's labels and Follow-Up marks, but I'm not sure.
Dovecot currently doesn't try to keep the UIDs too heavily itself
either, if it notices some corruption it just recreates the index files
with new UIDs. Supporting in-memory indexes requires still saving the
UIDs somewhere in disk so this should get fixed while doing it.
> I'm not too worried about bugs. I've yet to see a piece of software that we
> don't find oodles of small and large bugs in just by installing and trying
> to run on our clientbase. That's what testing is for :) But I wouldn't mind
> being happily suprised by dovecot, we'll see :)
We've had 3 people using it for a few months now, one of them still gets
sometimes "message not found" error from Outlook Express, I've yet to
figure out when exactly that happens. The mail isn't lost and restarting
OE helps, so it's probably something to do with having those two
simultaneous connections (and I'm just now making bigger changes there).
Other than that it's worked quite fine :)
Then of course CVS has a lot large changes which haven't been tested
much yet. Hopefully I'll get them fixed well enough this weekend to be
able to start using it myself.
From tss@iki.fi Sat Oct 19 19:11:49 2002
Received: with ECARTIS (v1.0.0; list dovecot); Sat, 19 Oct 2002 19:11:49 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from hurina (ip213-185-36-189.laajakaista.mtv3.fi [213.185.36.189])
by danu.procontrol.fi (Postfix) with ESMTP id D02AC2382D
for ; Sat, 19 Oct 2002 19:11:49 +0300 (EEST)
Received: by hurina (Postfix, from userid 1000)
id 9813D5E03E5C; Sat, 19 Oct 2002 19:11:49 +0300 (EEST)
Subject: [dovecot] Re: Architectural questions
From: Timo Sirainen
To: dovecot@procontrol.fi
In-Reply-To: <20021019151026.GM543@xs4all.nl>
References: <20021019123846.GK543@xs4all.nl>
<1035036115.1674.81.camel@hurina> <20021019151026.GM543@xs4all.nl>
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Organization:
Message-Id: <1035043909.31051.140.camel@hurina>
Mime-Version: 1.0
X-Mailer: Ximian Evolution 1.1.1.99 (Preview Release)
Date: 19 Oct 2002 19:11:49 +0300
X-archive-position: 51
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 51
Status: O
Content-Length: 1060
On Sat, 2002-10-19 at 18:10, Thomas Wouters wrote:
> > Am I right in that CPU usage still isn't any problem but rather the I/O?
>
> Yes. As I said, we use several netapp filers (currently two for /home and
> two for /var/spool/mail, with several hundred gigabytes filespace each) and
> though they're great boxes, their performance does tend to drop off when it
> gets flooded with I/O requests :) And they're used by a lot of machines, so
> if they are slow to respond, a lot of our services do too.
I was mostly wondering with this if the reason to add more computers to
cluster is because the imap processes are taking too much memory, too
much CPU or if neither of them is any problem and the cluster is just
for redundancy or because of other running programs.
I'd be interested to know how many dovecot processes could actually run
in a single computer especially with fast NFS file access :) I'd guess
it could run a lot, but when memory gets low the I/O usage raises since
it needs to read again the parts of indexes which got dropped from
memory.
From faulerhund@faulerhund.net Sat Oct 19 19:24:01 2002
Received: with ECARTIS (v1.0.0; list dovecot); Sat, 19 Oct 2002 19:24:02 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from moutvdom.kundenserver.de (moutvdom.kundenserver.de [195.20.224.130])
by danu.procontrol.fi (Postfix) with ESMTP id D51E6238A7
for ; Sat, 19 Oct 2002 19:24:01 +0300 (EEST)
Received: from [212.227.126.220] (helo=mrvdomng.kundenserver.de)
by moutvdom.kundenserver.de with esmtp (Exim 3.35 #1)
id 182wON-00080b-00
for dovecot@procontrol.fi; Sat, 19 Oct 2002 18:23:59 +0200
Received: from [217.229.166.89] (helo=Bini)
by mrvdomng.kundenserver.de with smtp (Exim 3.35 #1)
id 182wON-0001BG-00
for dovecot@procontrol.fi; Sat, 19 Oct 2002 18:23:59 +0200
Message-ID: <003b01c2778c$a6884370$0164a8c0@Bini>
From: "Korbinian Riedhammer"
To: "Dovecot Mailinglist"
Subject: [dovecot] still problems gettin it to work
Date: Sat, 19 Oct 2002 18:29:08 +0200
MIME-Version: 1.0
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2600.0000
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
X-archive-position: 52
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: faulerhund@faulerhund.net
Precedence: bulk
X-list: dovecot
X-UID: 52
Status: O
i checked all my options again. could it be, that it is a problem with my
compiler? does dovecot support gcc3.2 with glibc 2.2.5?
korbinian
From tss@iki.fi Sat Oct 19 19:27:45 2002
Received: with ECARTIS (v1.0.0; list dovecot); Sat, 19 Oct 2002 19:27:45 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from hurina (ip213-185-36-189.laajakaista.mtv3.fi [213.185.36.189])
by danu.procontrol.fi (Postfix) with ESMTP id 4F2F5238A7
for ; Sat, 19 Oct 2002 19:27:45 +0300 (EEST)
Received: by hurina (Postfix, from userid 1000)
id 0ECE55E03E5C; Sat, 19 Oct 2002 19:27:45 +0300 (EEST)
Subject: [dovecot] Re: still problems gettin it to work
From: Timo Sirainen
To: Korbinian Riedhammer
Cc: Dovecot Mailinglist
In-Reply-To: <003b01c2778c$a6884370$0164a8c0@Bini>
References: <003b01c2778c$a6884370$0164a8c0@Bini>
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Organization:
Message-Id: <1035044864.1674.143.camel@hurina>
Mime-Version: 1.0
X-Mailer: Ximian Evolution 1.1.1.99 (Preview Release)
Date: 19 Oct 2002 19:27:44 +0300
X-archive-position: 53
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 53
Status: O
On Sat, 2002-10-19 at 19:29, Korbinian Riedhammer wrote:
> i checked all my options again. could it be, that it is a problem with my
> compiler? does dovecot support gcc3.2 with glibc 2.2.5?
Well, what's the problem with it now? Didn't that /dev/urandom problem
go away with unsetting auth_chroot setting?
From faulerhund@faulerhund.net Sat Oct 19 19:56:55 2002
Received: with ECARTIS (v1.0.0; list dovecot); Sat, 19 Oct 2002 19:56:55 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from moutvdom.kundenserver.de (moutvdom.kundenserver.de [195.20.224.149])
by danu.procontrol.fi (Postfix) with ESMTP id 699AF238A7
for ; Sat, 19 Oct 2002 19:56:55 +0300 (EEST)
Received: from [212.227.126.220] (helo=mrvdomng.kundenserver.de)
by moutvdom.kundenserver.de with esmtp (Exim 3.35 #1)
id 182wuE-0005AD-00
for dovecot@procontrol.fi; Sat, 19 Oct 2002 18:56:54 +0200
Received: from [217.229.166.89] (helo=Bini)
by mrvdomng.kundenserver.de with smtp (Exim 3.35 #1)
id 182wuE-0002u1-00
for dovecot@procontrol.fi; Sat, 19 Oct 2002 18:56:54 +0200
Message-ID: <005d01c27791$4011d930$0164a8c0@Bini>
From: "Korbinian Riedhammer"
To: "Dovecot Mailinglist"
Subject: [dovecot] Re: still problems gettin it to work
Date: Sat, 19 Oct 2002 19:02:03 +0200
MIME-Version: 1.0
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2600.0000
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
X-archive-position: 54
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: faulerhund@faulerhund.net
Precedence: bulk
X-list: dovecot
X-UID: 54
Status: O
well i didn't know what to change, cause u wrote that this change u did is
pretty useless rinning the prog as root, what i do. next thing that i dont
have any idea about cvs (beside knowing its meaning) and dont hav it
installed yet.
From tss@iki.fi Sat Oct 19 19:59:22 2002
Received: with ECARTIS (v1.0.0; list dovecot); Sat, 19 Oct 2002 19:59:22 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from hurina (ip213-185-36-189.laajakaista.mtv3.fi [213.185.36.189])
by danu.procontrol.fi (Postfix) with ESMTP id C7B16238A7
for ; Sat, 19 Oct 2002 19:59:22 +0300 (EEST)
Received: by hurina (Postfix, from userid 1000)
id 99EA15E03E5C; Sat, 19 Oct 2002 19:59:22 +0300 (EEST)
Subject: [dovecot] Re: still problems gettin it to work
From: Timo Sirainen
To: Korbinian Riedhammer
Cc: Dovecot Mailinglist
In-Reply-To: <005d01c27791$4011d930$0164a8c0@Bini>
References: <005d01c27791$4011d930$0164a8c0@Bini>
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Organization:
Message-Id: <1035046762.1674.148.camel@hurina>
Mime-Version: 1.0
X-Mailer: Ximian Evolution 1.1.1.99 (Preview Release)
Date: 19 Oct 2002 19:59:22 +0300
X-archive-position: 55
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 55
Status: O
On Sat, 2002-10-19 at 20:02, Korbinian Riedhammer wrote:
> well i didn't know what to change, cause u wrote that this change u did is
> pretty useless rinning the prog as root, what i do. next thing that i dont
> have any idea about cvs (beside knowing its meaning) and dont hav it
> installed yet.
Well, just don't set the auth_chroot and ignores the rest of what I
said. That should get it working.
From faulerhund@faulerhund.net Sat Oct 19 20:04:30 2002
Received: with ECARTIS (v1.0.0; list dovecot); Sat, 19 Oct 2002 20:04:30 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from moutvdom.kundenserver.de (moutvdom.kundenserver.de [195.20.224.200])
by danu.procontrol.fi (Postfix) with ESMTP id 969F2238A7
for ; Sat, 19 Oct 2002 20:04:30 +0300 (EEST)
Received: from [195.20.224.206] (helo=mrvdomng.kundenserver.de)
by moutvdom.kundenserver.de with esmtp (Exim 3.35 #1)
id 182x1a-0002pK-00
for dovecot@procontrol.fi; Sat, 19 Oct 2002 19:04:30 +0200
Received: from [217.229.166.89] (helo=Bini)
by mrvdomng.kundenserver.de with smtp (Exim 3.35 #1)
id 182x1Z-0003C1-00
for dovecot@procontrol.fi; Sat, 19 Oct 2002 19:04:30 +0200
Message-ID: <006701c27792$4f863040$0164a8c0@Bini>
From: "Korbinian Riedhammer"
To: "Dovecot Mailinglist"
Subject: [dovecot] Re: still problems gettin it to work
Date: Sat, 19 Oct 2002 19:09:39 +0200
MIME-Version: 1.0
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2600.0000
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
X-archive-position: 56
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: faulerhund@faulerhund.net
Precedence: bulk
X-list: dovecot
X-UID: 56
Status: O
yeah, the nasty error msgs are gone :)
well next question. where looks the dovcot imapd for the mail dir? /var/mail
or the homedir? 'cause i saved my Maildir in /home/user
From tss@iki.fi Sat Oct 19 20:08:34 2002
Received: with ECARTIS (v1.0.0; list dovecot); Sat, 19 Oct 2002 20:08:34 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from hurina (ip213-185-36-189.laajakaista.mtv3.fi [213.185.36.189])
by danu.procontrol.fi (Postfix) with ESMTP id 4C086238A7
for ; Sat, 19 Oct 2002 20:08:34 +0300 (EEST)
Received: by hurina (Postfix, from userid 1000)
id 0FC605E03E5C; Sat, 19 Oct 2002 20:08:31 +0300 (EEST)
Subject: [dovecot] Re: still problems gettin it to work
From: Timo Sirainen
To: Dovecot Mailinglist
In-Reply-To: <006701c27792$4f863040$0164a8c0@Bini>
References: <006701c27792$4f863040$0164a8c0@Bini>
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Organization:
Message-Id: <1035047310.1674.152.camel@hurina>
Mime-Version: 1.0
X-Mailer: Ximian Evolution 1.1.1.99 (Preview Release)
Date: 19 Oct 2002 20:08:30 +0300
X-archive-position: 57
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 57
Status: O
On Sat, 2002-10-19 at 20:09, Korbinian Riedhammer wrote:
> yeah, the nasty error msgs are gone :)
> well next question. where looks the dovcot imapd for the mail dir? /var/mail
> or the homedir? 'cause i saved my Maildir in /home/user
It uses ~/Maildir always.
From faulerhund@faulerhund.net Sat Oct 19 20:17:48 2002
Received: with ECARTIS (v1.0.0; list dovecot); Sat, 19 Oct 2002 20:17:48 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from moutvdom.kundenserver.de (moutvdom.kundenserver.de [195.20.224.131])
by danu.procontrol.fi (Postfix) with ESMTP id 555172382D
for ; Sat, 19 Oct 2002 20:17:48 +0300 (EEST)
Received: from [212.227.126.220] (helo=mrvdomng.kundenserver.de)
by moutvdom.kundenserver.de with esmtp (Exim 3.35 #1)
id 182xES-0000X5-00
for dovecot@procontrol.fi; Sat, 19 Oct 2002 19:17:48 +0200
Received: from [217.229.166.89] (helo=Bini)
by mrvdomng.kundenserver.de with smtp (Exim 3.35 #1)
id 182xER-0003ij-00
for dovecot@procontrol.fi; Sat, 19 Oct 2002 19:17:47 +0200
Message-ID: <008901c27794$2b0de030$0164a8c0@Bini>
From: "Korbinian Riedhammer"
To: "Dovecot Mailinglist"
Subject: [dovecot] Re: still problems gettin it to work
Date: Sat, 19 Oct 2002 19:22:57 +0200
MIME-Version: 1.0
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2600.0000
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
X-archive-position: 58
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: faulerhund@faulerhund.net
Precedence: bulk
X-list: dovecot
X-UID: 58
Status: O
well finally it was my own stupidness!
i just forgot to set some correct permissions, now everythings fine!
thx anyway
korbinian
From ianj@ian-justman.com Sun Oct 20 09:38:35 2002
Received: with ECARTIS (v1.0.0; list dovecot); Sun, 20 Oct 2002 09:38:35 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from narshe.chocobo.org (dsl-207-126-72-242.dsl.netasset.net [207.126.72.242])
by danu.procontrol.fi (Postfix) with ESMTP id C321D238A7
for ; Sun, 20 Oct 2002 09:38:34 +0300 (EEST)
Received: from zozo.intrn.chocobo.org (zozo.chocobo.org [207.126.72.244])
by narshe.chocobo.org (Postfix) with ESMTP id 18975311609
for ; Sat, 19 Oct 2002 23:38:31 -0700 (PDT)
Received: by zozo.intrn.chocobo.org (Postfix, from userid 501)
id 5C94C870704; Sat, 19 Oct 2002 23:38:30 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1])
by zozo.intrn.chocobo.org (Postfix) with ESMTP id 10A40449AD4
for ; Sat, 19 Oct 2002 23:38:29 -0700 (PDT)
Date: Sat, 19 Oct 2002 23:38:29 -0700 (PDT)
From: "Ian R. Justman"
X-X-Sender: ianj@zozo.chocobo.org
To: Dovecot Mailinglist
Subject: [dovecot] observations/qestions of mbox concurrency ability (Yay!) on current
CVS
Message-ID:
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
X-archive-position: 59
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: ianj@ian-justman.com
Precedence: bulk
X-list: dovecot
X-UID: 59
Status: O
Hi, all.
Just been testing out the latest "CVS" (in quotes because I'm rsyncing
the source
tree) versions of Dovecot lately.
Tested having Pine on the machine on which Dovecot was running, talking
IMAP, as well as another Pine instance, this time from my Windows machine
over the wire. Things seem to be working wonderfully. Deleted a message
using one instance, then refreshed the list on the other. *poof* Gone.
Though I'm seeing periodic "IMAP protocol error" messages in Pine, but
they don't seem to be affecting the software's operation that I can see.
Anything in particular I should be on the lookout for?
Otherwise, you just made my day. Correction; week. :D
--Ian.
From tss@iki.fi Sun Oct 20 16:47:50 2002
Received: with ECARTIS (v1.0.0; list dovecot); Sun, 20 Oct 2002 16:47:50 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from hurina (ip213-185-36-189.laajakaista.mtv3.fi [213.185.36.189])
by danu.procontrol.fi (Postfix) with ESMTP id 84B752382D
for ; Sun, 20 Oct 2002 16:47:50 +0300 (EEST)
Received: by hurina (Postfix, from userid 1000)
id 6AC945E03E5C; Sun, 20 Oct 2002 16:47:49 +0300 (EEST)
Subject: [dovecot] Re: observations/qestions of mbox concurrency ability
(Yay!) on current CVS
From: Timo Sirainen
To: "Ian R. Justman"
Cc: Dovecot Mailinglist
In-Reply-To:
References:
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Organization:
Message-Id: <1035121669.24867.8.camel@hurina>
Mime-Version: 1.0
X-Mailer: Ximian Evolution 1.1.1.99 (Preview Release)
Date: 20 Oct 2002 16:47:49 +0300
X-archive-position: 60
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: tss@iki.fi
Precedence: bulk
X-list: dovecot
X-UID: 60
Status: O
Content-Length: 1458
On Sun, 2002-10-20 at 09:38, Ian R. Justman wrote:
> Tested having Pine on the machine on which Dovecot was running, talking
> IMAP, as well as another Pine instance, this time from my Windows machine
> over the wire. Things seem to be working wonderfully. Deleted a message
> using one instance, then refreshed the list on the other. *poof* Gone.
>
> Though I'm seeing periodic "IMAP protocol error" messages in Pine, but
> they don't seem to be affecting the software's operation that I can see.
>
> Anything in particular I should be on the lookout for?
By default Dovecot uses .lock file and flock() locking. Some programs
may use fcntl() which doesn't work with flock(), but most use .lock file
as well so it doesn't really matter. I'm going to make the locking style
configurable later.
Anyway, I don't know of any reasons for mailbox corruption, but we don't
lock the mailbox when reading from it, so it is possible that someone
else might modify (expunge) the mailbox while Dovecot is reading it and
the IMAP client gets corrupted message.
I was thinking about fixing this using read-locking fcntl() / flock().
Maybe it could optionally create the .lock file as well, but that'd slow
down simultaneous reads.
Oh, and the mbox parsing code could use better checking, if there's any
bugs it will corrupt the mailbox at expunge/flag rewrite. I haven't seen
those though for the month or two that I've been using Dovecot+mbox to
read my mail.
From thomas@xs4all.net Mon Oct 21 14:53:07 2002
Received: with ECARTIS (v1.0.0; list dovecot); Mon, 21 Oct 2002 14:53:07 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from smtpzilla1.xs4all.nl (smtpzilla1.xs4all.nl [194.109.127.137])
by danu.procontrol.fi (Postfix) with ESMTP id 4333B2382D
for ; Mon, 21 Oct 2002 14:53:07 +0300 (EEST)
Received: from centurion.xs4all.nl (centurion.xs4all.nl [194.109.0.100])
by smtpzilla1.xs4all.nl (8.12.0/8.12.0) with ESMTP id g9LBr6ni040943
for ; Mon, 21 Oct 2002 13:53:06 +0200 (CEST)
Received: by centurion.xs4all.nl (Postfix, from userid 1000)
id DC6BB224C; Mon, 21 Oct 2002 13:51:43 +0200 (CEST)
Date: Mon, 21 Oct 2002 13:51:43 +0200
From: Thomas Wouters
To: dovecot@procontrol.fi
Subject: [dovecot] Re: Architectural questions
Message-ID: <20021021115143.GS543@xs4all.nl>
References: <20021019123846.GK543@xs4all.nl> <1035036115.1674.81.camel@hurina>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1035036115.1674.81.camel@hurina>
User-Agent: Mutt/1.4i
X-message-flag: Danger Will Robinson!
X-archive-position: 61
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: thomas@xs4all.net
Precedence: bulk
X-list: dovecot
X-UID: 61
Status: O
Content-Length: 1279
On Sat, Oct 19, 2002 at 05:01:55PM +0300, Timo Sirainen wrote:
> If your clusters access the files through NFS, there should be no
> problem. Except I've never tried Dovecot through NFS, and I'm not sure
> how well mmap()ing works through NFS. I know there's been problems
> before but hopefully they've been fixed already.
Hmm. I'm not sure what kind of behaviour you're looking for, but here's what
I see, using a little Python script on our FreeBSD servers with a
netapp-mounted filesystem. Mapping MAP_SHARED and PROT_READ|PROT_WRITE, two
different machines mounting the same directory, two processes on each
machine mmap()ing the same file.
When one process alters the data, the other process on the same machine sees
it instantly. The processes on the other machine do not see it at all, not
even when re-opening the mmap or being restarted. After doing an msync() in
the process that altered the data, the processes on the other machine still
don't see the change; they have to re-open the mmap or be restarted before
they see the change -- but when one of the processes re-opens or restarts,
the other does see the change without doing anything.
--
Thomas Wouters
Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
From thomas@xs4all.net Mon Oct 21 15:40:02 2002
Received: with ECARTIS (v1.0.0; list dovecot); Mon, 21 Oct 2002 15:40:02 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from smtpzilla3.xs4all.nl (smtpzilla3.xs4all.nl [194.109.127.139])
by danu.procontrol.fi (Postfix) with ESMTP id 868CD23837
for ; Mon, 21 Oct 2002 15:40:02 +0300 (EEST)
Received: from centurion.xs4all.nl (centurion.xs4all.nl [194.109.0.100])
by smtpzilla3.xs4all.nl (8.12.0/8.12.0) with ESMTP id g9LCe2wC017139
for ; Mon, 21 Oct 2002 14:40:02 +0200 (CEST)
Received: by centurion.xs4all.nl (Postfix, from userid 1000)
id 19E90224C; Mon, 21 Oct 2002 14:38:39 +0200 (CEST)
Date: Mon, 21 Oct 2002 14:38:39 +0200
From: Thomas Wouters
To: dovecot@procontrol.fi
Subject: [dovecot] Re: Architectural questions
Message-ID: <20021021123839.GW543@xs4all.nl>
References: <20021019123846.GK543@xs4all.nl> <1035036115.1674.81.camel@hurina> <20021019151026.GM543@xs4all.nl> <1035042823.1679.127.camel@hurina>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1035042823.1679.127.camel@hurina>
User-Agent: Mutt/1.4i
X-message-flag: Danger Will Robinson!
X-archive-position: 62
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: thomas@xs4all.net
Precedence: bulk
X-list: dovecot
X-UID: 62
Status: O
On Sat, Oct 19, 2002 at 06:53:43PM +0300, Timo Sirainen wrote:
> > As long as dovecot doesn't read a different meaning into those flags
> > (ignoring them is just fine) we should be fine. I don't think we'll have
> > many customers switching back and forth between dovecot and UW-IMAP, just
> > people switching from UW-IMAP to dovecot.
> Keeping the UIDs untouched when changing could be important to some
> people whose mail clients can save some extra information related to
> specific messages and use UID to identify the mails. I think Evolution
> does this with it's labels and Follow-Up marks, but I'm not sure.
Well, what I meant was that currently, IMAP is being used only by
SquirrelMail, and I'm fairly sure SquirrelMail doesn't store UIDs anywhere.
Other IMAP clients would be an issue only if we allowed other IMAP clients
to connect, which we don't (except in internal tests :).
--
Thomas Wouters
Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
From thomas@xs4all.net Mon Oct 21 16:04:50 2002
Received: with ECARTIS (v1.0.0; list dovecot); Mon, 21 Oct 2002 16:04:50 +0300 (EEST)
Return-Path:
Delivered-To: dovecot@procontrol.fi
Received: from smtpzilla5.xs4all.nl (smtpzilla5.xs4all.nl [194.109.127.141])
by danu.procontrol.fi (Postfix) with ESMTP id 3F98523837
for ; Mon, 21 Oct 2002 16:04:50 +0300 (EEST)
Received: from centurion.xs4all.nl (centurion.xs4all.nl [194.109.0.100])
by smtpzilla5.xs4all.nl (8.12.0/8.12.0) with ESMTP id g9LD4gEH017726;
Mon, 21 Oct 2002 15:04:47 +0200 (CEST)
Received: by centurion.xs4all.nl (Postfix, from userid 1000)
id 4840D224C; Mon, 21 Oct 2002 15:03:19 +0200 (CEST)
Date: Mon, 21 Oct 2002 15:03:19 +0200
From: Thomas Wouters
To: Charlie Brady
Cc: dovecot@procontrol.fi
Subject: [dovecot] Re: Architectural questions
Message-ID: <20021021130319.GX543@xs4all.nl>
References: <20021019123846.GK543@xs4all.nl>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To:
User-Agent: Mutt/1.4i
X-message-flag: Danger Will Robinson!
X-archive-position: 63
X-ecartis-version: Ecartis v1.0.0
Sender: dovecot-bounce@procontrol.fi
Errors-to: dovecot-bounce@procontrol.fi
X-original-sender: thomas@xs4all.net
Precedence: bulk
X-list: dovecot
X-UID: 63
Status: O
Content-Length: 2995
On Sat, Oct 19, 2002 at 12:42:25PM -0400, Charlie Brady wrote:
> > So when we needed an IMAP server for use with our webmail (based on
> > SquirrelMail), we were forced to go with the UW-IMAP server, with the
> > maildir patch that's been scattered around the 'net. This worked, for a
> > while; we also use the maildir patch with pine after all. However, the
> > maildir patch is not very good. Not at all, even, and it only seems to
> > work by pure chance.
> The best one I've found is the patch last modified AFAICT by Miquel van
> Smoorenburg, which has maildir filenames like:
> time.pid.host,U=xxx,W=yyy:2,flags
We use two different patches, both of which have a bit of Mike in them. One
is for the IMAP server, which is an old version ('uw-imap-2000' or something
like that comes to mind, but it came from the pine 4.10 source) from before
the mailbox->append prototype changed (but with Mike's bilennium-patch.) A
big problem with later patches was that the append method changed in the
pine source, but not in the maildir patch. We considered using a newer
uw-imap but decided that the current one works good enough ;P
For pine we currently use the patch that comes with Debian's 'pine-tracker'
package (which is an installer for pine with patches, since you aren't
allowed to distributed modified binaries.) This patch also has some Mike in
it, but I'm not sure howmuch, as at least parts of it seem to be backed out
later. This patch works okay except for the two problems I noted in my
original mail: it depends on directory order not to change except when
'.uidvalidity' gets touched, and it depends on alphanumerical sort order of
files matching chronological (or at least uid-based, which should be the
same) sort order. The latter breaks with (standard) procmail, the former
occasionally with btree and (presumably) hashed directory indices.
> The storage of RFC822.SIZE, aka the on-the-wire size, in the filename
> makes a very big difference to performance.
That's interesting. I'll keep that in mind for when we begin to see
performance issues with UW-IMAP. (I'm hoping I never have to look at the
pine source again, though.)
> But even this patch (from, e.g.
> http://www.star.le.ac.uk/~tjg/misc/uw_imap-2001a_maildir-02.patch) has a
> number of bugs. I've fixed a few of them. You can find a source RPM at
> ftp://ftp.e-smith.org/pub/e-smith/dev/5.6dev/SRPMS/. The fixes are:
I couldn't find the source RPM for pine or (uw-)imap in that directory, but
it doesn't sound like your changes solve our fundamental problems. It's not
that big a deal, I think we've decided internally (I know _I_ have :) that
we can't offer real IMAP services based on UW-IMAP; we'd sooner go for Cyrus
or Courier, even if it does mean disallowing mboxes with IMAP. But dovecot
is an even better alternative, once it has all the features we need :)
--
Thomas Wouters