Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reading via a large number of goroutines breaks #289

Closed
anantn opened this issue Mar 21, 2016 · 4 comments
Closed

Reading via a large number of goroutines breaks #289

anantn opened this issue Mar 21, 2016 · 4 comments

Comments

@anantn
Copy link

anantn commented Mar 21, 2016

Reading from more than ~1000 goroutines simultaneously fails with an error "unable to open database file". Test program is attached. You can tweak maxRoutines to see the normal/broken behavior, on my local tests everything works fine until about 1000 goroutines.

package main

import (
    "database/sql"
    "fmt"
    "log"
    "math/rand"
    "os"
    "sync"

    _ "github.com/mattn/go-sqlite3"
)

const (
    maxKeys     = 20
    maxRoutines = 5000
)

func main() {
    // Create DB.
    os.Remove("./test.db")
    destDb, err := sql.Open("sqlite3", "./test.db")
    if err != nil {
        log.Fatal(err)
    }
    defer destDb.Close()
    destDb.Ping()

    // Create table.
    _, err = destDb.Exec("create table test(id int, value text)")
    if err != nil {
        log.Fatal(err)
    }

    // Write 20 values.
    for i := 0; i < maxKeys; i++ {
        _, err = destDb.Exec(fmt.Sprintf("insert into test values(%d, 'random_value')", i))
        if err != nil {
            log.Fatal(err)
        }
    }

    // Now test.
    stressTest(destDb)
    fmt.Printf("Success!\n")
}

func stressTest(db *sql.DB) {
    // Read values from all of these simultaneously.
    var wg sync.WaitGroup
    for i := 0; i < maxRoutines; i++ {
        wg.Add(1)
        go read(i, db, &wg)
    }
    wg.Wait()
}

func read(num int, db *sql.DB, wg *sync.WaitGroup) {
    defer wg.Done()
    _, err := db.Query(fmt.Sprintf("select value from test where id=%d", rand.Intn(maxKeys)))
    if err != nil {
        fmt.Printf("Goroutine %d failed read : %v\n", num, err)
    }
}
@mattn
Copy link
Owner

mattn commented Mar 23, 2016

I got Success! with maxRoutines=5000.

@anantn
Copy link
Author

anantn commented Jun 3, 2016

This may be a platform specific bug, I'm running on Mac OS X 10.11.5 (El Capitan)

$ uname -a
Darwin 15.5.0 Darwin Kernel Version 15.5.0: Tue Apr 19 18:36:36 PDT 2016; root:xnu-3248.50.21~8/RELEASE_X86_64 x86_64

@Allendar
Copy link

Allendar commented Jun 20, 2016

Yes it is. OS X limits OS-wide to not have more than 1000 files open simultaneously by default. This is also a very common issue when trying to use ApacheBench to test your Go server locally (also a reason to not use it anymore go Go server testing).

I think some Googling will likely get you to find how to adjust this limit based on your current OS X version. Take caution to tweak such things tho.

@phikal
Copy link

phikal commented Sep 9, 2017

How come this stops working, when one uses prepared statements?

gjrtimmer added a commit to gjrtimmer/go-sqlite3 that referenced this issue May 26, 2018
Compile Section Closes mattn#175
Compile Section Closes mattn#201
Compile Section Closes mattn#206
Compile Section Closes mattn#404
Compile Section Closes mattn#217
Compile Section Closes mattn#224
Compile Section Closes mattn#234
Compile Section Closes mattn#242
Feature table Closes mattn#255
Description Section Closes mattn#232
Golang:1.6 not supported Closes mattn#272
Golang:1.5 not supported + compilation section Closes mattn#283
usleep Implemented Closes mattn#285
FAQ Section Closes mattn#289
Compile Section closes mattn#295
FAQ Section Closes mattn#305
PR339 Closes mattn#318 mattn#321
Compilation Section Closes mattn#341
PR407 Closes mattn#364
Feature `sqlite_vtable` Closes mattn#393
Compile Section Closes mattn#416
sqlite_trace feature Closes mattn#433
Compilation Section Closes mattn#435
Compilation Section Closes mattn#443
Golang:1.6 Not Supported Closes mattn#445
Compilation Section Closes mattn#451
Compilation Section Closes mattn#467
Compilation Section Closes mattn#491
Compilation Section Closes mattn#495
Compilation Section Closes mattn#505
Compilation Section Closes mattn#557
Compilation Section Closes mattn#560
@mattn mattn closed this as completed in c9394b1 May 27, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants