Insert has data races that result in heap corruption #8
Description
Hello,
When using Pipe with a relatively high parallelism (12) I started getting some odd panics in the internal index heap while using Search. It looks like some of the index data became corrupted. When I reverted to using single-goroutine Insert, the problem disappears.
Running with go run -race, the unprotected write appears here:
Lines 95 to 100 in 89a7eb7
I think there are a few other writes which bypass the mutex as well.
WARNING: DATA RACE
Write at 0x00c00f225940 by goroutine 9:
github.com/fogfish/hnsw.(*HNSW[go.shape.struct { URI string "parquet:\"uri\" json:\"uri\""; Embedding512 string "parquet:\"embedding_512\" json:\"-\""; EmbeddingInput string "parquet:\"embedding_input\" json:\"input\""; Vec []float32 "parquet:\"-\" json:\"-\""; ID int "parquet:\"-\" json:\"-\"" }]).Insert()
/Users/zorin.alex/go/pkg/mod/github.com/fogfish/[email protected]/insert.go:98 +0x11d0
github.com/fogfish/hnsw.(*HNSW[go.shape.struct { URI string "parquet:\"uri\" json:\"uri\""; Embedding512 string "parquet:\"embedding_512\" json:\"-\""; EmbeddingInput string "parquet:\"embedding_input\" json:\"input\""; Vec []float32 "parquet:\"-\" json:\"-\""; ID int "parquet:\"-\" json:\"-\"" }]).Pipe.func1()
/Users/zorin.alex/go/pkg/mod/github.com/fogfish/[email protected]/pipe.go:25 +0xf0
Previous read at 0x00c00f225940 by goroutine 10:
github.com/fogfish/hnsw.(*HNSW[go.shape.struct { URI string "parquet:\"uri\" json:\"uri\""; Embedding512 string "parquet:\"embedding_512\" json:\"-\""; EmbeddingInput string "parquet:\"embedding_input\" json:\"input\""; Vec []float32 "parquet:\"-\" json:\"-\""; ID int "parquet:\"-\" json:\"-\"" }]).searchLayer()
/Users/zorin.alex/go/pkg/mod/github.com/fogfish/[email protected]/search.go:68 +0x308
github.com/fogfish/hnsw.(*HNSW[go.shape.struct { URI string "parquet:\"uri\" json:\"uri\""; Embedding512 string "parquet:\"embedding_512\" json:\"-\""; EmbeddingInput string "parquet:\"embedding_input\" json:\"input\""; Vec []float32 "parquet:\"-\" json:\"-\""; ID int "parquet:\"-\" json:\"-\"" }]).Insert()
/Users/zorin.alex/go/pkg/mod/github.com/fogfish/[email protected]/insert.go:83 +0x474
github.com/fogfish/hnsw.(*HNSW[go.shape.struct { URI string "parquet:\"uri\" json:\"uri\""; Embedding512 string "parquet:\"embedding_512\" json:\"-\""; EmbeddingInput string "parquet:\"embedding_input\" json:\"input\""; Vec []float32 "parquet:\"-\" json:\"-\""; ID int "parquet:\"-\" json:\"-\"" }]).Pipe.func1()
/Users/zorin.alex/go/pkg/mod/github.com/fogfish/[email protected]/pipe.go:25 +0xf0