Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Internal error while reading parquet #394

@as7492

Description

@as7492

Getting below error while querying parquet stored on GCS location

INTERNAL Error: Attempted to dereference unique_ptr that is NULL!

Stack Trace:

/tmp/libduckdb_java4521970872809828390.so(+0x469a76) [0x7f1431669a76]
/tmp/libduckdb_java4521970872809828390.so(+0x469b24) [0x7f1431669b24]
/tmp/libduckdb_java4521970872809828390.so(+0x46c511) [0x7f143166c511]
/tmp/libduckdb_java4521970872809828390.so(+0x38fb05) [0x7f143158fb05]
/tmp/libduckdb_java4521970872809828390.so(+0x1a06b66) [0x7f1432c06b66]
/tmp/libduckdb_java4521970872809828390.so(+0x1a06876) [0x7f1432c06876]
/tmp/libduckdb_java4521970872809828390.so(+0x1a18f92) [0x7f1432c18f92]
/tmp/libduckdb_java4521970872809828390.so(+0x1a195e0) [0x7f1432c195e0]
/tmp/libduckdb_java4521970872809828390.so(+0x1a69fa7) [0x7f1432c69fa7]
/tmp/libduckdb_java4521970872809828390.so(+0xa5a63c) [0x7f1431c5a63c]
/tmp/libduckdb_java4521970872809828390.so(+0x102886d) [0x7f143222886d]
/tmp/libduckdb_java4521970872809828390.so(+0x1032f88) [0x7f1432232f88]
/tmp/libduckdb_java4521970872809828390.so(+0x1033342) [0x7f1432233342]
/tmp/libduckdb_java4521970872809828390.so(+0x102a376) [0x7f143222a376]
/tmp/libduckdb_java4521970872809828390.so(+0x1032cc4) [0x7f1432232cc4]
/tmp/libduckdb_java4521970872809828390.so(+0xeba730) [0x7f14320ba730]
/tmp/libduckdb_java4521970872809828390.so(+0xeba96c) [0x7f14320ba96c]
/tmp/libduckdb_java4521970872809828390.so(+0xebab16) [0x7f14320bab16]
/tmp/libduckdb_java4521970872809828390.so(+0xebd220) [0x7f14320bd220]
/tmp/libduckdb_java4521970872809828390.so(+0x3e0ff2) [0x7f14315e0ff2]
/tmp/libduckdb_java4521970872809828390.so(Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1execute+0x11) [0x7f14315e5ec1]
[0x841ff1053c]

This error signals an assertion failure within DuckDB. This usually occurs due to unexpected conditions or errors in the program's logic.
For more information, see https://duckdb.org/docs/stable/dev/internal_errors.

Here is code where we are creating connection

@Bean(name = "duckDbConnection")
public DuckDBConnection duckDbConnections() throws SQLException, IOException {

    String dbFilePath = "/opt/app/db";
    String jdbcUrl = "jdbc:duckdb:" + dbFilePath;
    File dbFile = new File(dbFilePath);
    dbFile.deleteOnExit();

    String connectionInitSql =
                    "SET threads=4; " +
                    "SET memory_limit='2GB';";

    log.info("label=DUCK_DB_IN_MEM_INIT_SQL, sql={}", connectionInitSql);

    String keyId = parquetReaderConfig.getKeyId();
    String secret = parquetReaderConfig.getSecret();

    connectionInitSql += "CREATE OR REPLACE SECRET (TYPE gcs, KEY_ID '" + keyId + "',SECRET '" + secret + "');";

    Properties connProperties = new Properties();
    connProperties.setProperty("duckdb.read_only", "true");

    if (!dbFile.exists()) {
        try (Connection conn = DriverManager.getConnection(jdbcUrl)) {
        } catch (SQLException e) {
            log.error("Error while creating db file");
            throw new RuntimeException(e);
        }
    }

    DuckDBConnection conn = (DuckDBConnection) DriverManager.getConnection(jdbcUrl, connProperties);
    conn.createStatement().execute(connectionInitSql);
    return conn;
}

And here is the code where we are using above created connection

String selectQuery1 = "SELECT * FROM parquet_scan(['gs://<bucket-name>/historical/destinationlocation=1/*.parquet','gs://<bucket-name>/current/destinationlocation=1/*.parquet']) WHERE replgrpnbr = '1000000'";

try (Connection conn = duckDbConnection.duplicate(); Statement stmt = conn.createStatement()) {
        try (ResultSet rs = stmt.executeQuery(selectQuery1)) {
        while (rs.next()) {
            // utilise rs to prepare response object
        }
    }
} catch (SQLException e) {
    throw new SQLException("Failed to retrieve fulfillment plan due to SQL error", lastException);
}

Below is dependency we are using in our project

org.duckdb
duckdb_jdbc
1.3.0.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions