Hi everyone,

I'm currently working on a Spring Boot project using Tablesaw for data processing of large DataFrames. While using the dropDuplicateRows() method, I noticed some unexpected behavior.

Here’s a minimal reproducible example:

java

Table testTable = Table.create("test")
    .addColumns(
        IntColumn.create("part_id", new int[]{1, 1, 1, 2, 2, 2}),
        StringColumn.create("nsequence", new String[]{"N40", "N50", "N60", "N40", "N50", "N60"})
    );

Table testUnique = testTable.selectColumns("part_id", "nsequence").dropDuplicateRows();
logger.info("Test unique pairs: {}", testUnique.print());

Expected Output:

part_id  |  nsequence
----------------------
1        |  N40
1        |  N50
1        |  N60
2        |  N40
2        |  N50
2        |  N60

Actual Output:

part_id  |  nsequence
----------------------
1        |  N40
1        |  N50
1        |  N60
2        |  N60

As you can see, rows like (2, N40) and (2, N50) are missing after calling dropDuplicateRows(). This was surprising because the combination of part_id and nsequence is unique across all rows.

Investigation
I looked at the source for dropDuplicateRows():

java

public Table dropDuplicateRows() {
    Table temp = this.emptyCopy();
    Set uniqueRows = new HashSet();

    for (Row row : this) {
        if (!uniqueRows.contains(row)) {
            uniqueRows.add(row);
            temp.append(row);
        }
    }

    return temp;
}

It looks like the uniqueness is determined based on the Row object, but I suspect the Row class doesn't override equals() and hashCode() in a way that compares actual content — possibly relying on object identity or internal state like row index.

Environment
Tablesaw version: Using latest via Maven:

xml

<dependency>
    <groupId>tech.tablesaw</groupId>
    <artifactId>tablesaw-core</artifactId>
    <version>LATEST</version>
</dependency>

Java: OpenJDK 21

Framework: Spring Boot

Request
Could someone please confirm:

Is this an expected behavior?

Should Row properly override equals() and hashCode()?

Is there a workaround for getting proper unique row behavior based on content?

I'm new to Java, so if this is expected and there's a better way to achieve row de-duplication based on column values, I’d appreciate any guidance.

Thank you for your help!