[Java] Purge non-empty nulls when setting validity

[razajafri]

Description

When setting the validity buffer in mergeAndSetValidity method, we check to see two things

1. Purge any non-empty nulls that may have arise due to setting values to null which were previously non-null
2. If we override the existing null values in the validity buffer then throw an IllegalStateException.

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[razajafri]

Tests pass locally when I run ci/test_java.sh in rapidsai/ci:cuda11.8.0-ubuntu22.04-py3.10

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running ai.rapids.cudf.ColumnViewNonEmptyNullsTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.066 s - in ai.rapids.cudf.ColumnViewNonEmptyNullsTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  03:17 min
[INFO] Finished at: 2023-05-11T05:38:07Z
[INFO] ------------------------------------------------------------------------

[razajafri]

I am running benchmarks on a different version of this implementation based on suggestions from here. Will post an update soon

[razajafri]

I have run some numbers on my local workstation using different implementations. The script just reads a parquet file with a list[int] column and casts it to a string. The casting in turn calls the mergeAndSetValidity method

Baseline	Ifelse	Binop	CPP Modification	Input Size
5,975 ms	7,025 ms	6,013 ms	5,936 ms	5.8 MiB
26,365 ms	35,874 ms	26,335 ms	26,284 ms	52.43 MiB

It looks like the modifications in this PR, listed as cpp modification above, has no performance degradation.

[razajafri]

/merge