BUG: df.melt cannot does not handle duplicate value_vars correctly

[sfc-gh-jkew]

Modin version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest released version of Modin.

• I have confirmed this bug exists on the main branch of Modin. (In order to do this you can follow this guide.)

Reproducible Example

import modin.pandas as pd
df = pd.DataFrame([[1, 2], [3, 4]], columns=["dupe", "dupe"])
df.melt()

Issue Description

Index.difference changed in pandas 2.2.x (pandas-dev/pandas#55113) and it now calls self.unique(). This means that the set of value_vars in dataframe.py:melt is not calculated correctly. Replacing Index.difference with Index.drop would do the right thing at that DataFrame API layer.

In /Ray/ this results in a ValueError: Length mismatch: Expected axis has 4 elements, new values have 2 elements
Other execution engines may be different.

I'll submit a PR for this shortly, I've fixed it in a fork for another engine, but I need to validate that the fix also works for Ray/Dask.

Expected Behavior

In [5]: df.melt()
Out[5]: 
  variable  value
0     dupe      1
1     dupe      3
2     dupe      2
3     dupe      4

Error Logs

RayTaskError(ValueError): ray::remote_exec_func() (pid=41269, ip=127.0.0.1)
  File "/Users/jkew/anaconda3/envs/snowpandas-dev-test/lib/python3.9/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 923, in apply_idx_objs
    return df.set_axis(idx, axis="index")
  File "/Users/jkew/anaconda3/envs/snowpandas-dev-test/lib/python3.9/site-packages/pandas/core/frame.py", line 5344, in set_axis
    return super().set_axis(labels, axis=axis, copy=copy)
  File "/Users/jkew/anaconda3/envs/snowpandas-dev-test/lib/python3.9/site-packages/pandas/core/generic.py", line 791, in set_axis
    return self._set_axis_nocheck(labels, axis, inplace=False, copy=copy)
  File "/Users/jkew/anaconda3/envs/snowpandas-dev-test/lib/python3.9/site-packages/pandas/core/generic.py", line 803, in _set_axis_nocheck
    setattr(obj, obj._get_axis_name(axis), labels)
  File "/Users/jkew/anaconda3/envs/snowpandas-dev-test/lib/python3.9/site-packages/pandas/core/generic.py", line 6310, in __setattr__
    return object.__setattr__(self, name, value)
  File "properties.pyx", line 69, in pandas._libs.properties.AxisProperty.__set__
  File "/Users/jkew/anaconda3/envs/snowpandas-dev-test/lib/python3.9/site-packages/pandas/core/generic.py", line 813, in _set_axis
    self._mgr.set_axis(axis, labels)
  File "/Users/jkew/anaconda3/envs/snowpandas-dev-test/lib/python3.9/site-packages/pandas/core/internals/managers.py", line 238, in set_axis
    self._validate_set_axis(axis, new_labels)
  File "/Users/jkew/anaconda3/envs/snowpandas-dev-test/lib/python3.9/site-packages/pandas/core/internals/base.py", line 98, in _validate_set_axis
    raise ValueError(
ValueError: Length mismatch: Expected axis has 4 elements, new values have 2 elements

During handling of the above exception, another exception occurred:

ray::remote_exec_func() (pid=41269, ip=127.0.0.1)
  File "/Users/jkew/anaconda3/envs/snowpandas-dev-test/lib/python3.9/site-packages/modin/core/execution/ray/common/deferred_execution.py", line 810, in remote_exec_func
    obj = remote_executor.exec_func(fn, obj, flat_args, flat_kwargs)
  File "/Users/jkew/anaconda3/envs/snowpandas-dev-test/lib/python3.9/site-packages/modin/core/execution/ray/common/deferred_execution.py", line 605, in exec_func
    raise err
  File "/Users/jkew/anaconda3/envs/snowpandas-dev-test/lib/python3.9/site-packages/modin/core/execution/ray/common/deferred_execution.py", line 598, in exec_func
    return fn(obj.copy(), *args, **kwargs)
  File "/Users/jkew/anaconda3/envs/snowpandas-dev-test/lib/python3.9/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 923, in apply_idx_objs
    return df.set_axis(idx, axis="index")
  File "/Users/jkew/anaconda3/envs/snowpandas-dev-test/lib/python3.9/site-packages/pandas/core/frame.py", line 5344, in set_axis
    return super().set_axis(labels, axis=axis, copy=copy)
  File "/Users/jkew/anaconda3/envs/snowpandas-dev-test/lib/python3.9/site-packages/pandas/core/generic.py", line 791, in set_axis
    return self._set_axis_nocheck(labels, axis, inplace=False, copy=copy)
  File "/Users/jkew/anaconda3/envs/snowpandas-dev-test/lib/python3.9/site-packages/pandas/core/generic.py", line 803, in _set_axis_nocheck
    setattr(obj, obj._get_axis_name(axis), labels)
  File "/Users/jkew/anaconda3/envs/snowpandas-dev-test/lib/python3.9/site-packages/pandas/core/generic.py", line 6310, in __setattr__
    return object.__setattr__(self, name, value)
  File "properties.pyx", line 69, in pandas._libs.properties.AxisProperty.__set__
  File "/Users/jkew/anaconda3/envs/snowpandas-dev-test/lib/python3.9/site-packages/pandas/core/generic.py", line 813, in _set_axis
    self._mgr.set_axis(axis, labels)
  File "/Users/jkew/anaconda3/envs/snowpandas-dev-test/lib/python3.9/site-packages/pandas/core/internals/managers.py", line 238, in set_axis
    self._validate_set_axis(axis, new_labels)
  File "/Users/jkew/anaconda3/envs/snowpandas-dev-test/lib/python3.9/site-packages/pandas/core/internals/base.py", line 98, in _validate_set_axis
    raise ValueError(
ValueError: Length mismatch: Expected axis has 4 elements, new values have 2 elements

Installed Versions

INSTALLED VERSIONS

commit : 6d64e08
python : 3.9.18.final.0
python-bits : 64
OS : Darwin
OS-release : 23.3.0
Version : Darwin Kernel Version 23.3.0: Wed Dec 20 21:31:00 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T6020
machine : arm64
processor : arm
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

Modin dependencies

modin : 0.29.0
ray : 2.10.0
dask : None
distributed : None
hdk : None

pandas dependencies

pandas : 2.2.1
numpy : 1.26.4
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : 68.2.2
pip : 23.3.1
Cython : None
pytest : 7.4.0
hypothesis : None
sphinx : 5.0.2
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.3
IPython : 8.18.1
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
bottleneck : 1.3.7
dataframe-api-compat : None
fastparquet : None
fsspec : 2023.12.2
gcsfs : None
matplotlib : 3.8.4
numba : None
numexpr : 2.8.7
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 15.0.1
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : 2023.12.2
scipy : 1.12.0
sqlalchemy : None
tables : None
tabulate : 0.9.0
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None