BUG: Stata value label limits incorrect

[jmcoglianese]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

Here's an example writing a Stata file with value labels that raises an error: 

import string
import random
import pandas as pd
df = pd.DataFrame(range(65534), columns = ["col"])
value_labels = {"col": {i: ''.join(random.choices(string.ascii_letters, k=21)) for i in range(65534)}}
df.to_stata("label_test.dta", value_labels=value_labels)

```python-traceback
>>> df.to_stata("label_test.dta", value_labels=value_labels)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".venv/lib/python3.12/site-packages/pandas/core/frame.py", line 2893, in to_stata
    writer = statawriter(
             ^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pandas/io/stata.py", line 2365, in __init__
    self._prepare_pandas(data)
  File ".venv/lib/python3.12/site-packages/pandas/io/stata.py", line 2616, in _prepare_pandas
    non_cat_value_labels = self._prepare_non_cat_value_labels(data)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pandas/io/stata.py", line 2415, in _prepare_non_cat_value_labels
    svl = StataNonCatValueLabel(colname, labels, self._encoding)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pandas/io/stata.py", line 820, in __init__
    self._prepare_value_labels()
  File ".venv/lib/python3.12/site-packages/pandas/io/stata.py", line 727, in _prepare_value_labels
    raise ValueError(
ValueError: Stata value labels for a single variable must have a combined length less than 32,000 characters.

Here's an example in Stata that constructs the same value label (65,534 entries with 21 characters each) but does not produce an error:

clear all
set obs 65534
gen x = _n

* Set up label
label define test_lbl 0 ""

* Loop over rows and create unique label for each value
forvalues i = 1/`=_N' {
	* Add a unique length-21 string label
	label define test_lbl `i' "`: di %20.19f runiform()'", add
}

label values x test_lbl

save "label_test.dta", replace

clear all

use "label_test.dta", clear
forvalues i = 1/65534 {
	assert strlen("`: label test_lbl `i', strict'") == 21
}

### Issue Description

pandas requires that the _combined_ length of the value label not exceed 32,000 characters, but the limit in Stata is actually that _each_ value label can be up to 32,000 characters, and you can have up to 65,536 value labels (see the "label" section on this page: https://www.stata.com/products/detailed-size-limits/#label). 

### Expected Behavior

The expected behavior is for pandas to only raise an error if an individual label exceeds 32,000 characters, or if there are more than 65,536 labels. If neither of these limits are exceeded, then pandas should write out the value label, even if the combined length is more than 32,000 characters. 

### Installed Versions

INSTALLED VERSIONS

commit : d9cdd2e
python : 3.12.0.final.0
python-bits : 64
OS : Darwin
OS-release : 21.6.0
Version : Darwin Kernel Version 21.6.0: Wed Apr 24 06:02:02 PDT 2024; root:xnu-8020.240.18.708.4~1/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.2.2
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.9.0.post0
setuptools : 75.1.0
pip : 24.2
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 5.2.2
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 8.26.0
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
bottleneck : 1.3.7
dataframe-api-compat : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : 2.8.7
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : 2.0.32
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

[jbrockmendel]

Cc @bashtage

[bashtage]

From the stata documentation:

           The use of a 5-byte integer is awfully odd.  In the previous 117
           .dta format, the (v,o) values were written as two 4-byte values.
           This accommodated Stata's then-current maximum of roughly 32,000
           variables and roughly 2,000,000,000 observations.


           In the 118 format used in modern Stata when there are 32,767
           variables or less, the (v,o) values are written as a 2-byte value
           followed by a 6-byte value to allow for more observations.


           Since then, Stata has learned to deal with more variables and v no
           longer can be stored in just 2 bytes in the 119 format.  Stata
           allows up to 100,000 variables and up to 1 terabyte observations.
           This combination of numbers can be stored in a 3-byte integer and
           a 5-byte integer.

The docs make it sound like 32,000 is the limit for <= 118, and only 119 can do more. What format are you outputting to in Stata?

[bashtage]

Rereading the docs it does seem like the check might be implemented incorrectly, and it should be 32,000 variables, not 32,000 observations.