UTF-8 in/out

[shrektan]

Fixes #103

Hi, first of all, thanks for writing this fantasic package.

As a Chinese user, I often need to read / write Excel sheets contain Chinese letters on Windows, where the encoding issue is a huge headache. According to my test, there're at least two known issues::

1. When the comment contains non-ASCII letters, the comment will be garbaged
2. When the Excel table's name contains non-ASCII letters, the name will be garbaged

These two issues are difficult to avoid because by default, Excel will create locale language names for the comments (the author name) and Excel table's names.

And these two issues cause more trouble when I use such an Excel as a template. Resaving the excel object modified by openxlsx will malform the Excel files. I mean, when I open the Excel files that write by openxlsx, Excel will alarm that some contents are illegal.

The root cause of this is that R can't use UTF-8 as the native encoding on Windows as the time of writing (luckily, this may be available in the near future). So, the only reliable cure for this headache for now is to use UTF-8 encoded strings whenever we can.

This PR tries to do the two things:

1. Ensure all the values of C++ functions that return a character vector be marked as UTF-8 encoded.
2. Defined a helper function readUTF8() to ensure that we read the xml as UTF-8 encoded.

Despite it looks like lots of lines has been changed, I believe the PR is simple, if you read the commit in sequence.

Thanks!

TODO

• more manual tests on write/read: non-ASCII range names, sheet names, table names, formula
• add unit tests
• add NEWS

---

Think & Future

We should check all the C++ function's that used in R code and change the argument type from std::string to SEXP. Only so can we convert the strings to UTF-8 encoded later on the Cpp side, thus guarantee our policy UTF-8 in / out in everywhere. The reason is that we can only know the encoding info in SEXP but not std::string. Another possible solution is to see if there's some configs in Rcpp so that it can ensure the auto-SEXP-to-std::string ends up with a UTF-8 encoded string.

Anyway, I think we'd better not make this PR further complicate and leave this task to the future PRs.

[codecov]

Codecov Report

Merging #118 (2b6b5ba) into master (44014fa) will increase coverage by 2.64%.
The diff coverage is 93.18%.

@@            Coverage Diff             @@
##           master     #118      +/-   ##
==========================================
+ Coverage   59.68%   62.33%   +2.64%     
==========================================
  Files          30       30              
  Lines        8729     8740      +11     
==========================================
+ Hits         5210     5448     +238     
+ Misses       3519     3292     -227     

Impacted Files	Coverage Δ
R/writeData.R	77.41% <ø> (+11.17%)	⬆️
R/WorkbookClass.R	53.63% <66.66%> (+2.57%)	⬆️
src/helper_functions.cpp	74.09% <88.88%> (+0.84%)	⬆️
R/loadWorkbook.R	82.24% <92.30%> (+11.62%)	⬆️
R/helperFunctions.R	82.35% <100.00%> (+2.74%)	⬆️
R/readWorkbook.R	86.24% <100.00%> (ø)
R/wrappers.R	49.32% <100.00%> (+0.28%)	⬆️
src/load_workbook.cpp	94.64% <100.00%> (+0.21%)	⬆️
src/write_file.cpp	90.07% <100.00%> (+1.49%)	⬆️
... and 12 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 44014fa...2b6b5ba. Read the comment docs.

[ycphs]

I think the function would also fit into the helper.R file

[ycphs]

I am currently planning to replace all the readLines with an equivalent in cpp. That should boost the performance and the UTF8 encoding can be solved at the same time.

[ycphs]

Thank you for your work. This will improve the package a lot.