Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revise tools scripts to be python3 compatible on win32 and enable github ci for win32 #4856

Merged

Conversation

lygstate
Copy link
Contributor

@lygstate lygstate commented Dec 7, 2021

We introduce setup_stdio function to setup stdout/stderr properly.
For python <-> python pipe, we always use 'utf8'/'ignore' encoding for not lost
characters.
For tty <-> python, we using native encoding with xmlcharrefreplace to encode, to
preserve maximal information.
For python <-> native program, we use naive encoding with 'ignore' to not cause error

update_exclude_list with binary mode so that on win32 would not generate \r\n

run-test-suite.py: Handling skiplist properly on win32

Fixes #4854

Fixes test262-harness.py complain cannot use a string pattern on a bytes-like object with running test262 with python3

For reading/writing to file, we use 'utf8' /'ignore' encoding for not lost characters.
For decoding from process stdout, using native encoding with decoding error ignored for not lost data.
Execute commands also ignore errors
Fixes #4853

Fixes running test262-esnext failed with installed python3.9 on win32 with space in path
Fixes #4852

support both / \ in --test262-test-list arg

On win32.
python tools/run-tests.py --test262-es2015=update --test262-test-list=built-ins/decodeURI/
python tools/run-tests.py --test262-es2015=update --test262-test-list=built-ins\decodeURI
should be both valid,
currently only --test262-test-list=built-ins\decodeURI\ are valid.

Support snapshot-tests-skiplist.txt on win32 by use os.path.normpath

Guard run-tests.py with timer.

All run-tests.py are finished in 30 minutes in normal situation.
May increase the timeout future.

wait JERRY_CHECK_TIMEOUT


Move Windows CI to github actions

Convert run-debugger-test.sh to run-debugger-test.py

After this change, run-debugger-test.py could running on Win32 and OSX

Define TERM colors for win32 properly


flush stderr.write stdout.write

On CI, the stderr are redirect to stdout, and if we don't flush stderr and stdout,
The output from stderr/stdout would out of sync.

Testing new Date(-8640000000000000) and fixes date for win32
So that the CI can passed

Fixes compiling warnings/errors for mingw/gcc

Fixes: #5107

CI for mingw also setup at the same time

JerryScript-DCO-1.0-Signed-off-by: Yonggang Luo luoyonggang@gmail.com

@robertsipka @LaszloLango

Currently there is no full CI for windows, and the Date is not implement valid win32,
by enabling the Windows ci for catching those

@lygstate lygstate force-pushed the python3-stdio-encoding branch from e401d3d to 4fd4566 Compare November 14, 2024 19:54
@lygstate lygstate changed the title Revise tools scripts to be python3 compatible Revise tools scripts to be python3 compatible on win32 Nov 14, 2024
@lygstate lygstate force-pushed the python3-stdio-encoding branch from 4fd4566 to d59a19a Compare November 14, 2024 20:05
@lygstate lygstate changed the title Revise tools scripts to be python3 compatible on win32 Revise tools scripts to be python3 compatible on win32 and enable github ci for win32 Nov 14, 2024
Copy link
Contributor

@LaszloLango LaszloLango left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, great PR in general. Just a few comments and questions from my side.

tools/runners/util.py Outdated Show resolved Hide resolved
tools/run-tests.py Outdated Show resolved Hide resolved
tools/run-tests.py Outdated Show resolved Hide resolved
.github/workflows/gh-actions.yml Show resolved Hide resolved
tools/run-tests.py Outdated Show resolved Hide resolved
tools/runners/run-test-suite-test262.py Outdated Show resolved Hide resolved
tools/runners/util.py Show resolved Hide resolved
@lygstate lygstate force-pushed the python3-stdio-encoding branch 10 times, most recently from 0788b03 to 70abe3d Compare November 20, 2024 13:26
@lygstate
Copy link
Contributor Author

@LaszloLango ready for take a look again

@lygstate lygstate requested a review from LaszloLango November 25, 2024 19:48
@lygstate lygstate force-pushed the python3-stdio-encoding branch from 70abe3d to d203264 Compare November 25, 2024 21:54
@lygstate
Copy link
Contributor Author

rebased

tools/run-tests.py Outdated Show resolved Hide resolved
tools/runners/run-test-suite-test262.py Outdated Show resolved Hide resolved
tools/run-tests.py Outdated Show resolved Hide resolved
@lygstate
Copy link
Contributor Author

lygstate commented Nov 26, 2024

@LaszloLango errors="ignore" For python <-> native program, we use default encoding with error='ignore' to not lost data

This is for win32, for linux, no data will lost
as the native encoding is gbk and not every utf8 character will be print into it, and python failure will cause test failure that misleading

@lygstate lygstate force-pushed the python3-stdio-encoding branch 2 times, most recently from 4b7044a to fe0a264 Compare November 26, 2024 13:31
@lygstate lygstate requested a review from LaszloLango November 26, 2024 13:37
@lygstate lygstate requested a review from LaszloLango November 26, 2024 16:28
@lygstate lygstate force-pushed the python3-stdio-encoding branch 2 times, most recently from beb567e to df9bd4c Compare November 27, 2024 06:03
@lygstate
Copy link
Contributor Author

lygstate commented Dec 2, 2024

@LaszloLango ping for take a look, I'd like it's merged before Release 3.0.0

jerry-core/CMakeLists.txt Show resolved Hide resolved
tools/run-tests.py Show resolved Hide resolved
tools/run-tests.py Show resolved Hide resolved
@LaszloLango
Copy link
Contributor

@lygstate, Can we move out the valgrind update to a separate PR? Is it necessary in this patch? What valgrind version are you updating? Did you do any modification in those headers beyond the update? Are there any manual changes in them?

@lygstate
Copy link
Contributor Author

lygstate commented Dec 11, 2024

@lygstate, Can we move out the valgrind update to a separate PR? Is it necessary in this patch? What valgrind version are you updating? Did you do any modification in those headers beyond the update? Are there any manual changes in them?

OK, only affect mingw CI, let's me drop it first, done

We introduce setup_stdio function to setup stdout/stderr properly.
For python <-> python pipe, we always use 'utf8'/'ignore' encoding for not lost
characters.
For tty <-> python, we using native encoding with xmlcharrefreplace to encode, to
preserve maximal information.
For python <-> native program, we use naive encoding with 'ignore' to not cause error

update_exclude_list with binary mode so that on win32 would not generate \r\n

run-test-suite.py: Handling skiplist properly on win32

Fixes jerryscript-project#4854

Fixes test262-harness.py complain cannot use a string pattern on a bytes-like object with running test262 with python3

For reading/writing to file, we use 'utf8' /'ignore' encoding for not lost characters.
For decoding from process stdout, using native encoding with decoding error ignored for not lost data.
Execute commands also ignore errors
Fixes jerryscript-project#4853

Fixes running test262-esnext failed with installed python3.9 on win32 with space in path
Fixes jerryscript-project#4852

support both / \ in --test262-test-list arg

On win32.
python tools/run-tests.py  --test262-es2015=update --test262-test-list=built-ins/decodeURI/
python tools/run-tests.py  --test262-es2015=update --test262-test-list=built-ins\decodeURI\
should be both valid,
currently only --test262-test-list=built-ins\decodeURI\ are valid.

Support snapshot-tests-skiplist.txt on win32 by use os.path.normpath

Guard run-tests.py with timer.

All run-tests.py are finished in 30 minutes in normal situation.
May change the timeout by command line option

Move Windows CI to github actions
Define TERM colors for win32 properly

flush stderr.write stdout.write

On CI, the stderr are redirect to stdout, and if we don't flush stderr and stdout,
The output from stderr/stdout would out of sync.

`Testing new Date(-8640000000000000) and fixes date for win32`
So that the CI can passed

if sys.version_info.major >= 3: remove, as we do not support python2 anymore

Fixes compiling warnings/errors for mingw/gcc

JerryScript-DCO-1.0-Signed-off-by: Yonggang Luo luoyonggang@gmail.com
@lygstate lygstate force-pushed the python3-stdio-encoding branch from b25d3b7 to 535a5eb Compare December 11, 2024 09:30
Copy link
Member

@zherczeg zherczeg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zherczeg zherczeg merged commit d2d30df into jerryscript-project:master Dec 17, 2024
30 checks passed
@lygstate lygstate deleted the python3-stdio-encoding branch December 17, 2024 10:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants