Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: add openssl add-on test #6274

Merged
merged 1 commit into from
Jul 5, 2016
Merged

Conversation

@bnoordhuis bnoordhuis added test Issues and PRs related to the tests. openssl Issues and PRs related to the OpenSSL dependency. labels Apr 19, 2016
@jasnell
Copy link
Member

jasnell commented Apr 19, 2016

@bnoordhuis
Copy link
Member Author

It's the age-old problem of the windows build not exporting openssl symbols. I'm going to tinker with deps/openssl/openssl/util/mkdef.pl and see if I can make the .def part of the build somehow.

@bnoordhuis
Copy link
Member Author

I haven't had much luck on Windows so far. I added a ExportNamedFunctions directive to common.gypi and I see a <Tool ExportNamedFunctions="RAND_bytes;RAND_poll" Name="VCLibrarianTool"/> in node.vcproj but building the add-on still fails with missing symbol errors:

binding.obj : error LNK2001: unresolved external symbol RAND_bytes [c:\workspace\node-test-binary-windows\RUN_SUBSET\1\VS_VERSION\vs2015\label\win2012r2\test\addons\openssl-binding\build\binding.vcxproj]
binding.obj : error LNK2001: unresolved external symbol RAND_poll [c:\workspace\node-test-binary-windows\RUN_SUBSET\1\VS_VERSION\vs2015\label\win2012r2\test\addons\openssl-binding\build\binding.vcxproj]
c:\workspace\node-test-binary-windows\RUN_SUBSET\1\VS_VERSION\vs2015\label\win2012r2\test\addons\openssl-binding\build\Release\binding.node : fatal error LNK1120: 2 unresolved externals [c:\workspace\node-test-binary-windows\RUN_SUBSET\1\VS_VERSION\vs2015\label\win2012r2\test\addons\openssl-binding\build\binding.vcxproj]

@nodejs/platform-windows Ideas?

@nodejs/build Is there a way to extract build artifacts from the buildbots? The node.lib and node.exp files in particular?

@rvagg
Copy link
Member

rvagg commented May 10, 2016

@nodejs/build Is there a way to extract build artifacts from the buildbots? The node.lib and node.exp files in particular?

@bnoordhuis what do you need here exactly? You want the build artifacts after the jobs are finished or beforehand? Are you wanting to inspect artifacts to debug the job? If that's what you're after we have to instruct Jenkins what files to grab using globs

@bnoordhuis
Copy link
Member Author

You want the build artifacts after the jobs are finished or beforehand?

After. I'd like the node.lib and the node.exp files that are built. The binary itself might come in handy, too.

@joaocgreis
Copy link
Member

@bnoordhuis do you want the files to download so you can check them locally or do you want them to be present in the test machines?

There is no automated way to get the files from the CI slaves. If you want the files let me know exactly what to build (this PR head?) and I can email them to you.

If you want the files on the test machines, you can try https://ci.nodejs.org/job/node-test-commit-windows/ , that builds and tests in the same job. If you can make that work, I'll make it work with the fanned job.

@bnoordhuis
Copy link
Member Author

Thanks, I was hoping I could download the files somewhere, but I'll tweak this PR some more and ping you when I need them.

@bnoordhuis
Copy link
Member Author

bnoordhuis commented May 11, 2016

I may have inadvertently broken the Windows CI with my latest update, all the bots are hanging at the "Building add-ons" step now... https://ci.nodejs.org/job/node-test-binary-windows/2077/

@joaocgreis
Copy link
Member

@bnoordhuis when that happens, please abort the job, it's no big issue. I tried to build this locally, and node.exe is a DLL.

@bnoordhuis
Copy link
Member Author

You mean it's not an executable but a library? Le sigh, it must be something that gyp is doing. I'm going to try it one more time in case the CI was in a funk yesterday: https://ci.nodejs.org/job/node-test-pull-request/2607/

@kurtextrem
Copy link

kurtextrem commented Jun 13, 2016

Any news?
This is a blocker for https://github.com/alexhultman/uWebSockets/issues/72

@ghost
Copy link

ghost commented Jun 20, 2016

I took a peek at this and it seems to be two things: OpenSSL and zlib. Both of these are missing from the symbol table (dumpbin /EXPORTS node.exe). I ran the vcbuild.bat which generated the VS solution and then I just looked at the project and couldn't really see any reference to any .def file. So I just went to Project -> Properties -> Linker -> Input -> Module Definition File and put the path to the zlib.def file as a test and built the program. Now I get all the zlib functions exported and I can see there are inflate and such functions in node.lib. So maybe you could just focus on generating one .def file with both openssl and zlib functions and just pass that one file as /DEF:filename? V8 and uv are already exported but I couldn't see any .def file so maybe you are using __declspec(dllexport) or something - I don't know but passing the zlib.def worked and I can send you my node.lib and node.exe if you want to compare the symbol table.

@ghost
Copy link

ghost commented Jul 3, 2016

What is the status on this? If things get way to complex, wouldn't it be pretty simple to manually run the mkdef perl script offline and commit as one fix .def file? Or just write a new script that just scrapes functions from the documentation. With /DEF:file.def /FORCE:UNRESOLVED it won't fail if the def file contains symbols that are not in the binary, they will be ignored, so even if you configure Node.js in some way it should still work with one single fix .def file.

@ghost
Copy link

ghost commented Jul 4, 2016

I just built a node.exe that exports all OpenSSL symbols and runs as an executable. The size is about 700kb larger than normal and RAND_poll is listed when dumpbin / exports. I did not use /FORCE:UNDEFINED, only /DEF. I ended up writing a "script" that exports the symbols from openssl.lib (I couldn't scrape the symbols from the documentation because some of the symbols were just macros and thus not real symbols).

I could commit my node.def file and add instructions on how to generate it (including code). Basically I used nm -s openssl.lib and filtered mangled non-C functions out. I also have a dump from dumpbin if you want to see.

@bnoordhuis
Copy link
Member Author

I just built a node.exe that exports all OpenSSL symbols

My plan was to restrict what we export to just public symbols. Right now we export everything on UNIX platforms.

@ghost
Copy link

ghost commented Jul 4, 2016

You could do a combination where you scrape the documentation + limit the symbols to what is in openssl.lib. That way macros are not added and only the documented public symbols are added. I have a tool that can scrape all the public symbols and also limit them to what is in openssl.lib.

To get the /DEF option added to the VC++ node project I did this in node.gyp:

'VCLinkerTool': {
  'ModuleDefinitionFile': 'node.def'
}

As an addition after the 'VCManifestTool' under 'msvs_settings'. It shows up in Visual Studio and I'm building it right now.

@bnoordhuis
Copy link
Member Author

Unrelated flake, fortunately. Barring nays from other reviewers I'll merge this tomorrow.

Export symbols from the bundled openssl for add-ons to link against.

Fixes: nodejs/node-v0.x-archive#4051
PR-URL: nodejs#6274
Reviewed-By: James M Snell <jasnell@gmail.com>
@bnoordhuis bnoordhuis closed this Jul 5, 2016
@bnoordhuis bnoordhuis deleted the openssl-addon branch July 5, 2016 14:13
@bnoordhuis bnoordhuis merged commit b4d4fd9 into nodejs:master Jul 5, 2016
Fishrock123 pushed a commit that referenced this pull request Jul 6, 2016
Export symbols from the bundled openssl for add-ons to link against.

Fixes: nodejs/node-v0.x-archive#4051
PR-URL: #6274
Reviewed-By: James M Snell <jasnell@gmail.com>
@Fishrock123 Fishrock123 mentioned this pull request Jul 6, 2016
@ghost
Copy link

ghost commented Jul 6, 2016

@bnoordhuis There's a bug in the mkssldef.py script. I'm missing SSL_set_fd and two others. They are present in the ssleay.num file but have a different format of the rightmost column. Most symbols are present now in 6.3.0 though.

@ghost
Copy link

ghost commented Jul 6, 2016

This is my output when compiling on Windows with 6.3.0:

Server.obj : error LNK2001: unresolved external symbol SSL_CTX_use_PrivateKey_file [C:\Users\alex\uWebSockets\nodejs\dist\build\uws.vcxproj]
Server.obj : error LNK2001: unresolved external symbol SSL_set_fd [C:\Users\alex\uWebSockets\nodejs\dist\build\uws.vcxproj]
Server.obj : error LNK2001: unresolved external symbol inflateInit2_ [C:\Users\alex\uWebSockets\nodejs\dist\build\uws.vcxproj]
Server.obj : error LNK2001: unresolved external symbol deflateInit2_ [C:\Users\alex\uWebSockets\nodejs\dist\build\uws.vcxproj]
Server.obj : error LNK2001: unresolved external symbol deflateReset [C:\Users\alex\uWebSockets\nodejs\dist\build\uws.vcxproj]
Server.obj : error LNK2001: unresolved external symbol SSL_CTX_use_certificate_chain_file [C:\Users\alex\uWebSockets\nodejs\dist\build\uws.vcxproj]
Server.obj : error LNK2001: unresolved external symbol deflateEnd [C:\Users\alex\uWebSockets\nodejs\dist\build\uws.vcxproj]
Server.obj : error LNK2001: unresolved external symbol deflate [C:\Users\alex\uWebSockets\nodejs\dist\build\uws.vcxproj]
WebSocket.obj : error LNK2001: unresolved external symbol inflateEnd [C:\Users\alex\uWebSockets\nodejs\dist\build\uws.vcxproj]
WebSocket.obj : error LNK2001: unresolved external symbol inflate [C:\Users\alex\uWebSockets\nodejs\dist\build\uws.vcxproj]
C:\Users\alex\uWebSockets\nodejs\dist\build\Release\uws.node : fatal error LNK1120: 10 unresolved externals [C:\Users\alex\uWebSockets\nodejs\dist\build\uws.vcxproj]

@MylesBorins
Copy link
Contributor

@bnoordhuis this does not land cleanly. It will require a backport.

@bnoordhuis
Copy link
Member Author

#7676 - TBD if it should land.

@Fishrock123
Copy link
Contributor

I don't see an actual commit link here so I dug it up: b4d4fd9

bnoordhuis added a commit to bnoordhuis/io.js that referenced this pull request Oct 18, 2016
Export symbols from the bundled openssl for add-ons to link against.

Fixes: nodejs/node-v0.x-archive#4051
PR-URL: nodejs#6274
Reviewed-By: James M Snell <jasnell@gmail.com>
MylesBorins pushed a commit that referenced this pull request Nov 14, 2016
Export symbols from the bundled openssl for add-ons to link against.

Fixes: nodejs/node-v0.x-archive#4051
PR-URL: #6274
Reviewed-By: James M Snell <jasnell@gmail.com>
@MylesBorins MylesBorins mentioned this pull request Nov 22, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
openssl Issues and PRs related to the OpenSSL dependency. test Issues and PRs related to the tests.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants