SSL/TLS support for cosocket API?

[Doridian]

So far all I needed to do SSL with has been HTTPS so I settled with using ngx.location.capture and internal proxy locations.
But now I have come accross an issue.
I need to send E-Mail over a server which requires TLS (Amazon SES), which seems undoable at the moment as LuaSec does not work (and prolly would ruin cosocket's asynchronity anyway).
Is there any possible way I could establish communication with the server?

[agentzh]

Hello!

On Sat, Nov 17, 2012 at 4:19 AM, Mark Dietzer notifications@github.com wrote:

So far all I needed to do SSL with has been HTTPS so I settled with using ngx.location.capture and internal proxy locations.
But now I have come accross an issue.
I need to send E-Mail over a server which requires TLS (Amazon SES), which seems undoable at the moment as LuaSec does not work (and prolly would ruin cosocket's asynchronity anyway).
Is there any possible way I could establish communication with the server?

This is still a TODO. I hope I can implement this feature as soon as
possible :) This is my next big item for ngx_lua. I'll keep you posted
:)

Thanks!
-agentzh

[ghost]

What about just providing a Lua interface to the OpenSSL functions in the core (assuming they're there) and then creating the handshaking logic in a resty module like the others?

This it would then be possible to create potentially any kind of encrypted system (I'm planning on creating an SSH one for instance).

[Doridian]

Theres a reason that even Java provides native implementations for SSL/TCP connections.
That reason is pure speed.
You just cannot implement SSL at the same speed in C versus Lua.

[ghost]

Of course. The computationally heavy part would be done in C - using the OpenSSL library (assuming that's the one that's compiled in with the server, which is usually the case).

You'd have a bunch of interfaces like:

ngx.openssl.func()

or even just

openssl.func()

The network logic of dealing with the back and forth communications (which is much less computationally heavy) would be done using Lua/cosockets, like it is in resty_memcached, resty_mysql etc.

[Doridian]

I honestly wish resty_mysql and especially resty_redis were C modules and not Lua ones...
Would manage to get out a few more percent I am sure.
But that's a different topic.

[bakins]

I honestly wish resty_mysql and especially resty_redis were C modules and not Lua ones...
Would manage to get out a few more percent I am sure.

Not really. Luajit is very, very fast. The overhead of "switching" between Lua and C is often more than any gains. Actually read the implementation of the mysql and redis clients. They are mostly string manipulation and Luajit is often faster than C at this.

[ghost]

If they were in pure C, then they may be a bit faster (see Brian's commen), but they may not, and there are many more issues and potential sources of bugs with creating them in C.

What ngx.cosocket provides, though, is a "safe" platform from a networking/socket level (which is usually the most difficult part of it to get right), and then just leaves it up to the Lua code to deal with the text-parsing side of things (which is generally much, much easier, and more programmers are comfortable doing it).

This makes the platform much more developer-friendly, and provides a greater speed of development. To show this in real terms, just look at how many modules have been created to connect to back-ends since it was first released (e.g. Redis, Mongo, HTTP, MySQL, Memcached). They all popped up in a matter of months (I know agentzh/chaoslawful wrote several of them, but the dev speed is much higher using this method).

By providing a full OpenSSL interface, it would similarly be possible to create SSL, TLS, HTTPS, SSH, SFTP, FTPS (and any others that use encryption). Sure the implementation might be very slightly slower than a completely native TSL implementation, but it would be much more flexible and the development of libraries that use encryption would probably happen much quicker than if the only implementations were native ones.

[agentzh]

Hello!

On Wed, Nov 21, 2012 at 3:01 PM, simpl.it notifications@github.com wrote:

What about just providing a Lua interface to the OpenSSL functions in the core (assuming they're there) and then creating the handshaking logic in a resty module like the others?

We've already started doing that in the lua-resty-string library:

https://github.com/agentzh/lua-resty-string

We're exposing quite a few OpenSSL's C API via LuaJIT 2.0's FFI mechanism :)

But it may still worthwhile to just expose the existing SSL C API in
the Nginx core to the Lua land. I'm not 100% sure right now :)

This it would then be possible to create potentially any kind of encrypted system (I'm planning on creating an SSH one for instance).

Agreed. I've also been thinking about creating the nonblocking
lua-resty-ssh client library for SSH servers :D

Thanks!
-agentzh

[ghost]

I think FFI might be a good way to get things started on that. It has some very interesting possibilites for relatively easy integration of other third-party libraries, too. Have you used it for anything much, yet?

[agentzh]

Hello!

On Wed, Nov 21, 2012 at 5:53 PM, simpl.it notifications@github.com wrote:

I think FFI might be a good way to get things started on that. It has some
very interesting possibilites for relatively easy integration of other
third-party libraries, too. Have you used it for anything much, yet?

Until now, just in lua-resty-string :)

But I'm going to port all of the existing Nginx API for Lua provided by
ngx_lua over to FFI very soon, in order to take full advantage of the
LuaJIT 2.0's JIT compiler (because CFunction calls will abort the traces
and make LuaJIT fall back to the interpreter mode).

When the FFI move in ngx_lua happens, I think I'd preserve the old
CFunction-based API to keep compatibility with the standard Lua 5.1.x
interpreters. I may use C macros to toggle these two separate code paths in
ngx_lua.

Best regards,
-agentzh

[ghost]

If I were you, I'd consider splitting them into two separate branches. Converting everything to FFI is a pretty major change, and considering the benefits of JIT, I think it would be less headache in the long-run to make a clean break.

Both versions can be supported, but the focus can be on the JIT-only implementation and the C-only implementation could be considered a legacy version.

I think doing it this way would be less of a headache in the long-run.

I can envisage the importing of a large number of third-party C libraries using FFI, and there could even be an organized platform for loading ngx_lua-compatible modules (perhaps a bit like an ngx_lua equivalent to luarocks - ngxluarocks?).

[agentzh]

Hello!

On Wed, Nov 21, 2012 at 7:05 PM, simpl.it notifications@github.com wrote:

If I were you, I'd consider splitting them into two separate branches. Converting everything to FFI is a pretty major change, and considering the benefits of JIT, I think it would be less headache in the long-run to make a clean break.

Both versions can be supported, but the focus can be on the JIT-only implementation and the C-only implementation could be considered a legacy version.

I think doing it this way would be less of a headache in the long-run.

I must say it makes a lot of sense for future development :) I surely
don't want to write two versions of implementation everytime I try to
add something new to ngx_lua :)

Forking the code base could be a better approach and I'll think more
about it :) Before it really happens, I'll experiment the FFI move in
a git branch first :)

I can envisage the importing of a large number of third-party C libraries using FFI, and there could even be an organized platform for loading ngx_lua-compatible modules (perhaps a bit like an ngx_lua equivalent to luarocks - ngxluarocks?).

I'm glad that we're thinking along the same line ;) I've indeed been
thinking about launching a luaresty.org site hosting ngx_lua
compatible Lua libraries, similar to CPAN and LuaRocks :)

Best regards,
-agentzh

[ghost]

luaresty.org sounds like a good idea. It might be worth considering organizing everything around openresty.org, too. I think if everything's in one place, it might make things easier for (some) people.

Wrt the module system, you may want to think about whether it is worth considering adding DSO support to the lua core, so that Lua modules that use FFI could potentially load the shared library on the fly (or at least as part of lua_init). I think when you're considering Nginx largely as a webserver with a few 'bonus' features, it's probably just as well to not include DSO, but if you're envisaging ngx_lua to being a complete replacement for PHP/Python etc (which I am long-term), then I think it makes more sense to not require that all objects be compiled (or at least linked) from the beginning.

It also may be worth considering a interface for FFI-loaded modules that provide may assist with e.g. memory management with libraries that allow the passing of one or several memory-allocation functions (e.g. by saving things like a pointer to the current request to a global variable that any FFI-loaded module could access).

[agentzh]

Hello!

On Wed, Nov 21, 2012 at 8:00 PM, simpl.it notifications@github.com wrote:

luaresty.org sounds like a good idea. It might be worth considering organizing everything around openresty.org, too. I think if everything's in one place, it might make things easier for (some) people.

Maybe. We'll see :)

Wrt the module system, you may want to think about whether it is worth considering adding DSO support to the lua core, so that Lua modules that use FFI could potentially load the shared library on the fly (or at least as part of lua_init). I think when you're considering Nginx largely as a webserver with a few 'bonus' features, it's probably just as well to not include DSO, but if you're envisaging ngx_lua to being a complete replacement for PHP/Python etc (which I am long-term), then I think it makes more sense to not require that all objects be compiled (or at least linked) from the beginning.

The LuaJIT FFI can already load .so on-the-fly. Please check out the
ffi.load() API:

http://luajit.org/ext_ffi_api.html

Every Lua module can just look for .so/.dylib/.dll files in the search
paths specified by package.cpath and load it via ffi.load to the Lua
land on demand ;)

It also may be worth considering a interface for FFI-loaded modules that provide may assist with e.g. memory management with libraries that allow the passing of one or several memory-allocation functions (e.g. by saving things like a pointer to the current request to a global variable that any FFI-loaded module could access).

Maybe.

But accessing the raw ngx_http_request_t struct from within FFI
directly is a bad idea because the memory layout of ngx_http_request_t
may change upon different ./configure options (like the presence of
libssl, libz, or libpcre). We'll have to think more about it.

For the FFI move in the ngx_lua core, I'm going to wrap the Nginx C
APIs with an intermediate C function layer so that we don't have to
directly manipulate the underlying Nginx C structures (like the
troublesome ngx_http_request_t struct) from within Lua via FFI at all.
I'll work out all these details very soon :)

Best regards,
-agentzh

[ghost]

The LuaJIT FFI can already load .so on-the-fly.

Cool. No need to add anything, then. :-)

But accessing the raw ngx_http_request_t struct from within FFI
directly is a bad idea because the memory layout of ngx_http_request_t
may change upon different ./configure options (like the presence of
libssl, libz, or libpcre). We'll have to think more about it.

What I was thinking about specifically was providing a pointer to the request pool. It would be useful for libraries that allow the defining of external memory allocators (which usually require a pointer to some custom struct).

Wrt providing direct access to the http_request_t struct, it would certanly be possible to create a lua interface with the same struct that was compiled into the server, but would require some extra ./configure time code to generate the correct struct, and then load it as a single interface.

That said... I think wrapping the FFI stuff around an intermediate layer may well be better idea. I can see all sorts of problems starting if Lua has complete access to r*.

[ghost]

You could provide a dummied-down struct, with useful things like ngx_pool_t_, and set that at a global level instead of ngx_request_t_.

[dirkfeytons]

While I'm all for exploiting the power of LuaJIT please keep in mind not every platform is supported. That means there will still be users (like me...) who're stuck with the standard Lua interpreter. It would be a shame if they lose out on ngx_lua updates (although I understand that maintaining two code bases isn't free).

[agentzh]

Hello!

On Thu, Nov 22, 2012 at 1:00 AM, Dirk Feytons notifications@github.comwrote:

While I'm all for exploiting the power of LuaJIT please keep in mind not
every platform is supported. That means there will still be users (like
me...) who're stuck with the standard Lua interpreter. It would be a shame
if they lose out on ngx_lua updates (although I understand that maintaining
two code bases isn't free).

Yes, you're right here.

Last night chaoslawful also expressed his opinion that ngx_lua should
preserve compatibility with the standard Lua 5.1.x interpreter.

So I'd keep the two code branches in ngx_lua for the foreseeable future at
least :) I think a lot of the time these two code branches (one for
CFunction and one for FFI) can share the intermediate C wrapper layer, so a
lot of the code can still be shared, maybe :)

Best regards,
-agentzh

[ghost]

Yes, I'd agree that 5.1 compatability should be kept alive if there's a strong user base that can't use JIT for now.

One alternative that might be a possibility is creating an API layer, which is largely a file of macros that generalize calls to functions that would be slightly different to the different versions, should there be a difference at the code level between the 5.1/FFI codeset.

In fact I thought about this a while ago, not for ngx_lua specifcally, but for Nginx module code in general.

In a large amount of your code in all your modules you guys have been very careful to keep backwards compatability, using lots of pre-processor checks on the nginx_version macro. This is great, but requires lots of extra work, and means you really need to know the internals of a lot of Nginx versions.

Most developers don't have your long-term knowledge of the specific internals, and don't have the rigour to test every version of Nginx for compatability and the more versions that come out, the worse it's going to get.

I think it would be better in general to keep such checks to a single file, which defines macros based on the version, and then to use them in the places where you're currently checking nginx_version. Given the API changes a lot, it would probably be wise to include r* in pretty much all the macros, even if we don't use them now, in case of future changes.

It would be quite an effort to do the conversion, but I think there would be significant benefits wrt maintaining general Nginx module code, and I actually started the ngx_api module, but never got around to developing it. I thought it would be best to offer it both as part of the NDK as well as a separate file which module developers can included.

Would this be something you guys would be interested in working together on at some stage?

[agentzh]

To be honest, I no longer want to support too many versions of Nginx
in our modules nowadays :)

The earliest Nginx version officially supported by ngx_lua is 0.8.54
but I myself only regularly test against the latest releases in the
latest stable series and devel series of Nginx in the last few months
or so.

The current policy is to minimize the number of #if/#else in the code
base, by only supporting recent enough nginx releases.

Best regards,
-agentzh

[rwygand]

Curious how this support has come along in the last few months. I find myself in need of it!

[agentzh]

@rwygand No progress yet. Will try to work on it soon :)

[agentzh]

@shenvsv For now, you still need to use the standard ngx_proxy module with the ngx.location.capture API from ngx_lua for https upstream access.

The SSL/TLS support in the cosocket API still needs more time.

[shenvsv]

I am new to lua and openresty ,do you have any demo code for https request, thanks.

[agentzh]

@shenvsv The following tested example accesses the https site of google search from within Lua via subrequests:

location /google {
    internal;
    rewrite ^/google(.*) $1 break;    # to strip the leading /google prefix in subrequests
    proxy_set_header Accept-Encoding '';   # to disable gzip compression on the backend
    proxy_pass https://www.google.com;
}

location = /t {
    content_by_lua '
        -- search for "openresty" in google over https:
        local res = ngx.location.capture("/google/search",
                                         { args = { q = "openresty" } })
        if res.status ~= 200 then
            ngx.say("failed to query google: ", res.status, ": ", res.body)
            return
        end

        -- here we just forward the Google search result page intact:
        ngx.header["Content-Type"] = "text/html; charset=UTF-8"
        ngx.say(res.body)
    ';
}

[agentzh]

I've implemented the SSL/TLS cosocket API in the ssl-cosocket git branch. Please check out my latest comment on #290 for more details: #290 (comment) Feedback welcome :)

[shenvsv]

Thanks for u code.

[agentzh]

The ssl-cosocket branch has been merged into master. I'm closing this ticket. If you have any further issues with the SSL cosocket implementation, please file new separate tickets.