Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"too many open files" when serving static files for the concurrent requests #420

Closed
steadylearner opened this issue Jan 27, 2020 · 12 comments
Closed

"too many open files" when serving static files for the concurrent requests #420

steadylearner opened this issue Jan 27, 2020 · 12 comments

Comments

@steadylearner
Copy link

steadylearner commented Jan 27, 2020 •
edited
Loading

I was testing it with React app to test.

I want to compare it with other frameworks to find how Warp will behave with concurrent requests.

INFO Requests: 1086, requests per second: 216, mean latency: 133.9 ms
INFO Requests: 2548, requests per second: 295, mean latency: 582.9 ms
INFO Requests: 4011, requests per second: 293, mean latency: 1354.5 ms
INFO Errors: 115, accumulated errors: 115, 2.9% of total requests
INFO
INFO Target URL:          http://0.0.0.0:8000/
INFO Max time (s):        20
INFO Concurrency level:   10
INFO Agent:               keepalive
INFO Requests per second: 2000
INFO
INFO Completed requests:  5589
INFO Total errors:        492
INFO Total time:          20.006069541 s
INFO Requests per second: 279
INFO Mean latency:        1037.8 ms
INFO
INFO Percentage of the requests served within a certain time
INFO   50%      747 ms
INFO   90%      2419 ms
INFO   95%      2872 ms
INFO   99%      4934 ms
INFO  100%      7192 ms (longest request)
INFO
INFO  100%      7192 ms (longest request)
INFO
INFO   404:   492 errors

You can see that it shows many errors while other frameworks such as Express, Actix, Golang etc aren't and I also think latency and other numbers are comparavely high.

Is there a reason for them?

I made loadtest.bash and warp.bash to help you test the code locally at Rust Full Stack repository

Hope this can be solved fast because I want to write a blog post relevant to it to help others how it is easy to use Warp.
@jxs
Copy link
Collaborator

jxs commented Jan 27, 2020

Hi, from your output your errors seem to be 404's

@steadylearner
Copy link
Author

steadylearner commented Jan 27, 2020 •
edited
Loading

Thanks for the comment. I will reread the code and write here again if necessary.

(I am currently editing this again.)

I think that there were some errors at my side. I wanted to make an example similar to this while I refer to https://github.com/seanmonstar/warp/blob/master/examples/file.rs.

@seanmonstar
Copy link
Owner

Are the 404s from requesting directory pages? Warp doesn't generate those automatically.

@steadylearner
Copy link
Author

steadylearner commented Jan 27, 2020 •
edited
Loading

I retested the example but I still see the errors.

What I wanted to implement is very simple.

  1. When user visit 0.0.0.0:8000, return production React file at /public/index.html
  2. To make that work, I have to serve all the relevant files such as vendors.js, main.js, main.css etc and I include them inside /public

Current folder structure is similar to this.

├── Cargo.toml
├── loadtest.md
├── public
│   ├── index.html
│   ├── main.css
│   ├── main.js
│   ├── src
│   │   └── images
│   └── vendors.js
├── src
│   └── main.rs

and the main.rs is similar to this.

use warp::{self, path, Filter};

use console::Style;

#[tokio::main]
async fn main() {
    let target: String = "0.0.0.0:8000".parse().unwrap();
    let blue = Style::new()
        .blue();

    // When I visit the 0.0.0.0:8000, it shows the React app.
    // $curl 0.0.0.0:8000 also return index.html page
    let single_page_app = warp::get()
        .and(warp::path::end())
        .and(warp::fs::file("./public/index.html"));

    // 2. $curl 0.0.0.0:8000/public/index.html
    // dir already includes GET / ...
    // (You don't have to prefix warp::get("/")" here)

    // They all work when manually visiting them with a local browser and CURL.
    // $curl 0.0.0.0:8000/vendors.js
    // $curl 0.0.0.0:8000/main.js
    // $curl 0.0.0.0:8000/main.css
    // $curl 0.0.0.0:8000/index.html
    // $curl 0.0.0.0:8000/src/images/rust-chat-app.png
    let public_files = warp::fs::dir("./public/");

    // GET / => ./public/index.html
    // GET /public/... => ./public/..
    let routes = single_page_app.or(public_files);

    println!("\nRust Warp Server ready at {}", blue.apply_to(&target));
    println!("Use $curl http://0.0.0.0:8000 to test the end point.");

    warp::serve(routes).run(([0, 0, 0, 0], 8000)).await;
}

and I use this from loadtest.

npm install -g loadtest
loadtest http://0.0.0.0:8000/ -t 20 -c 10 --keepalive --rps 2000

I think that the entire code is very simple and no points for something to become wrong.

I already tested the same example with Actix, Golang, Express etc but the errors from Warp were unexpected ones.

@seanmonstar
I don't know I am requesting directory pages because I just test with /(0.0.0.0:8000).(Sorry, but I really didn't get it what you want to tell with that. I will spend time to read the documenation before you respond)

When I manually tested app, there were no problems.

I know that there will be solutions at documenation but please, hope you help with this.

If there are errors in my code, please let me know which part is making 404 errors.

@seanmonstar
Copy link
Owner

What if you enable logging to see what is responding with a 404?

// top of main()
pretty_env_logger::init();

let routes = single_page_app.or(public_files).with(warp::log("rust_full_stack"));

And then run with RUST_LOG=rust_full_stack=info cargo run --release.

@steadylearner
Copy link
Author

steadylearner commented Jan 27, 2020 •
edited
Loading

Ok, I will test with this and report later. Thanks for the instruction.

@steadylearner
Copy link
Author

steadylearner commented Jan 27, 2020 •
edited
Loading

I tested them with CURL and loadtest.

  1. curl http://0.0.0.0:8000
[Server] - INFO  warp_react > 127.0.0.1:55928 "GET / HTTP/1.1" 200 "-" "curl/7.58.0" 895.632µs
  1. Loadtest

I used $loadtest http://0.0.0.0:8000/ -t 20 -c 10 --keepalive --rps 2000.

[loadtest]
INFO Requests: 0, requests per second: 0, mean latency: 0 ms
INFO Requests: 1566, requests per second: 314, mean latency: 204.7 ms
INFO Requests: 3362, requests per second: 359, mean latency: 1034.1 ms
INFO Errors: 32, accumulated errors: 32, 1% of total requests
INFO Requests: 5235, requests per second: 375, mean latency: 1406.9 ms
INFO Errors: 251, accumulated errors: 283, 5.4% of total requests
INFO 
INFO Target URL:          http://0.0.0.0:8000/
INFO Max time (s):        20
INFO Concurrency level:   10
INFO Agent:               keepalive
INFO Requests per second: 2000
INFO 
INFO Completed requests:  7918
INFO Total errors:        1306
INFO Total time:          20.008969989 s
INFO Requests per second: 396
INFO Mean latency:        1120 ms
INFO 
INFO Percentage of the requests served within a certain time
INFO   50%      814 ms
INFO   90%      2088 ms
INFO   95%      2716 ms
INFO   99%      8379 ms
INFO  100%      10357 ms (longest request)
INFO 
INFO  100%      10357 ms (longest request)
INFO 
INFO   404:   1306 errors

You can see that 200 and 404 responses are mixed in server log.(This is not total result. There are so many of them.)

Why this shows 200 sometimes but 404 also? I can't see which file is making a problem with this.

[Server]

 INFO  warp_react > 127.0.0.1:57036 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 229.554828ms
 INFO  warp_react > 127.0.0.1:57196 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 229.756112ms
 INFO  warp_react > 127.0.0.1:57038 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 229.905541ms
 INFO  warp_react > 127.0.0.1:55986 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 604.462401ms
 INFO  warp_react > 127.0.0.1:56706 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 605.173089ms
 INFO  warp_react > 127.0.0.1:56870 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 606.370393ms
 INFO  warp_react > 127.0.0.1:56902 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 606.292547ms
 INFO  warp_react > 127.0.0.1:56294 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 1.062560339s
 INFO  warp_react > 127.0.0.1:57054 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 610.406673ms
 INFO  warp_react > 127.0.0.1:56862 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 1.06524146s
 INFO  warp_react > 127.0.0.1:56416 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 1.065375958s
 INFO  warp_react > 127.0.0.1:55974 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 572.777459ms
 INFO  warp_react > 127.0.0.1:56696 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 1.038466871s
 INFO  warp_react > 127.0.0.1:56078 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 1.780839ms
 INFO  warp_react > 127.0.0.1:56610 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 1.038760923s
 INFO  warp_react > 127.0.0.1:56460 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 1.032582737s
 INFO  warp_react > 127.0.0.1:56680 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 557.460073ms
 INFO  warp_react > 127.0.0.1:56550 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 558.866439ms
 INFO  warp_react > 127.0.0.1:56324 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 559.660298ms
 INFO  warp_react > 127.0.0.1:56130 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 559.883835ms
 INFO  warp_react > 127.0.0.1:56604 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 559.974029ms
 INFO  warp_react > 127.0.0.1:56358 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 559.951088ms
 INFO  warp_react > 127.0.0.1:57260 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 560.335344ms
 INFO  warp_react > 127.0.0.1:57016 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 1.036242268s
 INFO  warp_react > 127.0.0.1:56312 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 1.023025747s
 INFO  warp_react > 127.0.0.1:56444 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 1.022572515s
 INFO  warp_react > 127.0.0.1:57058 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 1.023882079s
 INFO  warp_react > 127.0.0.1:56226 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 1.024208055s
 INFO  warp_react > 127.0.0.1:55952 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 1.020332277s
 INFO  warp_react > 127.0.0.1:56838 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 1.007834037s
 INFO  warp_react > 127.0.0.1:55934 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 1.002842755s
 INFO  warp_react > 127.0.0.1:56664 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 519.957194ms
 INFO  warp_react > 127.0.0.1:57446 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 520.536729ms
 INFO  warp_react > 127.0.0.1:56488 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 521.669722ms
 INFO  warp_react > 127.0.0.1:57312 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 520.71929ms
 INFO  warp_react > 127.0.0.1:57048 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 522.304354ms
 INFO  warp_react > 127.0.0.1:56284 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 523.428197
 INFO  warp_react > 127.0.0.1:56714 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 252.03459ms
 INFO  warp_react > 127.0.0.1:56968 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 1.278372ms
 INFO  warp_react > 127.0.0.1:56868 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 2.080321ms
 INFO  warp_react > 127.0.0.1:56860 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 10.090626ms
 INFO  warp_react > 127.0.0.1:56354 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 277.846633ms
 INFO  warp_react > 127.0.0.1:56124 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 278.857069ms
 INFO  warp_react > 127.0.0.1:56958 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 277.310423ms
 INFO  warp_react > 127.0.0.1:56118 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 279.686289ms
 INFO  warp_react > 127.0.0.1:56742 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 278.8943ms
 INFO  warp_react > 127.0.0.1:56338 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 278.76094ms
 INFO  warp_react > 127.0.0.1:57160 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 279.221222ms
 INFO  warp_react > 127.0.0.1:56776 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 282.017837ms
 INFO  warp_react > 127.0.0.1:56880 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 263.970205ms
 INFO  warp_react > 127.0.0.1:56992 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 264.310567ms
 INFO  warp_react > 127.0.0.1:56058 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 269.490571ms
 INFO  warp_react > 127.0.0.1:56174 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 271.146927ms
 INFO  warp_react > 127.0.0.1:56474 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 273.229774ms
 INFO  warp_react > 127.0.0.1:56744 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 274.23584ms
 INFO  warp_react > 127.0.0.1:56036 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 264.687822ms
 INFO  warp_react > 127.0.0.1:57040 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 4.660063ms
 INFO  warp_react > 127.0.0.1:56630 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 4.607532ms
 INFO  warp_react > 127.0.0.1:56436 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 259.683677ms
 INFO  warp_react > 127.0.0.1:56622 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 260.97856ms
 INFO  warp_react > 127.0.0.1:56438 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 261.949881ms
 INFO  warp_react > 127.0.0.1:56426 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 253.616125ms
 INFO  warp_react > 127.0.0.1:56948 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 254.315707ms
 INFO  warp_react > 127.0.0.1:57240 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 254.983865ms
 INFO  warp_react > 127.0.0.1:57172 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 254.930886ms
 INFO  warp_react > 127.0.0.1:56478 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 254.245135ms
 INFO  warp_react > 127.0.0.1:56798 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 257.188653ms
 INFO  warp_react > 127.0.0.1:56648 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 249.079582ms
 INFO  warp_react > 127.0.0.1:56764 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 247.930583ms
 INFO  warp_react > 127.0.0.1:56456 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 245.675487ms
 INFO  warp_react > 127.0.0.1:57060 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 215.576747ms
 INFO  warp_react > 127.0.0.1:56694 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 216.359251ms
 INFO  warp_react > 127.0.0.1:56590 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 217.639427ms
 INFO  warp_react > 127.0.0.1:57080 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 217.72922ms
 INFO  warp_react > 127.0.0.1:55946 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 217.931476ms
 INFO  warp_react > 127.0.0.1:56626 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 1.98291ms
 INFO  warp_react > 127.0.0.1:56184 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 221.302583ms
 INFO  warp_react > 127.0.0.1:56176 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 222.095119ms
 INFO  warp_react > 127.0.0.1:56828 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 222.179595ms
 INFO  warp_react > 127.0.0.1:57116 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 222.292763ms
 INFO  warp_react > 127.0.0.1:57232 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 222.700313ms
 INFO  warp_react > 127.0.0.1:56930 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 486.298258ms
 INFO  warp_react > 127.0.0.1:57254 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 486.473932ms
 INFO  warp_react > 127.0.0.1:56060 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 486.542797ms
 INFO  warp_react > 127.0.0.1:57266 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 486.602055ms
 INFO  warp_react > 127.0.0.1:58018 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 486.627554ms
 INFO  warp_react > 127.0.0.1:57112 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 486.274772ms
 INFO  warp_react > 127.0.0.1:56418 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 486.255623ms
 INFO  warp_react > 127.0.0.1:58032 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 490.229206ms
 INFO  warp_react > 127.0.0.1:58022 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 490.270359ms
 INFO  warp_react > 127.0.0.1:56646 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 215.298081ms
 INFO  warp_react > 127.0.0.1:56310 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 215.632645ms
 INFO  warp_react > 127.0.0.1:57090 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 215.690396ms
 INFO  warp_react > 127.0.0.1:56998 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 215.973071ms
 INFO  warp_react > 127.0.0.1:56446 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 216.14338ms
 INFO  warp_react > 127.0.0.1:57082 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 216.206688ms
 INFO  warp_react > 127.0.0.1:56804 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 216.481953ms
 INFO  warp_react > 127.0.0.1:58060 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 493.569801ms
 INFO  warp_react > 127.0.0.1:58058 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 493.805383ms
 INFO  warp_react > 127.0.0.1:58056 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 493.890308ms
 INFO  warp_react > 127.0.0.1:57010 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 494.000365ms
 INFO  warp_react > 127.0.0.1:57130 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 218.060625ms
 INFO  warp_react > 127.0.0.1:57076 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 494.42781ms
 INFO  warp_react > 127.0.0.1:56770 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 494.548491ms
 INFO  warp_react > 127.0.0.1:57468 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 495.925645ms
 INFO  warp_react > 127.0.0.1:56858 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 218.355937ms
 INFO  warp_react > 127.0.0.1:57092 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 218.456964ms
 INFO  warp_react > 127.0.0.1:57470 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 498.487777ms
 INFO  warp_react > 127.0.0.1:58066 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 498.696031ms
 INFO  warp_react > 127.0.0.1:58072 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 498.57041ms
 INFO  warp_react > 127.0.0.1:58064 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 499.046152ms
 INFO  warp_react > 127.0.0.1:57500 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 487.389618ms
 INFO  warp_react > 127.0.0.1:57430 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 487.213415ms
 INFO  warp_react > 127.0.0.1:58088 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 487.184789ms
 INFO  warp_react > 127.0.0.1:58090 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 487.214523ms
 INFO  warp_react > 127.0.0.1:58100 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 487.082077ms
 INFO  warp_react > 127.0.0.1:58094 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 487.76754ms
 INFO  warp_react > 127.0.0.1:58104 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 485.04857ms
 INFO  warp_react > 127.0.0.1:57150 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 485.058249ms
 INFO  warp_react > 127.0.0.1:57256 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 484.970491ms
 INFO  warp_react > 127.0.0.1:56498 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 484.954213ms
 INFO  warp_react > 127.0.0.1:56016 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 485.011371ms
 INFO  warp_react > 127.0.0.1:57122 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 352.213µs
 INFO  warp_react > 127.0.0.1:56466 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 485.073821ms
 INFO  warp_react > 127.0.0.1:56892 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 3.394546ms
 INFO  warp_react > 127.0.0.1:58106 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 489.183889ms
 INFO  warp_react > 127.0.0.1:57250 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 481.795194ms
 INFO  warp_react > 127.0.0.1:55980 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 486.462065ms
 INFO  warp_react > 127.0.0.1:56116 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 486.490307ms
 INFO  warp_react > 127.0.0.1:57174 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 486.493121ms
 INFO  warp_react > 127.0.0.1:56598 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 486.53574ms
 INFO  warp_react > 127.0.0.1:57306 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 486.486401ms
 INFO  warp_react > 127.0.0.1:56514 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 486.248334ms
 INFO  warp_react > 127.0.0.1:56752 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 486.25576ms
 INFO  warp_react > 127.0.0.1:56516 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 486.192532ms
 INFO  warp_react > 127.0.0.1:56196 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 486.211457ms
 INFO  warp_react > 127.0.0.1:56688 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 486.153514ms
 INFO  warp_react > 127.0.0.1:56964 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 486.179847ms
 INFO  warp_react > 127.0.0.1:57300 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 481.16794ms
 INFO  warp_react > 127.0.0.1:56086 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 486.210855ms
 INFO  warp_react > 127.0.0.1:57298 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 481.091473ms
 INFO  warp_react > 127.0.0.1:57464 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 486.219066ms
 INFO  warp_react > 127.0.0.1:56710 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 483.251621ms
 INFO  warp_react > 127.0.0.1:57284 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 480.23231ms
 INFO  warp_react > 127.0.0.1:57280 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 480.047712ms
 INFO  warp_react > 127.0.0.1:58240 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 478.882925ms
 INFO  warp_react > 127.0.0.1:58246 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 478.987096ms
 INFO  warp_react > 127.0.0.1:58202 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 478.781432ms
 INFO  warp_react > 127.0.0.1:56320 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 478.275613ms
 INFO  warp_react > 127.0.0.1:56782 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 478.316917ms
 INFO  warp_react > 127.0.0.1:58204 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 478.182379ms
 INFO  warp_react > 127.0.0.1:57136 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 478.331559ms
 INFO  warp_react > 127.0.0.1:58014 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 478.242055ms
 INFO  warp_react > 127.0.0.1:56572 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 477.958938ms
 INFO  warp_react > 127.0.0.1:55982 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 477.972603ms
 INFO  warp_react > 127.0.0.1:57328 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 477.121306ms
 INFO  warp_react > 127.0.0.1:57324 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 476.892276ms
 INFO  warp_react > 127.0.0.1:57318 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 476.68905ms
 INFO  warp_react > 127.0.0.1:56952 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 476.323206ms
 INFO  warp_react > 127.0.0.1:56110 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 476.204818ms
 INFO  warp_react > 127.0.0.1:56704 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 476.133877ms
 INFO  warp_react > 127.0.0.1:56822 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 476.075829ms
 INFO  warp_react > 127.0.0.1:56448 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 475.882659ms
 INFO  warp_react > 127.0.0.1:56934 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 475.864665ms
 INFO  warp_react > 127.0.0.1:57268 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 475.610625ms
 INFO  warp_react > 127.0.0.1:56038 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 475.895144ms
 INFO  warp_react > 127.0.0.1:58206 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 474.211828ms
 INFO  warp_react > 127.0.0.1:55992 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 216.526586ms
 INFO  warp_react > 127.0.0.1:56726 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 216.136357ms
 INFO  warp_react > 127.0.0.1:56942 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 215.987417ms
 INFO  warp_react > 127.0.0.1:57104 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 471.132514ms
 INFO  warp_react > 127.0.0.1:56756 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 197.498677ms
 INFO  warp_react > 127.0.0.1:57074 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 190.660048ms
 INFO  warp_react > 127.0.0.1:56148 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 449.525987ms
 INFO  warp_react > 127.0.0.1:56146 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 449.423628ms
 INFO  warp_react > 127.0.0.1:56450 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 447.400785ms
 INFO  warp_react > 127.0.0.1:57044 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 175.966107ms
 INFO  warp_react > 127.0.0.1:57208 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 175.670041ms
 INFO  warp_react > 127.0.0.1:55972 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 438.725867ms
 INFO  warp_react > 127.0.0.1:56328 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 437.944348ms
 INFO  warp_react > 127.0.0.1:56008 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 436.722596ms
 INFO  warp_react > 127.0.0.1:57018 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 436.770347ms
 INFO  warp_react > 127.0.0.1:56156 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 436.923104ms
 INFO  warp_react > 127.0.0.1:56360 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 437.194938ms
 INFO  warp_react > 127.0.0.1:56012 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 437.595864ms
 INFO  warp_react > 127.0.0.1:56382 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 176.78443ms
 INFO  warp_react > 127.0.0.1:56528 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 176.918542ms
 INFO  warp_react > 127.0.0.1:56256 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 437.800132ms
 INFO  warp_react > 127.0.0.1:57156 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 171.247331ms
 INFO  warp_react > 127.0.0.1:56982 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 170.770147ms
 INFO  warp_react > 127.0.0.1:56978 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 172.840316ms
 INFO  warp_react > 127.0.0.1:56202 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 169.345786ms
 INFO  warp_react > 127.0.0.1:57234 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 162.267766ms
 INFO  warp_react > 127.0.0.1:56292 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 162.332838ms
 INFO  warp_react > 127.0.0.1:56010 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 163.041257ms
 INFO  warp_react > 127.0.0.1:56552 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 163.544093ms
 INFO  warp_react > 127.0.0.1:56166 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 427.367285ms
 INFO  warp_react > 127.0.0.1:57204 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 427.257092ms
 INFO  warp_react > 127.0.0.1:56808 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 438.811258ms
 INFO  warp_react > 127.0.0.1:57020 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 428.014874ms
 INFO  warp_react > 127.0.0.1:57014 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 424.44731ms
 INFO  warp_react > 127.0.0.1:56784 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 423.847704ms
 INFO  warp_react > 127.0.0.1:56650 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 424.207727ms
 INFO  warp_react > 127.0.0.1:56066 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 423.808701ms
 INFO  warp_react > 127.0.0.1:56700 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 423.652031ms
 INFO  warp_react > 127.0.0.1:56684 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 423.572542ms
 INFO  warp_react > 127.0.0.1:57200 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 423.28779ms
 INFO  warp_react > 127.0.0.1:57032 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 423.421895ms
 INFO  warp_react > 127.0.0.1:56468 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 423.305745ms
 INFO  warp_react > 127.0.0.1:57098 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 154.293281ms
 INFO  warp_react > 127.0.0.1:56606 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 416.842984ms
 INFO  warp_react > 127.0.0.1:55942 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 415.848529ms
 INFO  warp_react > 127.0.0.1:56386 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 415.604256ms
 INFO  warp_react > 127.0.0.1:55950 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 415.535623ms

@seanmonstar
Copy link
Owner

Interesting! Ok, could you change your RUST_LOG value to RUST_LOG=warp::filters::fs? That should output some trace or debug lines about what may have failed when trying to get the file.

@steadylearner
Copy link
Author

steadylearner commented Jan 27, 2020 •
edited
Loading

I tested it with your instruction.

I don't know about the low level details of your framework. But, I think that there are problems in closing the file when it is read to serve a / concurrently.

I also had similar issue when I used Rocket that used hyper behind. But, I am not sure that they are relevant here because it was very long ago.

I could find the issue that maybe relevant to it and another one.

[Server]

 INFO  warp::filters::fs > 127.0.0.1:59908 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 284.210366ms
 INFO  warp::filters::fs > 127.0.0.1:58864 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 288.461113ms
 INFO  warp::filters::fs > 127.0.0.1:59428 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 289.666135ms
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 INFO  warp::filters::fs > 127.0.0.1:60022 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 800.32µs
 INFO  warp::filters::fs > 127.0.0.1:59278 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 298.776792ms
 INFO  warp::filters::fs > 127.0.0.1:59464 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 299.334186ms
 INFO  warp::filters::fs > 127.0.0.1:59056 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 299.560746ms
 INFO  warp::filters::fs > 127.0.0.1:59108 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 299.64264ms
 INFO  warp::filters::fs > 127.0.0.1:59758 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 307.303545ms
 INFO  warp::filters::fs > 127.0.0.1:59556 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 307.523578ms
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 INFO  warp::filters::fs > 127.0.0.1:59446 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 308.873ms
 INFO  warp::filters::fs > 127.0.0.1:59954 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 308.916814ms
 INFO  warp::filters::fs > 127.0.0.1:59432 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 309.534878ms
 INFO  warp::filters::fs > 127.0.0.1:58906 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 303.918478ms
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 INFO  warp::filters::fs > 127.0.0.1:59396 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 10.879736ms
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 INFO  warp::filters::fs > 127.0.0.1:59478 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 3.214405ms
 INFO  warp::filters::fs > 127.0.0.1:60004 "GET / HTTP/1.1" 200 "-" "loadtest/4.0.0" 311.42398ms
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 ERROR warp::filters::fs > file open error (path="./public/index.html"): Too many open files (os error 24) 
 INFO  warp::filters::fs > 127.0.0.1:59448 "GET / HTTP/1.1" 404 "-" "loadtest/4.0.0" 3.843751ms

My machine is with very low computation power which is used at Linux 18.04 so I think that it may contributed it also. But, I wanted to report this because I had no problems with other frameworks.

Hope this helped.(I am not a native speaker.)

@seanmonstar seanmonstar changed the title [Bug] errors when serving static files for the concurrent requests? "too many open files" when serving static files for the concurrent requests Jan 27, 2020
@seanmonstar
Copy link
Owner

The logs do help, they show the issue is "too many open files". However, I'm not sure if that's a problem in warp... It implies the OS limit is being reaching, but later requests work fine, so the file descriptors aren't being leaked. I wonder if it's related at all to how tokio::fs is using a blocking threadpool to handle file operations, and if your machine doesn't have many CPUs, if that could mean its slower at getting back to the thread that needs to drop the file.

@seanmonstar
Copy link
Owner

Though, looking through your Cargo.toml, I see you've got warp v0.2.0, try updating to v0.2.1, there was a bug fix that had file reads being very slow.

@steadylearner
Copy link
Author

steadylearner commented Jan 27, 2020 •
edited
Loading

I updated the version and I think that it was the reason of the problem.

You may see the result at React_Rust also but let here to save your time.

[Loadtest]

INFO Requests: 0, requests per second: 0, mean latency: 0 ms
INFO Requests: 3832, requests per second: 767, mean latency: 8.3 ms
INFO Requests: 11189, requests per second: 1473, mean latency: 5.2 ms
INFO Requests: 19382, requests per second: 1638, mean latency: 4.8 ms
INFO
INFO Target URL:          http://0.0.0.0:8000/
INFO Max time (s):        20
INFO Concurrency level:   10
INFO Agent:               keepalive
INFO Requests per second: 2000
INFO
INFO Completed requests:  27364
INFO Total errors:        0
INFO Total time:          20.001672077 s
INFO Requests per second: 1368
INFO Mean latency:        5.5 ms
INFO
INFO Percentage of the requests served within a certain time
INFO   50%      4 ms
INFO   90%      8 ms
INFO   95%      10 ms
INFO   99%      16 ms
INFO  100%      82 ms (longest request)
INFO Requests: 27364, requests per second: 1597, mean latency: 5 ms

Its memory usage is comparavely low also with 37.44 Mb.

No errors anymore with the v 0.2.1 and it is very fast and shows low mean latency.

I like the design of your framework, reusability and many FP concepts behind it etc.(I am not sure of this but that was what I felt and could see your experty.)

I don't know you already read the "How to use Rust Warp" and liked it.

But, I think I can write "How to use React with Actix" version of Warp better with your help here.

Thanks for making the great framework and I will use it a lot more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@seanmonstar @jxs @steadylearner