Question

我有一个clojure / jetty服务器（在端口8081上）并使用nginx（端口8080）代理它。我一直在尝试单独对clojure应用程序进行基准测试，以及代理角色中的nginx。

当我直接对抗clojure进行测试时，我可能得到Connection reset by peer，每10次运行一次。通常，测试完成，性能可以接受。

$ ulimit -n 4096
$ ab -n 20000 -c 2048 -k localhost:8081
...
Concurrency Level:      2048
Time taken for tests:   8.713 seconds
Complete requests:      20000
Failed requests:        0
Keep-Alive requests:    20000
Total transferred:      15160000 bytes
HTML transferred:       11720000 bytes
Requests per second:    2295.43 [#/sec] (mean)
Time per request:       892.208 [ms] (mean)
Time per request:       0.436 [ms] (mean, across all concurrent requests)
Transfer rate:          1699.16 [Kbytes/sec] received
...

我开始测试完整的本地配置，端口8080上的nginx和8081上的clojure。事情进展顺利，直到我超过1024个并发连接。

我注意到，使用ss -tl，接收队列不是尖峰，或者至少如果它们是闪光灯。但我确实发现，使用netstat -s，正在发送大量TCP RST。有时，dmesg告诉我，它看起来像是SYN泛滥。此外，nginx响应HTTP状态499，这应该表明客户端关闭连接...

所以，我的诊断程序已经交叉，apache bench和nginx声称对方关闭了连接！？

$ ulimit -n 4096
$ ab -n 20000 -c 2048 -k localhost:8080
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
apr_socket_recv: Connection reset by peer (104)
Total of 26 requests completed

配置

我为nginx和clojure设置ulimit max open files为4096。

无效的网络变化

net.core.netdev_max_backlog=30000
# yes, we are using jumbo frames
net.ipv4.tcp_mtu_probing=1

net.core.somaxconn=4096
net.ipv4.ip_local_port_range=4096     61000
net.ipv4.tcp_fin_timeout=30
net.ipv4.tcp_max_syn_backlog=2048

/etc/nginx/nginx.conf

user www-data;
worker_processes  2;
worker_rlimit_nofile 100000;

error_log  /var/log/nginx/error.log;
pid        /run/nginx.pid;

events {
  worker_connections  2048;
  use epoll;
}

http {

  include       /etc/nginx/mime.types;
  default_type  application/octet-stream;


  access_log    /var/log/nginx/access.log;

  sendfile on;
  tcp_nopush on;
  tcp_nodelay on;

  keepalive_requests 1000;
  keepalive_timeout  65;


  gzip  on;
  gzip_http_version 1.0;
  gzip_comp_level 2;
  gzip_proxied any;
  gzip_vary off;
  gzip_types text/plain text/css application/x-javascript text/xml application/xml application/rss+xml application/atom+xml text/javascript application/javascript application/json text/mathml;
  gzip_min_length  1000;
  gzip_disable     "MSIE [1-6]\.";


  variables_hash_max_size 1024;
  variables_hash_bucket_size 64;
  server_names_hash_bucket_size 128;
  types_hash_max_size 2048;
  types_hash_bucket_size 64;



  include /etc/nginx/conf.d/*.conf;
  include /etc/nginx/sites-enabled/*;
}

的/ etc / nginx的/启用的站点 - / UPS

upstream ups {
  server localhost:8081 fail_timeout=0;
}


server {
  listen 8080 backlog=1024;

  server_name example.com;

  proxy_buffer_size   128k;
  proxy_buffers   4 256k;
  proxy_busy_buffers_size   256k;

  client_max_body_size 3M;

  large_client_header_buffers 4 128k;

  proxy_read_timeout 300;
  proxy_send_timeout 300;
  send_timeout 300;
  keepalive_timeout 300;

  server_tokens off;

  access_log /var/log/nginx/ups_access.log enhanced-combined;
  error_log /var/log/nginx/ups_error.log;
  root /apps/ups/current/public/;

  error_page 403 /errors/403_maintenance.html;
  error_page 500 /errors/500.html;
  location ^~ /errors/ {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-Request-Id $request_uuid;
    satisfy any;
    allow all;
  }

  location / {

    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-Request-Id $request_uuid;

    if ($http_x_forwarded_proto = 'http') {
      rewrite ^ https://$host$request_uri? permanent;
    }


    if (-f $request_filename/index.html) {
      rewrite (.*) $1/index.html break;
    }

    if (-f $request_filename.html) {
      rewrite (.*) $1.html break;
    }

    if (!-f $request_filename) {
      proxy_pass http://ups;
      break;
    }
  }
}

Answer 1

@Terra几乎得到了解决问题的答案。

accept_mutex off;
worker_connections 4096;

我尝试自己添加每个更改并重新加载，但我仍然看到了同样的错误。直到我改变了两个，我才能超过1024个连接。

看起来，由于代理，我需要有两倍的工作连接，因为我打算接受。

为什么nginx不能接受来自apache bench的超过1024个连接

配置

无效的网络变化

/etc/nginx/nginx.conf

的/ etc / nginx的/启用的站点 - / UPS

1 个答案: