local backend error: too many open files

[cyliu0]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.
   When importing massive data (2.8T, 40k warehouses tpcc data), tidb-lightning will raise error "too many open files" constantly without quit.

2. What did you expect to see?
   Check the limit files before importing

3. What did you see instead?
   Error "too many open files" all the time

4. Versions of the cluster

   • TiDB-Lightning version (run tidb-lightning -V):

     [root@localhost tidb-lightning]# ./bin/tidb-lightning -V
     Release Version: v3.0.16-2-g4af19d5
     Git Commit Hash: 4af19d5d44837d6c6fe3ba3ad191df354e310ad3
     Git Branch: master
     UTC Build Time: 2020-07-02 10:23:10
     Go Version: go version go1.13.11 linux/amd64
     

   • TiKV-Importer version (run tikv-importer -V)

     (paste TiKV-Importer version here)
     

   • TiKV version (run tikv-server -V):

     (paste TiKV version here)
     

   • TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

     (paste TiDB cluster version here)
     

   • Other interesting information (system version, hardware config, etc):

5. Operation logs

   • Please upload tidb-lightning.log for TiDB-Lightning if possible
   • Please upload tikv-importer.log from TiKV-Importer if possible
   • Other interesting logs

6. Configuration of the cluster and the task

   • tidb-lightning.toml for TiDB-Lightning if possible
   • tikv-importer.toml for TiKV-Importer if possible
   • inventory.ini if deployed by Ansible

7. Screenshot/exported-PDF of Grafana dashboard or metrics' graph in Prometheus for TiDB-Lightning if possible