事象内容
Openstack ダッシュボードでログインしようとすると、Internal Server Error （内部エラー）が発生する。
httpd の エラーログ（ /var/log/httpd/error_log ）を確認すると、以下エラーメッセージが出力されている。
[:error] [pid 3395] IOError: [Errno 24] Too many open files
[root@controller ~]# tail -F /var/log/httpd/error_log [Mon Nov 05 21:54:04.706474 2018] [:error] [pid 3395] INFO openstack_auth.plugin.base Attempted scope to domain default failed, will attempt to scope to another domain. [Mon Nov 05 21:54:04.763415 2018] [:error] [pid 3395] ERROR django.request Internal Server Error: /dashboard/auth/login/ [Mon Nov 05 21:54:04.763437 2018] [:error] [pid 3395] Traceback (most recent call last): [Mon Nov 05 21:54:04.763439 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner [Mon Nov 05 21:54:04.763441 2018] [:error] [pid 3395] response = get_response(request) [Mon Nov 05 21:54:04.763442 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response [Mon Nov 05 21:54:04.763444 2018] [:error] [pid 3395] response = self.process_exception_by_middleware(e, request) [Mon Nov 05 21:54:04.763446 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response [Mon Nov 05 21:54:04.763448 2018] [:error] [pid 3395] response = wrapped_callback(request, *callback_args, **callback_kwargs) [Mon Nov 05 21:54:04.763449 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/views/decorators/debug.py", line 76, in sensitive_post_parameters_wrapper [Mon Nov 05 21:54:04.763451 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/utils/decorators.py", line 149, in _wrapped_view [Mon Nov 05 21:54:04.763453 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/views/decorators/cache.py", line 57, in _wrapped_view_func [Mon Nov 05 21:54:04.763454 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/openstack_auth/views.py", line 110, in login [Mon Nov 05 21:54:04.763456 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/contrib/auth/views.py", line 54, in inner [Mon Nov 05 21:54:04.763458 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/contrib/auth/views.py", line 150, in login [Mon Nov 05 21:54:04.763464 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view [Mon Nov 05 21:54:04.763465 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/utils/decorators.py", line 67, in _wrapper [Mon Nov 05 21:54:04.763467 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/views/decorators/debug.py", line 76, in sensitive_post_parameters_wrapper [Mon Nov 05 21:54:04.763469 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/utils/decorators.py", line 63, in bound_func [Mon Nov 05 21:54:04.763470 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/utils/decorators.py", line 67, in _wrapper [Mon Nov 05 21:54:04.763472 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/utils/decorators.py", line 149, in _wrapped_view [Mon Nov 05 21:54:04.763474 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/utils/decorators.py", line 63, in bound_func [Mon Nov 05 21:54:04.763475 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/utils/decorators.py", line 67, in _wrapper [Mon Nov 05 21:54:04.763477 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/views/decorators/cache.py", line 57, in _wrapped_view_func [Mon Nov 05 21:54:04.763478 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/utils/decorators.py", line 63, in bound_func [Mon Nov 05 21:54:04.763480 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/contrib/auth/views.py", line 90, in dispatch [Mon Nov 05 21:54:04.763482 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 88, in dispatch [Mon Nov 05 21:54:04.763483 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/views/generic/edit.py", line 182, in post [Mon Nov 05 21:54:04.763485 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/forms/forms.py", line 183, in is_valid [Mon Nov 05 21:54:04.763486 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/forms/forms.py", line 175, in errors [Mon Nov 05 21:54:04.763488 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/forms/forms.py", line 385, in full_clean [Mon Nov 05 21:54:04.763499 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/forms/forms.py", line 412, in _clean_form [Mon Nov 05 21:54:04.763501 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/views/decorators/debug.py", line 36, in sensitive_variables_wrapper [Mon Nov 05 21:54:04.763503 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/openstack_auth/forms.py", line 136, in clean [Mon Nov 05 21:54:04.763504 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/contrib/auth/__init__.py", line 70, in authenticate [Mon Nov 05 21:54:04.763506 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/contrib/auth/__init__.py", line 116, in _authenticate_with_backend [Mon Nov 05 21:54:04.763507 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/openstack_auth/backend.py", line 132, in authenticate [Mon Nov 05 21:54:04.763509 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/openstack_auth/plugin/base.py", line 181, in get_project_scoped_auth [Mon Nov 05 21:54:04.763511 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access [Mon Nov 05 21:54:04.763512 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref [Mon Nov 05 21:54:04.763514 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 1019, in post [Mon Nov 05 21:54:04.763516 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 814, in request [Mon Nov 05 21:54:04.763517 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 903, in _send_request [Mon Nov 05 21:54:04.763520 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 512, in request [Mon Nov 05 21:54:04.763522 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 622, in send [Mon Nov 05 21:54:04.763523 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 445, in send [Mon Nov 05 21:54:04.763525 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 588, in urlopen [Mon Nov 05 21:54:04.763527 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 241, in _get_conn [Mon Nov 05 21:54:04.763528 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/urllib3/util/connection.py", line 27, in is_connection_dropped [Mon Nov 05 21:54:04.763530 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/urllib3/util/wait.py", line 33, in wait_for_read [Mon Nov 05 21:54:04.763531 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/urllib3/util/wait.py", line 22, in _wait_for_io_events [Mon Nov 05 21:54:04.763533 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/urllib3/util/selectors.py", line 581, in DefaultSelector [Mon Nov 05 21:54:04.763535 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/urllib3/util/selectors.py", line 394, in __init__ [Mon Nov 05 21:54:04.763536 2018] [:error] [pid 3395] IOError: [Errno 24] Too many open files [Mon Nov 05 21:54:04.763838 2018] [:error] [pid 3395] ERROR django.request Internal Server Error: /dashboard/auth/login/ [Mon Nov 05 21:54:04.763843 2018] [:error] [pid 3395] Traceback (most recent call last): [Mon Nov 05 21:54:04.763845 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner [Mon Nov 05 21:54:04.763846 2018] [:error] [pid 3395] response = get_response(request) [Mon Nov 05 21:54:04.763848 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/utils/deprecation.py", line 140, in __call__ [Mon Nov 05 21:54:04.763850 2018] [:error] [pid 3395] response = self.get_response(request) [Mon Nov 05 21:54:04.763851 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 43, in inner [Mon Nov 05 21:54:04.763853 2018] [:error] [pid 3395] response = response_for_exception(request, exc) [Mon Nov 05 21:54:04.763854 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 93, in response_for_exception [Mon Nov 05 21:54:04.763856 2018] [:error] [pid 3395] response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info()) [Mon Nov 05 21:54:04.763858 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 143, in handle_uncaught_exception [Mon Nov 05 21:54:04.763859 2018] [:error] [pid 3395] return callback(request, **param_dict) [Mon Nov 05 21:54:04.763861 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/utils/decorators.py", line 149, in _wrapped_view [Mon Nov 05 21:54:04.763862 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/views/defaults.py", line 68, in server_error [Mon Nov 05 21:54:04.763864 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loader.py", line 21, in get_template [Mon Nov 05 21:54:04.763866 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/backends/django.py", line 39, in get_template [Mon Nov 05 21:54:04.763867 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/engine.py", line 162, in get_template [Mon Nov 05 21:54:04.763869 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/engine.py", line 136, in find_template [Mon Nov 05 21:54:04.763870 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loaders/base.py", line 38, in get_template [Mon Nov 05 21:54:04.763874 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loaders/filesystem.py", line 28, in get_contents [Mon Nov 05 21:54:04.763876 2018] [:error] [pid 3395] IOError: [Errno 24] Too many open files: '/usr/share/openstack-dashboard/openstack_dashboard/themes/default/templates/500.html' [Mon Nov 05 21:54:04.764132 2018] [:error] [pid 3395] ERROR django.request Internal Server Error: /dashboard/auth/login/ [Mon Nov 05 21:54:04.764137 2018] [:error] [pid 3395] Traceback (most recent call last): [Mon Nov 05 21:54:04.764139 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner [Mon Nov 05 21:54:04.764141 2018] [:error] [pid 3395] response = get_response(request) [Mon Nov 05 21:54:04.764142 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/utils/deprecation.py", line 140, in __call__ [Mon Nov 05 21:54:04.764144 2018] [:error] [pid 3395] response = self.get_response(request) [Mon Nov 05 21:54:04.764146 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 43, in inner [Mon Nov 05 21:54:04.764147 2018] [:error] [pid 3395] response = response_for_exception(request, exc) [Mon Nov 05 21:54:04.764149 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 93, in response_for_exception [Mon Nov 05 21:54:04.764151 2018] [:error] [pid 3395] response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info()) [Mon Nov 05 21:54:04.764156 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 143, in handle_uncaught_exception [Mon Nov 05 21:54:04.764282 2018] [:error] [pid 3395] return callback(request, **param_dict) [Mon Nov 05 21:54:04.764303 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/utils/decorators.py", line 149, in _wrapped_view [Mon Nov 05 21:54:04.764305 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/views/defaults.py", line 68, in server_error [Mon Nov 05 21:54:04.764307 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loader.py", line 21, in get_template [Mon Nov 05 21:54:04.764309 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/backends/django.py", line 39, in get_template [Mon Nov 05 21:54:04.764311 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/engine.py", line 162, in get_template [Mon Nov 05 21:54:04.764312 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/engine.py", line 136, in find_template [Mon Nov 05 21:54:04.764317 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loaders/base.py", line 38, in get_template [Mon Nov 05 21:54:04.764325 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loaders/filesystem.py", line 28, in get_contents [Mon Nov 05 21:54:04.764327 2018] [:error] [pid 3395] IOError: [Errno 24] Too many open files: '/usr/share/openstack-dashboard/openstack_dashboard/themes/default/templates/500.html' [Mon Nov 05 21:54:04.764617 2018] [:error] [pid 3395] ERROR django.request Internal Server Error: /dashboard/auth/login/ [Mon Nov 05 21:54:04.764628 2018] [:error] [pid 3395] Traceback (most recent call last): [Mon Nov 05 21:54:04.764630 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner [Mon Nov 05 21:54:04.764632 2018] [:error] [pid 3395] response = get_response(request) [Mon Nov 05 21:54:04.764634 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/horizon/themes.py", line 100, in __call__ [Mon Nov 05 21:54:04.764636 2018] [:error] [pid 3395] response = self.get_response(request) [Mon Nov 05 21:54:04.764643 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 43, in inner [Mon Nov 05 21:54:04.764645 2018] [:error] [pid 3395] response = response_for_exception(request, exc) [Mon Nov 05 21:54:04.764647 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 93, in response_for_exception [Mon Nov 05 21:54:04.764650 2018] [:error] [pid 3395] response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info()) [Mon Nov 05 21:54:04.764658 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 143, in handle_uncaught_exception [Mon Nov 05 21:54:04.764660 2018] [:error] [pid 3395] return callback(request, **param_dict) [Mon Nov 05 21:54:04.764662 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/utils/decorators.py", line 149, in _wrapped_view [Mon Nov 05 21:54:04.764664 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/views/defaults.py", line 68, in server_error [Mon Nov 05 21:54:04.764666 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loader.py", line 21, in get_template [Mon Nov 05 21:54:04.764667 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/backends/django.py", line 39, in get_template [Mon Nov 05 21:54:04.764671 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/engine.py", line 162, in get_template [Mon Nov 05 21:54:04.764707 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/engine.py", line 136, in find_template [Mon Nov 05 21:54:04.764709 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loaders/base.py", line 38, in get_template [Mon Nov 05 21:54:04.764711 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loaders/filesystem.py", line 28, in get_contents [Mon Nov 05 21:54:04.764714 2018] [:error] [pid 3395] IOError: [Errno 24] Too many open files: '/usr/share/openstack-dashboard/openstack_dashboard/themes/default/templates/500.html' [Mon Nov 05 21:54:04.764988 2018] [:error] [pid 3395] ERROR django.request Internal Server Error: /dashboard/auth/login/ [Mon Nov 05 21:54:04.764997 2018] [:error] [pid 3395] Traceback (most recent call last): [Mon Nov 05 21:54:04.764999 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner [Mon Nov 05 21:54:04.765001 2018] [:error] [pid 3395] response = get_response(request) [Mon Nov 05 21:54:04.765003 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/horizon/middleware/base.py", line 51, in __call__ [Mon Nov 05 21:54:04.765004 2018] [:error] [pid 3395] response = self.get_response(request) [Mon Nov 05 21:54:04.765006 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 43, in inner [Mon Nov 05 21:54:04.765007 2018] [:error] [pid 3395] response = response_for_exception(request, exc) [Mon Nov 05 21:54:04.765009 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 93, in response_for_exception [Mon Nov 05 21:54:04.765011 2018] [:error] [pid 3395] response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info()) [Mon Nov 05 21:54:04.765012 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 143, in handle_uncaught_exception [Mon Nov 05 21:54:04.765014 2018] [:error] [pid 3395] return callback(request, **param_dict) [Mon Nov 05 21:54:04.765016 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/utils/decorators.py", line 149, in _wrapped_view [Mon Nov 05 21:54:04.765017 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/views/defaults.py", line 68, in server_error [Mon Nov 05 21:54:04.765019 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loader.py", line 21, in get_template [Mon Nov 05 21:54:04.765023 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/backends/django.py", line 39, in get_template [Mon Nov 05 21:54:04.765024 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/engine.py", line 162, in get_template [Mon Nov 05 21:54:04.765026 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/engine.py", line 136, in find_template [Mon Nov 05 21:54:04.765027 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loaders/base.py", line 38, in get_template [Mon Nov 05 21:54:04.765029 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loaders/filesystem.py", line 28, in get_contents [Mon Nov 05 21:54:04.765031 2018] [:error] [pid 3395] IOError: [Errno 24] Too many open files: '/usr/share/openstack-dashboard/openstack_dashboard/themes/default/templates/500.html' [Mon Nov 05 21:54:04.765276 2018] [:error] [pid 3395] ERROR django.request Internal Server Error: /dashboard/auth/login/ [Mon Nov 05 21:54:04.765285 2018] [:error] [pid 3395] Traceback (most recent call last): [Mon Nov 05 21:54:04.765287 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner [Mon Nov 05 21:54:04.765289 2018] [:error] [pid 3395] response = get_response(request) [Mon Nov 05 21:54:04.765291 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/utils/deprecation.py", line 140, in __call__ [Mon Nov 05 21:54:04.765293 2018] [:error] [pid 3395] response = self.get_response(request) [Mon Nov 05 21:54:04.765294 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 43, in inner [Mon Nov 05 21:54:04.765296 2018] [:error] [pid 3395] response = response_for_exception(request, exc) [Mon Nov 05 21:54:04.765298 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 93, in response_for_exception [Mon Nov 05 21:54:04.765299 2018] [:error] [pid 3395] response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info()) [Mon Nov 05 21:54:04.765301 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 143, in handle_uncaught_exception [Mon Nov 05 21:54:04.765303 2018] [:error] [pid 3395] return callback(request, **param_dict) [Mon Nov 05 21:54:04.765304 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/utils/decorators.py", line 149, in _wrapped_view [Mon Nov 05 21:54:04.765306 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/views/defaults.py", line 68, in server_error [Mon Nov 05 21:54:04.765308 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loader.py", line 21, in get_template [Mon Nov 05 21:54:04.765311 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/backends/django.py", line 39, in get_template [Mon Nov 05 21:54:04.765325 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/engine.py", line 162, in get_template [Mon Nov 05 21:54:04.765362 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/engine.py", line 136, in find_template [Mon Nov 05 21:54:04.765365 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loaders/base.py", line 38, in get_template [Mon Nov 05 21:54:04.765391 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loaders/filesystem.py", line 28, in get_contents [Mon Nov 05 21:54:04.765393 2018] [:error] [pid 3395] IOError: [Errno 24] Too many open files: '/usr/share/openstack-dashboard/openstack_dashboard/themes/default/templates/500.html' [Mon Nov 05 21:54:04.765650 2018] [:error] [pid 3395] ERROR django.request Internal Server Error: /dashboard/auth/login/ [Mon Nov 05 21:54:04.765663 2018] [:error] [pid 3395] Traceback (most recent call last): [Mon Nov 05 21:54:04.765666 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner [Mon Nov 05 21:54:04.765668 2018] [:error] [pid 3395] response = get_response(request) [Mon Nov 05 21:54:04.765669 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/utils/deprecation.py", line 140, in __call__ [Mon Nov 05 21:54:04.765671 2018] [:error] [pid 3395] response = self.get_response(request) [Mon Nov 05 21:54:04.765672 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 43, in inner [Mon Nov 05 21:54:04.765674 2018] [:error] [pid 3395] response = response_for_exception(request, exc) [Mon Nov 05 21:54:04.765676 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 93, in response_for_exception [Mon Nov 05 21:54:04.765678 2018] [:error] [pid 3395] response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info()) [Mon Nov 05 21:54:04.765679 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 143, in handle_uncaught_exception [Mon Nov 05 21:54:04.765681 2018] [:error] [pid 3395] return callback(request, **param_dict) [Mon Nov 05 21:54:04.765682 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/utils/decorators.py", line 149, in _wrapped_view [Mon Nov 05 21:54:04.765684 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/views/defaults.py", line 68, in server_error [Mon Nov 05 21:54:04.765686 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loader.py", line 21, in get_template [Mon Nov 05 21:54:04.765688 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/backends/django.py", line 39, in get_template [Mon Nov 05 21:54:04.765689 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/engine.py", line 162, in get_template [Mon Nov 05 21:54:04.765691 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/engine.py", line 136, in find_template [Mon Nov 05 21:54:04.765692 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loaders/base.py", line 38, in get_template [Mon Nov 05 21:54:04.765694 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loaders/filesystem.py", line 28, in get_contents [Mon Nov 05 21:54:04.765696 2018] [:error] [pid 3395] IOError: [Errno 24] Too many open files: '/usr/share/openstack-dashboard/openstack_dashboard/themes/default/templates/500.html' [Mon Nov 05 21:54:04.765946 2018] [:error] [pid 3395] ERROR django.request Internal Server Error: /dashboard/auth/login/ [Mon Nov 05 21:54:04.765954 2018] [:error] [pid 3395] Traceback (most recent call last): [Mon Nov 05 21:54:04.765956 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner [Mon Nov 05 21:54:04.765958 2018] [:error] [pid 3395] response = get_response(request) [Mon Nov 05 21:54:04.765960 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/utils/deprecation.py", line 140, in __call__ [Mon Nov 05 21:54:04.765961 2018] [:error] [pid 3395] response = self.get_response(request) [Mon Nov 05 21:54:04.765963 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 43, in inner [Mon Nov 05 21:54:04.765964 2018] [:error] [pid 3395] response = response_for_exception(request, exc) [Mon Nov 05 21:54:04.765966 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 93, in response_for_exception [Mon Nov 05 21:54:04.765968 2018] [:error] [pid 3395] response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info()) [Mon Nov 05 21:54:04.765971 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 143, in handle_uncaught_exception [Mon Nov 05 21:54:04.765973 2018] [:error] [pid 3395] return callback(request, **param_dict) [Mon Nov 05 21:54:04.765974 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/utils/decorators.py", line 149, in _wrapped_view [Mon Nov 05 21:54:04.765976 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/views/defaults.py", line 68, in server_error [Mon Nov 05 21:54:04.765978 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loader.py", line 21, in get_template [Mon Nov 05 21:54:04.765979 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/backends/django.py", line 39, in get_template [Mon Nov 05 21:54:04.765981 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/engine.py", line 162, in get_template [Mon Nov 05 21:54:04.765982 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/engine.py", line 136, in find_template [Mon Nov 05 21:54:04.765984 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loaders/base.py", line 38, in get_template [Mon Nov 05 21:54:04.765986 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loaders/filesystem.py", line 28, in get_contents [Mon Nov 05 21:54:04.765987 2018] [:error] [pid 3395] IOError: [Errno 24] Too many open files: '/usr/share/openstack-dashboard/openstack_dashboard/themes/default/templates/500.html' [Mon Nov 05 21:54:04.766227 2018] [:error] [pid 3395] ERROR django.request Internal Server Error: /dashboard/auth/login/ [Mon Nov 05 21:54:04.766234 2018] [:error] [pid 3395] Traceback (most recent call last): [Mon Nov 05 21:54:04.766236 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner [Mon Nov 05 21:54:04.766238 2018] [:error] [pid 3395] response = get_response(request) [Mon Nov 05 21:54:04.766240 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/utils/deprecation.py", line 140, in __call__ [Mon Nov 05 21:54:04.766242 2018] [:error] [pid 3395] response = self.get_response(request) [Mon Nov 05 21:54:04.766243 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 43, in inner [Mon Nov 05 21:54:04.766245 2018] [:error] [pid 3395] response = response_for_exception(request, exc) [Mon Nov 05 21:54:04.766247 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 93, in response_for_exception [Mon Nov 05 21:54:04.766248 2018] [:error] [pid 3395] response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info()) [Mon Nov 05 21:54:04.766252 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 143, in handle_uncaught_exception [Mon Nov 05 21:54:04.766264 2018] [:error] [pid 3395] return callback(request, **param_dict) [Mon Nov 05 21:54:04.766295 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/utils/decorators.py", line 149, in _wrapped_view [Mon Nov 05 21:54:04.766303 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/views/defaults.py", line 68, in server_error [Mon Nov 05 21:54:04.766339 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loader.py", line 21, in get_template [Mon Nov 05 21:54:04.766342 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/backends/django.py", line 39, in get_template [Mon Nov 05 21:54:04.766344 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/engine.py", line 162, in get_template [Mon Nov 05 21:54:04.766346 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/engine.py", line 136, in find_template [Mon Nov 05 21:54:04.766349 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loaders/base.py", line 38, in get_template [Mon Nov 05 21:54:04.766351 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loaders/filesystem.py", line 28, in get_contents [Mon Nov 05 21:54:04.766353 2018] [:error] [pid 3395] IOError: [Errno 24] Too many open files: '/usr/share/openstack-dashboard/openstack_dashboard/themes/default/templates/500.html' [Mon Nov 05 21:54:04.766605 2018] [:error] [pid 3395] ERROR django.request Internal Server Error: /dashboard/auth/login/ [Mon Nov 05 21:54:04.766614 2018] [:error] [pid 3395] Traceback (most recent call last): [Mon Nov 05 21:54:04.766617 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner [Mon Nov 05 21:54:04.766618 2018] [:error] [pid 3395] response = get_response(request) [Mon Nov 05 21:54:04.766627 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/utils/deprecation.py", line 140, in __call__ [Mon Nov 05 21:54:04.766629 2018] [:error] [pid 3395] response = self.get_response(request) [Mon Nov 05 21:54:04.766637 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 43, in inner [Mon Nov 05 21:54:04.766668 2018] [:error] [pid 3395] response = response_for_exception(request, exc) [Mon Nov 05 21:54:04.766671 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 93, in response_for_exception [Mon Nov 05 21:54:04.766673 2018] [:error] [pid 3395] response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info()) [Mon Nov 05 21:54:04.766675 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 143, in handle_uncaught_exception [Mon Nov 05 21:54:04.766677 2018] [:error] [pid 3395] return callback(request, **param_dict) [Mon Nov 05 21:54:04.766678 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/utils/decorators.py", line 149, in _wrapped_view [Mon Nov 05 21:54:04.766680 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/views/defaults.py", line 68, in server_error [Mon Nov 05 21:54:04.766682 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loader.py", line 21, in get_template [Mon Nov 05 21:54:04.766683 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/backends/django.py", line 39, in get_template [Mon Nov 05 21:54:04.766685 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/engine.py", line 162, in get_template [Mon Nov 05 21:54:04.766687 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/engine.py", line 136, in find_template [Mon Nov 05 21:54:04.766688 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loaders/base.py", line 38, in get_template [Mon Nov 05 21:54:04.766690 2018] [:error] [pid 3395] File "/usr/lib/python2.7/site-packages/django/template/loaders/filesystem.py", line 28, in get_contents [Mon Nov 05 21:54:04.766691 2018] [:error] [pid 3395] IOError: [Errno 24] Too many open files: '/usr/share/openstack-dashboard/openstack_dashboard/themes/default/templates/500.html' [Mon Nov 05 21:54:04.766790 2018] [:error] [pid 3395] [remote 192.168.24.254:664] mod_wsgi (pid=3395): Exception occurred processing WSGI script '/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi'. [Mon Nov 05 21:54:04.766804 2018] [:error] [pid 3395] [remote 192.168.24.254:664] Traceback (most recent call last): [Mon Nov 05 21:54:04.766814 2018] [:error] [pid 3395] [remote 192.168.24.254:664] File "/usr/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 157, in __call__ [Mon Nov 05 21:54:04.766825 2018] [:error] [pid 3395] [remote 192.168.24.254:664] response = self.get_response(request) [Mon Nov 05 21:54:04.766829 2018] [:error] [pid 3395] [remote 192.168.24.254:664] File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 124, in get_response [Mon Nov 05 21:54:04.766839 2018] [:error] [pid 3395] [remote 192.168.24.254:664] response = self._middleware_chain(request) [Mon Nov 05 21:54:04.766843 2018] [:error] [pid 3395] [remote 192.168.24.254:664] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 43, in inner [Mon Nov 05 21:54:04.766850 2018] [:error] [pid 3395] [remote 192.168.24.254:664] response = response_for_exception(request, exc) [Mon Nov 05 21:54:04.766854 2018] [:error] [pid 3395] [remote 192.168.24.254:664] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 93, in response_for_exception [Mon Nov 05 21:54:04.766863 2018] [:error] [pid 3395] [remote 192.168.24.254:664] response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info()) [Mon Nov 05 21:54:04.766867 2018] [:error] [pid 3395] [remote 192.168.24.254:664] File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 143, in handle_uncaught_exception [Mon Nov 05 21:54:04.766874 2018] [:error] [pid 3395] [remote 192.168.24.254:664] return callback(request, **param_dict) [Mon Nov 05 21:54:04.766879 2018] [:error] [pid 3395] [remote 192.168.24.254:664] File "/usr/lib/python2.7/site-packages/django/utils/decorators.py", line 149, in _wrapped_view [Mon Nov 05 21:54:04.766892 2018] [:error] [pid 3395] [remote 192.168.24.254:664] File "/usr/lib/python2.7/site-packages/django/views/defaults.py", line 68, in server_error [Mon Nov 05 21:54:04.766904 2018] [:error] [pid 3395] [remote 192.168.24.254:664] File "/usr/lib/python2.7/site-packages/django/template/loader.py", line 21, in get_template [Mon Nov 05 21:54:04.766915 2018] [:error] [pid 3395] [remote 192.168.24.254:664] File "/usr/lib/python2.7/site-packages/django/template/backends/django.py", line 39, in get_template [Mon Nov 05 21:54:04.766932 2018] [:error] [pid 3395] [remote 192.168.24.254:664] File "/usr/lib/python2.7/site-packages/django/template/engine.py", line 162, in get_template [Mon Nov 05 21:54:04.766944 2018] [:error] [pid 3395] [remote 192.168.24.254:664] File "/usr/lib/python2.7/site-packages/django/template/engine.py", line 136, in find_template [Mon Nov 05 21:54:04.766954 2018] [:error] [pid 3395] [remote 192.168.24.254:664] File "/usr/lib/python2.7/site-packages/django/template/loaders/base.py", line 38, in get_template [Mon Nov 05 21:54:04.766965 2018] [:error] [pid 3395] [remote 192.168.24.254:664] File "/usr/lib/python2.7/site-packages/django/template/loaders/filesystem.py", line 28, in get_contents [Mon Nov 05 21:54:04.766979 2018] [:error] [pid 3395] [remote 192.168.24.254:664] IOError: [Errno 24] Too many open files: '/usr/share/openstack-dashboard/openstack_dashboard/themes/default/templates/500.html'
エラーとなるプロセスのファイルディスクリプタを確認する。 open files が 1024 となっている。
[root@controller ~]# ls -l /proc/3395/fd | wc -l 1022 [root@controller ~]# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 15065 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 15065 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited [root@controller ~]#
ユーザごとに使用可能なシステムリソースの制限を確認する。open files がソフトウェアで 1024、ハードウェアで 4096 に制限されていることが確認できる。
ソフトウェアで制限されたリソースを確認
[root@controller ~]# ulimit -Sa core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 15065 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 15065 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited [root@controller ~]#
ハードウェアで制限されたリソースを確認
[root@controller ~]# ulimit -Ha core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 15065 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 4096 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) unlimited cpu time (seconds, -t) unlimited max user processes (-u) 15065 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited [root@controller ~]#
解決方法
プロセスが open しているファイル数がソフトウェアで制限された値に達し、これ以上ファイルを開くことができなくなっていることが原因であるため、httpd サービスのプロセスで open にできるファイル数を拡張する。
httpd サービス の起動用設定ファイルを起動時に上書きするためのディレクトリを作成する。
[root@controller ~]# mkdir /etc/systemd/system/httpd.service.d/ [root@controller ~]# [root@controller ~]# ls /etc/systemd/system/ basic.target.wants local-fs.target.wants dbus-org.fedoraproject.FirewallD1.service multi-user.target.wants default.target network-online.target.wants default.target.wants sockets.target.wants getty.target.wants sysinit.target.wants httpd.service.d system-update.target.wants [root@controller ~]#
作成したディレクトリに上書き用設定ファイルを新規作成する。
[root@controller ~]# vi /etc/systemd/system/httpd.service.d/override.conf
作成したファイルに以下設定を登録する。
[Service] LimitNOFILE=50000
daemonをリロードし設定を反映する。
[root@controller ~]# systemctl daemon-reload [root@controller ~]#
httpd サービスを再起動する。
■ 再起動
[root@controller ~]# systemctl restart httpd
[root@controller ~]#
■ 起動状態を確認
[root@controller ~]# systemctl status httpd
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/httpd.service.d
└─openstack-dashboard.conf
/etc/systemd/system/httpd.service.d
└─override.conf
Active: active (running) since 月 2018-11-05 22:44:40 JST; 6s ago
Docs: man:httpd(8)
man:apachectl(8)
Process: 27594 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
Process: 27630 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py compress --force -v0 (code=exited, status=0/SUCCESS)
Process: 27607 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py collectstatic --noinput --clear -v0 (code=exited, status=0/SUCCESS)
Main PID: 27653 (httpd)
Status: "Processing requests..."
CGroup: /system.slice/httpd.service
├─27653 /usr/sbin/httpd -DFOREGROUND
├─27655 /usr/sbin/httpd -DFOREGROUND
├─27656 /usr/sbin/httpd -DFOREGROUND
├─27657 /usr/sbin/httpd -DFOREGROUND
├─27658 /usr/sbin/httpd -DFOREGROUND
├─27659 (wsgi:keystone- -DFOREGROUND
├─27660 (wsgi:keystone- -DFOREGROUND
├─27661 (wsgi:keystone- -DFOREGROUND
├─27662 (wsgi:keystone- -DFOREGROUND
├─27663 (wsgi:keystone- -DFOREGROUND
├─27664 /usr/sbin/httpd -DFOREGROUND
├─27665 /usr/sbin/httpd -DFOREGROUND
├─27666 /usr/sbin/httpd -DFOREGROUND
├─27667 /usr/sbin/httpd -DFOREGROUND
└─27668 /usr/sbin/httpd -DFOREGROUND
11月 05 22:44:24 controller systemd[1]: Starting The Apache HTTP Server...
11月 05 22:44:40 controller python[27630]: Compressing... done
11月 05 22:44:40 controller python[27630]: Compressed 7 block(s) from 4 tem....
11月 05 22:44:40 controller systemd[1]: Started The Apache HTTP Server.
Hint: Some lines were ellipsized, use -l to show in full.
[root@controller ~]#
httpd サービスで起動しているプロセスを確認する。
[root@controller ~]# ps axf | grep httpd 13800 pts/1 S+ 0:00 \_ grep --color=auto httpd 27653 ? Ss 0:00 /usr/sbin/httpd -DFOREGROUND 27655 ? Sl 0:00 \_ /usr/sbin/httpd -DFOREGROUND 27656 ? Sl 0:00 \_ /usr/sbin/httpd -DFOREGROUND 27657 ? Sl 0:00 \_ /usr/sbin/httpd -DFOREGROUND 27658 ? Sl 0:00 \_ /usr/sbin/httpd -DFOREGROUND 27659 ? Sl 0:01 \_ (wsgi:keystone- -DFOREGROUND 27660 ? Sl 0:00 \_ (wsgi:keystone- -DFOREGROUND 27661 ? Sl 0:00 \_ (wsgi:keystone- -DFOREGROUND 27662 ? Sl 0:00 \_ (wsgi:keystone- -DFOREGROUND 27663 ? Sl 0:00 \_ (wsgi:keystone- -DFOREGROUND 27664 ? S 0:00 \_ /usr/sbin/httpd -DFOREGROUND 27665 ? S 0:00 \_ /usr/sbin/httpd -DFOREGROUND 27666 ? S 0:00 \_ /usr/sbin/httpd -DFOREGROUND 27667 ? S 0:00 \_ /usr/sbin/httpd -DFOREGROUND 27668 ? S 0:00 \_ /usr/sbin/httpd -DFOREGROUND 27723 ? S 0:00 \_ /usr/sbin/httpd -DFOREGROUND [root@controller ~]#
httpd サービスのプロセスのファイルディスクリプタを確認する。 open files が 50000 に拡張されていることが確認できる。
[root@controller ~]# more /proc/27653/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15065 15065 processes Max open files 50000 50000 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15065 15065 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us [root@controller ~]#
補足
Openstack Horizon では、アクセスするたびに プロセスで open するファイル数がリセットされずにどんどん増加してしまう問題（バグ）があるようである。そのため、open できるファイル数を拡張してもいづれは制限値に達し同事象が発生してしまうため、上記は根本解決ではない。 httpd サービスを再起動すれば、 open できるファイル数がリセットできるため、一日一回 httpd サービスを再起動するなど運用でカバーする必要がありそうである。
下記は、アクセスするたびに open するファイル数が増加していく状態が確認できる。
[root@controller ~]# ls -l /proc/27658/fd | wc -l 32 [root@controller ~]# [root@controller ~]# ls -l /proc/27658/fd | wc -l 41 [root@controller ~]# [root@controller ~]# ls -l /proc/27658/fd | wc -l 206 [root@controller ~]#