Apache Considerations UNIX®, Linux, and Mac OS X
This appendix contains information that pertains to the recommended option for UNIX®, Linux, and Mac OS X (“Recommended Option: NSAPI Modules (CSPn3.so)”) and atypical option 1 (“Alternative Option 1: Apache API Module with NSD (mod_csp24.so)”.
Apache Process Management and Capacity Planning
Apache provides two process management modules for UNIX-based operating systems. In this architecture, the Gateway modules are directly bound to the Apache worker processes. Therefore, the way Apache is configured to manage its process pool has a direct affect on the Gateway.
Apache implements its two process management models as Multi-Processing Modules (or MPMs).
Prefork MPM. This is the traditional multi-process (UNIX®) server architecture. It does not use threads and, as a result, there is no requirement that third-party API modules (DSOs) should be thread-safe. Reference: http://httpd.apache.org/docs/2.2/mod/prefork.html
Worker MPM. This is the newer hybrid multithread/multi-process server architecture. It does use threads and all third-party API modules (DSOs) used should be thread-safe. Reference: http://httpd.apache.org/docs/2.2/mod/worker.html.
In order to determine which of the two server models is in use for an existing installation, call the Apache executable directly but qualified as follows:
The MPM in use will be listed as one of the following:
Server MPM: Prefork
Server MPM: Worker
Two further (and related) listings are provided.
httpd –l List all modules built-in to the server (including the choice of MPM):
httpd –L List all modules and related configuration directives:
The Gateway DSOs are thread-safe and can be deployed in either server model. A useful guide for Apache tuning can be found here:http://httpd.apache.org/docs/2.2/misc/perf-tuning.html
Security. The parent process (of both server architectures) is usually started from an account with superuser privileges assigned (root under UNIX) in order to bind to TCP port 80. The child processes launched by Apache run as a lesser-privileged user. The User and Group directives (in the Apache configuration) are used to set the privileges of the Apache child processes. The child processes must be able to read all the content that they are responsible for serving (and have read/write access to the Gateway’s configuration and Event Log files), but, beyond this, should be granted as few privileges as possible.
A brief description of the two Apache MPMs follows. Refer to the Apache documentation (see links above) for further information.
Apache Prefork MPM
The traditional multi-process server architecture.
A single core Apache process (the parent) is responsible for launching child processes. The child processes are responsible for listening on the appropriate TCP port (usually 80), accepting incoming requests from clients, reading the request data and forwarding to whichever module can service the request and generate a response. The latter part of this request processing cycle is in accordance with directives in the Apache configuration file. For example, requests for CSP resources are forwarded to the modules configured for this purpose.
The management of child processes is governed by the following configuration parameters.
StartServers (Defaults to 5): This directive sets the number of child server processes created on startup. As the number of processes is dynamically controlled depending on the load, there is usually little reason to adjust this parameter.
MinSpareServers (Defaults to 10): The MinSpareServers directive sets the desired minimum number of idle child server processes. An idle process is one which is not handling a request. If there are fewer than MinSpareServers idle, then the parent process creates new children at a maximum rate of 1 per second.
MaxSpareServers (Defaults to 5): This directive sets the desired maximum number of idle child server processes. An idle process is one which is not handling a request. If there are more than MaxSpareServers idle, then the parent process kills off the excess processes.
MaxClients (Defaults to 256): This directive sets the limit on the number of simultaneous requests that are served. Any connection attempts over the MaxClients limit are normally queued, up to a number based on the ListenBacklog directive. Once a child process is freed at the end of a different request, the connection is then serviced. For non-threaded servers, MaxClients translates into the maximum number of child processes that are launched to serve requests. To increase this parameter, ServerLimit must also be raised.
ServerLimit (Defaults to 5): This directive sets the maximum configured value for MaxClients for the lifetime of the Apache instance.
MaxRequestsPerChild (Defaults to 10000): This directive sets the limit on the number of requests that an individual child server process handles. After MaxRequestsPerChild requests, the child process dies. If MaxRequestsPerChild is 0, then the process never expires.
In general, Apache is self-regulating, so most sites do not need to adjust these directives from their default values. Sites which need to serve more than 256 simultaneous requests may need to increase the value of MaxClients, while sites with limited memory may need to decrease MaxClients to prevent the server from thrashing (swapping memory to disk and back).
Apache Worker MPM
The hybrid multithread/multi-process server architecture.
A single core Apache process (the parent) is responsible for launching child processes. Each child process creates a fixed number of server threads as specified in the ThreadsPerChild directive, in addition to a listener thread which listens for connections and passes them to a server thread for processing when they arrive.
By using threads to serve requests, the Apache server instance is able to serve a large number of requests with fewer system resources than a process-based server. However, it retains much of the stability of a process-based server by keeping multiple processes available, each with many threads.
Apache always tries to maintain a pool of spare or idle server threads, which stand ready to serve incoming requests. This way, clients do not need to wait for new threads or processes to be created before their requests can be served. The following parameters are provided to control this mechanism.
StartServers (Defaults to 3): This directive sets the number of child server processes created on startup. As the number of processes is dynamically controlled depending on the load, there is usually little reason to adjust this parameter.
ThreadsPerChild (Defaults to 25): This directive sets the number of threads created by each child process.
MinSpareThreads (Defaults to 75): This directive sets the minimum number of idle threads to handle request spikes. If there aren't enough idle threads in the server then child processes are created until the number of idle threads is greater than the number specified here.
MaxSpareThreads (Defaults to 250): This directive sets the maximum number of idle threads. The number of idle threads is considered on a server-wide basis. If there are too many idle threads in the server then child processes are killed until the number of idle threads is less than this number.
MaxClients: This directive restricts the total number of threads that are available to serve clients. The default value is 16 (ServerLimit) multiplied by the value of 25 (ThreadsPerChild). Therefore, to increase MaxClients to a value that requires more than 16 processes, the value of ServerLimit must be raised.
ServerLimit (Defaults to 16): Use this directive only if the MaxClients and ThreadsPerChild settings require more than 16 server processes (default). Do not set the value of this directive any higher than the number of server processes required to honor the requirements specified in the MaxClients and ThreadsPerChild parameters.
ThreadLimit (Defaults to 64): This directive sets the maximum configured value for ThreadsPerChild for the lifetime of the Apache process. Any attempts to change this directive during a restart are ignored, but ThreadsPerChild can be modified during a restart up to the value of this directive.
MaxRequestsPerChild (Defaults to 10000): This directive sets the limit on the number of requests that an individual child server process can handle. After MaxRequestsPerChild requests, the child process dies and a new process is created (a recycle event). If MaxRequestsPerChild is 0, then the process never expires.
The number of child (or worker) processes that are initially launched is set by the StartServers directive. Then during operation, Apache assesses the total number of idle threads in all processes, and forks or kills processes to keep this number within the boundaries specified by MinSpareThreads and MaxSpareThreads. Since this process is self-regulating, it is rarely necessary to modify these directives from their default values. The maximum number of clients that may be served simultaneously (i.e., the maximum total number of threads in all processes) is determined by the MaxClients directive. The maximum number of active child processes is determined by the MaxClients directive divided by the ThreadsPerChild directive.
Two directives set hard limits on the number of active child processes and the number of server threads in a child process, and can only be changed by fully stopping the server and then starting it again. ServerLimit is a hard limit on the number of active child processes, and must be greater than or equal to the MaxClients directive divided by the ThreadsPerChild directive. ThreadLimit is a hard limit of the number of server threads, and must be greater than or equal to the ThreadsPerChild directive. If non-default values are specified for these directives, they should appear before other worker directives.
In addition to the set of active child processes, there may be additional child processes which are terminating but where at least one server thread is still handling an existing client connection. Up to MaxClients terminating processes may be present, though the actual number can be expected to be much smaller. This behavior can be avoided by disabling the termination of individual child processes as follows:
Set the value of MaxRequestsPerChild to zero
Set the value of MaxSpareThreads to the same value as MaxClients
Apache MPMs and the Gateway DSOs
The Gateway DSOs are thread-safe and can be deployed in either server model.
For both MPMs the StartServers directive specifies the number of child (worker) processes to start. This directive also indicates the number of instances of the CSP Gateway DSOs that can be present – such as one per Apache child process.
Both MPMs involve spreading the load over multiple child (worker) processes and different versions of the Gateway behave differently:
In Caché version 2010.2 and earlier: Each Gateway instance operates independently. Each instance manages its own running configuration, connection table and form cache. The contents of the Gateway System Status form change with every refresh. This is because the Status displayed is that of the Gateway instance that happens to be returning the result.
In Caché version 2011.1 and later: Although each Gateway instance is independently loaded by each and every Apache child process, the running configuration, connection table and form cache is held in a shared memory sector. The contents of the Gateway System Status form remain constant with every refresh (apart from changes happening as a result of activity updates, of course). The connection table (and connection numbers) displayed is common to the whole Apache instance and, because of this, an additional column indicating the web server process ID to which each Caché connection is associated is included.
The Gateway load is spread over multiple web servers processes and this has an effect on the following Gateway configuration parameter:
This parameter allows you to effectively limit the number of connections that the Gateway can make to a particular Caché server. It is a throttle. In setting this parameter bear in mind that the value applies to each web server worker process and not to the web server installation taken as a whole.
For example, if the Maximum Server Connections parameter is set to 10 and the hosting Apache server starts 5 worker processes then the total number of connections that the Gateway can theoretically create is 50 (10x5).
This is the simplistic view. However, the effect of the maximum connections throttle is further influenced by the choice of MPM. The Perfork MPM is straightforward: since threads are not used, each Apache worker process (Gateway instance) can only possibly create one connection to Caché. Each worker process can only serve one request at a time. Also bear in mind the effect of the MaxClients Apache setting: Apache does not attempt to simultaneously serve more than the number of requests specified here. The Gateway maximum connections throttle, therefore, has no effect when used with the Prefork MPM since it can only have a possible value of one.
The Worker MPM is slightly more complicated since each Apache worker process can start many threads. In theory the total number of connections that can be made to Caché for the whole Gateway installation is the maximum number of server processes (ServerLimit) multiplied by the number of threads per process (ThreadsPerChild) up to the limits imposed by the MaxClients directive. Also bear in mind the ceiling on the number of threads imposed by the ThreadLimit setting.
In the Gateway configuration, the limit on the number of connections imposed by the Maximum Server Connections directive applies to each individual Apache worker process. A value higher than the maximum number of threads allowed per process (ThreadsPerChild) has little effect since Apache cannot allocate more concurrent work than can be accommodated by the number of threads available in each process. Setting Maximum Server Connections to a value lower than the number of threads allowed per process (ThreadsPerChild) potentially leads to queuing in the Gateway modules since Apache can potentially allocate more work to each worker process than can be handled by the number of allowed connections to Caché. Such a configuration can potentially lead to congestion in the Apache environment so care should therefore be taken.
For installations where most of the Apache workload is made up of CSP requests, it is better to not assign a value to the Gateway’s Maximum Server Connections directive and control the amount of concurrent work that can be done (and by implication the number of connections to Caché) with the corresponding Apache configuration parameters.
Setting an independent value for the Gateway’s Maximum Server Connections directive would, however, make sense in installations where CSP requests only represent part of the workload for the Apache installation as a whole.
State-Aware Sessions (Preserve mode 1)
Support for state-aware sessions in a web server that distributes load over multiple worker processes relies on InterProcess Communications (IPC) protocols to manage the routing of requests between individual worker processes. Operating in this web server architecture, the Gateway has no control over which worker process will handle any particular request.
The Gateway uses UNIX domain sockets for its IPC protocol and the method for supporting state-aware sessions is described below.
As an example, consider a web server installation that distributes its load over 3 worker processes: P1, P2 and P3. Each worker process can potentially start any number of threads (T1, T2 … Tn) according to the web server MPM and configuration in use.
Suppose an application makes a request to mark its session as state-aware (preserve mode 1) and the Gateway acknowledges this instruction in process P2. The connection and (security context) to the, now private Caché process is hosted by web server worker process P2. All further requests for that user/session must now be processed by worker process P2. However, the Gateway has no control over which worker process the web server will route subsequent requests to so the Gateway must establish an IPC channel between P2 and (potentially) any other worker process in the set.
When the Gateway marks the connection as state-aware in P2 it starts a listening service in a separate, detached, thread. If Gateway Log Level v2 is in use, a message similar to the one shown below is written to the Event Log.
IPC Server Process ID: 28457 Listening on Domain Socket: /tmp/csp28457.str
Now, let’s say a further request for the same session is processed by worker process P3. The Gateway forwards that request to process P2 via the IPC channel previously established and wait for the response. If Log Level v2 is in use then a message similar to the one shown below is recorded:
Route request over IPC to another web server process PrivateSession=2; pid_self=28456; ipc_to_pid=28457;
Of course, if a request for the session happens to be routed by the web server directly to P2 then no further routing is necessary in the Gateway environment since P2 is hosting the session’s private connection.
If, for whatever reason, the Gateway is unable to connect and forward a request to a previously created IPC channel, one of the following messages is recorded depending on the context in which the error was raised:
IPC CLIENT: Error Cannot connect
IPC CLIENT: Error Cannot send request
The most common reason for problems in this area is if Apache has closed (or recycled) a worker process (in the case of the example: P2). Of course, a process can crash (for example with an access violation/SIGSEGV error) and, in this case, an error message will probably be reported in the Apache error log.
Apache also, by default, periodically recycles worker processes.
If you use state-aware sessions, configure Apache such that it doesn’t recycle worker processes by configuring the installation as follows.
Set the value of MaxRequestsPerChild to zero
Set the value of MaxSpareThreads to the same value as MaxClients
If, for whatever reason, it is not possible to prevent Apache periodically recycling processes (perhaps as a result of a malfunctioning module) and state-aware sessions must be used, then an NSD based Gateway configuration can be used. An NSD-based architecture avoids the problems discussed above because it effectively separates the process management of the Gateway from the web server. Options for using the Gateway Network Service Daemon (NSD) are covered in later sections.