CSP Gateway Configuration Guide
Apache Considerations UNIX®, Linux, and Mac OS X
[Home] [Back] [Next]
InterSystems: The power behind what matters   
Class Reference   
Search:    

This appendix contains information that pertains to the recommended option for UNIX®, Linux, and Mac OS X (Recommended Option: NSAPI Modules (CSPn3.so)) and atypical option 1 (Alternative Option 1: Apache API Module with NSD (mod_csp24.so).

Apache Process Management and Capacity Planning
Apache provides two process management modules for UNIX-based operating systems. In this architecture, the Gateway modules are directly bound to the Apache worker processes. Therefore, the way Apache is configured to manage its process pool has a direct affect on the Gateway.
Apache implements its two process management models as Multi-Processing Modules (or MPMs).
Prefork MPM. This is the traditional multi-process (UNIX®) server architecture. It does not use threads and, as a result, there is no requirement that third-party API modules (DSOs) should be thread-safe. Reference: http://httpd.apache.org/docs/2.2/mod/prefork.html
Worker MPM. This is the newer hybrid multithread/multi-process server architecture. It does use threads and all third-party API modules (DSOs) used should be thread-safe. Reference: http://httpd.apache.org/docs/2.2/mod/worker.html.
In order to determine which of the two server models is in use for an existing installation, call the Apache executable directly but qualified as follows:
httpd -V 
The MPM in use will be listed as one of the following:
Server MPM:     Prefork
Or:
Server MPM:     Worker
Two further (and related) listings are provided.
httpd –l List all modules built-in to the server (including the choice of MPM):
httpd –L List all modules and related configuration directives:
The Gateway DSOs are thread-safe and can be deployed in either server model. A useful guide for Apache tuning can be found here:http://httpd.apache.org/docs/2.2/misc/perf-tuning.html
Security. The parent process (of both server architectures) is usually started from an account with superuser privileges assigned (root under UNIX) in order to bind to TCP port 80. The child processes launched by Apache run as a lesser-privileged user. The User and Group directives (in the Apache configuration) are used to set the privileges of the Apache child processes. The child processes must be able to read all the content that they are responsible for serving (and have read/write access to the Gateway’s configuration and Event Log files), but, beyond this, should be granted as few privileges as possible.
A brief description of the two Apache MPMs follows. Refer to the Apache documentation (see links above) for further information.
Apache Prefork MPM
The traditional multi-process server architecture.
A single core Apache process (the parent) is responsible for launching child processes. The child processes are responsible for listening on the appropriate TCP port (usually 80), accepting incoming requests from clients, reading the request data and forwarding to whichever module can service the request and generate a response. The latter part of this request processing cycle is in accordance with directives in the Apache configuration file. For example, requests for CSP resources are forwarded to the modules configured for this purpose.
The management of child processes is governed by the following configuration parameters.
In general, Apache is self-regulating, so most sites do not need to adjust these directives from their default values. Sites which need to serve more than 256 simultaneous requests may need to increase the value of MaxClients, while sites with limited memory may need to decrease MaxClients to prevent the server from thrashing (swapping memory to disk and back).
Apache Worker MPM
The hybrid multithread/multi-process server architecture.
A single core Apache process (the parent) is responsible for launching child processes. Each child process creates a fixed number of server threads as specified in the ThreadsPerChild directive, in addition to a listener thread which listens for connections and passes them to a server thread for processing when they arrive.
By using threads to serve requests, the Apache server instance is able to serve a large number of requests with fewer system resources than a process-based server. However, it retains much of the stability of a process-based server by keeping multiple processes available, each with many threads.
Apache always tries to maintain a pool of spare or idle server threads, which stand ready to serve incoming requests. This way, clients do not need to wait for new threads or processes to be created before their requests can be served. The following parameters are provided to control this mechanism.
The number of child (or worker) processes that are initially launched is set by the StartServers directive. Then during operation, Apache assesses the total number of idle threads in all processes, and forks or kills processes to keep this number within the boundaries specified by MinSpareThreads and MaxSpareThreads. Since this process is self-regulating, it is rarely necessary to modify these directives from their default values. The maximum number of clients that may be served simultaneously (i.e., the maximum total number of threads in all processes) is determined by the MaxClients directive. The maximum number of active child processes is determined by the MaxClients directive divided by the ThreadsPerChild directive.
Two directives set hard limits on the number of active child processes and the number of server threads in a child process, and can only be changed by fully stopping the server and then starting it again. ServerLimit is a hard limit on the number of active child processes, and must be greater than or equal to the MaxClients directive divided by the ThreadsPerChild directive. ThreadLimit is a hard limit of the number of server threads, and must be greater than or equal to the ThreadsPerChild directive. If non-default values are specified for these directives, they should appear before other worker directives.
In addition to the set of active child processes, there may be additional child processes which are terminating but where at least one server thread is still handling an existing client connection. Up to MaxClients terminating processes may be present, though the actual number can be expected to be much smaller. This behavior can be avoided by disabling the termination of individual child processes as follows:
  1. Set the value of MaxRequestsPerChild to zero
  2. Set the value of MaxSpareThreads to the same value as MaxClients
Apache MPMs and the Gateway DSOs
The Gateway DSOs are thread-safe and can be deployed in either server model.
For both MPMs the StartServers directive specifies the number of child (worker) processes to start. This directive also indicates the number of instances of the CSP Gateway DSOs that can be present – such as one per Apache child process.
Both MPMs involve spreading the load over multiple child (worker) processes and different versions of the Gateway behave differently:
The Gateway load is spread over multiple web servers processes and this has an effect on the following Gateway configuration parameter:
Maximum Server Connections
This parameter allows you to effectively limit the number of connections that the Gateway can make to a particular Caché server. It is a throttle. In setting this parameter bear in mind that the value applies to each web server worker process and not to the web server installation taken as a whole.
For example, if the Maximum Server Connections parameter is set to 10 and the hosting Apache server starts 5 worker processes then the total number of connections that the Gateway can theoretically create is 50 (10x5).
This is the simplistic view. However, the effect of the maximum connections throttle is further influenced by the choice of MPM. The Perfork MPM is straightforward: since threads are not used, each Apache worker process (Gateway instance) can only possibly create one connection to Caché. Each worker process can only serve one request at a time. Also bear in mind the effect of the MaxClients Apache setting: Apache does not attempt to simultaneously serve more than the number of requests specified here. The Gateway maximum connections throttle, therefore, has no effect when used with the Prefork MPM since it can only have a possible value of one.
The Worker MPM is slightly more complicated since each Apache worker process can start many threads. In theory the total number of connections that can be made to Caché for the whole Gateway installation is the maximum number of server processes (ServerLimit) multiplied by the number of threads per process (ThreadsPerChild) up to the limits imposed by the MaxClients directive. Also bear in mind the ceiling on the number of threads imposed by the ThreadLimit setting.
In the Gateway configuration, the limit on the number of connections imposed by the Maximum Server Connections directive applies to each individual Apache worker process. A value higher than the maximum number of threads allowed per process (ThreadsPerChild) has little effect since Apache cannot allocate more concurrent work than can be accommodated by the number of threads available in each process. Setting Maximum Server Connections to a value lower than the number of threads allowed per process (ThreadsPerChild) potentially leads to queuing in the Gateway modules since Apache can potentially allocate more work to each worker process than can be handled by the number of allowed connections to Caché. Such a configuration can potentially lead to congestion in the Apache environment so care should therefore be taken.
For installations where most of the Apache workload is made up of CSP requests, it is better to not assign a value to the Gateway’s Maximum Server Connections directive and control the amount of concurrent work that can be done (and by implication the number of connections to Caché) with the corresponding Apache configuration parameters.
Setting an independent value for the Gateway’s Maximum Server Connections directive would, however, make sense in installations where CSP requests only represent part of the workload for the Apache installation as a whole.
State-Aware Sessions (Preserve mode 1)
Support for state-aware sessions in a web server that distributes load over multiple worker processes relies on InterProcess Communications (IPC) protocols to manage the routing of requests between individual worker processes. Operating in this web server architecture, the Gateway has no control over which worker process will handle any particular request.
The Gateway uses UNIX domain sockets for its IPC protocol and the method for supporting state-aware sessions is described below.
As an example, consider a web server installation that distributes its load over 3 worker processes: P1, P2 and P3. Each worker process can potentially start any number of threads (T1, T2 … Tn) according to the web server MPM and configuration in use.
Suppose an application makes a request to mark its session as state-aware (preserve mode 1) and the Gateway acknowledges this instruction in process P2. The connection and (security context) to the, now private Caché process is hosted by web server worker process P2. All further requests for that user/session must now be processed by worker process P2. However, the Gateway has no control over which worker process the web server will route subsequent requests to so the Gateway must establish an IPC channel between P2 and (potentially) any other worker process in the set.
When the Gateway marks the connection as state-aware in P2 it starts a listening service in a separate, detached, thread. If Gateway Log Level v2 is in use, a message similar to the one shown below is written to the Event Log.
IPC Server 
Process ID: 28457 Listening on Domain Socket: /tmp/csp28457.str
Now, let’s say a further request for the same session is processed by worker process P3. The Gateway forwards that request to process P2 via the IPC channel previously established and wait for the response. If Log Level v2 is in use then a message similar to the one shown below is recorded:
Route request over IPC to another web server process
PrivateSession=2; pid_self=28456; ipc_to_pid=28457; 
Of course, if a request for the session happens to be routed by the web server directly to P2 then no further routing is necessary in the Gateway environment since P2 is hosting the session’s private connection.
If, for whatever reason, the Gateway is unable to connect and forward a request to a previously created IPC channel, one of the following messages is recorded depending on the context in which the error was raised:
IPC CLIENT: Error
Cannot connect
Or:
IPC CLIENT: Error
Cannot send request
The most common reason for problems in this area is if Apache has closed (or recycled) a worker process (in the case of the example: P2). Of course, a process can crash (for example with an access violation/SIGSEGV error) and, in this case, an error message will probably be reported in the Apache error log.
Apache also, by default, periodically recycles worker processes.
If you use state-aware sessions, configure Apache such that it doesn’t recycle worker processes by configuring the installation as follows.
If, for whatever reason, it is not possible to prevent Apache periodically recycling processes (perhaps as a result of a malfunctioning module) and state-aware sessions must be used, then an NSD based Gateway configuration can be used. An NSD-based architecture avoids the problems discussed above because it effectively separates the process management of the Gateway from the web server. Options for using the Gateway Network Service Daemon (NSD) are covered in later sections.