Apache Considerations (UNIX®/Linux/macOS)
This article contains information about the recommended option for UNIX®, Linux, and macOS (Recommended Option: NSAPI Modules (CSPn3.so)) and atypical option 1 (Alternative Option 1: Apache API Module with NSD (mod_csp24.so)).
Apache Process Management and Capacity Planning
Apache provides three process management modules for UNIX® operating systems. In this architecture, the InterSystems Web Gateway modules are directly bound to the Apache worker processes. Therefore, the way Apache is configured to manage its process pool has a direct effect on the Web Gateway.
Apache implements each of its process management models as a Multi-Processing Module (MPM).
Prefork MPM is the traditional multi-process (UNIX®) server architecture. It does not use threads and, as a result, there is no requirement that third-party API modules (DSOs) should be thread-safe. Reference: http://httpd.apache.org/docs/current/mod/prefork.html.
Worker MPM is the newer hybrid multithread/multi-process server architecture. It does use threads and all third-party API modules (DSOs) used should be thread-safe. Reference: http://httpd.apache.org/docs/current/mod/worker.html.
Event MPM is designed to allow more requests to be served simultaneously by passing off some processing work to the listener threads, freeing up the worker threads to serve new requests. Reference: http://httpd.apache.org/docs/current/mod/event.html.
In order to determine which of the server models is in use for an existing installation, call the Apache executable directly but qualified as follows:
Two further related listings are provided:
httpd –l List all modules built-in to the server
httpd –L List all modules and related configuration directives
The Web Gateway DSOs are thread-safe and can be deployed in any server model. A useful guide for Apache tuning can be found here: http://httpd.apache.org/docs/current/misc/perf-tuning.html.
The parent process of all three server architectures is usually started from an account with superuser privileges assigned (root under UNIX®) in order to bind to TCP port 80. The child processes launched by Apache run as a lesser-privileged user. The User and Group directives (in the Apache configuration) are used to set the privileges of the Apache child processes. The child processes must be able to read all the content that they are responsible for serving (and have read/write access to the Web Gateway’s configuration and Event Log files), but, beyond this, should be granted as few privileges as possible. Refer to the Apache documentation for further information.
Apache MPMs and the Web Gateway DSOs
The Web Gateway DSOs are thread-safe and can be deployed in any server model.
For all MPMs the StartServers directive specifies the number of child (worker) processes to start. This directive also indicates the number of instances of the Web Gateway DSOs that can be present – such as one per Apache child process.
All MPMs involve spreading the load over multiple child (worker) processes.
Although each Gateway instance is independently loaded by each and every Apache child process, the running configuration, connection table and form cache is held in a shared memory sector. The contents of the Web Gateway System Status form remain constant with every refresh (apart from changes happening as a result of activity updates, of course). The connection table (and connection numbers) displayed is common to the whole Apache instance and, because of this, an additional column indicating the web server process ID to which each InterSystems IRIS connection is associated is included.
Maximum Server Connections
The Web Gateway load is spread over multiple web servers processes and this has an effect on the Maximum Server Connections configuration parameter. This parameter allows you to effectively limit the number of connections that the Web Gateway can make to a particular InterSystems IRIS server. It is a throttle. In setting this parameter, remember that the value applies to each web server worker process and not to the web server installation taken as a whole.
For example, if the Maximum Server Connections parameter is set to 10 and the hosting Apache server starts 5 worker processes then the total number of connections that the Web Gateway can theoretically create is 50 (10x5).
This is the simplistic view. However, the effect of the maximum connections throttle is further influenced by the choice of MPM. The Perfork MPM is straightforward: since threads are not used, each Apache worker process (Gateway instance) can only possibly create one connection to InterSystems IRIS. Each worker process can only serve one request at a time. Also bear in mind the effect of the MaxRequestWorkers Apache setting: Apache does not attempt to simultaneously serve more than the number of requests specified here. The Web Gateway maximum connections throttle, therefore, is unavailable when used with the Prefork MPM since it can only have a possible value of one
The Worker MPM is slightly more complicated since each Apache worker process can start many threads. In theory the total number of connections that can be made to InterSystems IRIS for the whole Gateway installation is the maximum number of server processes (ServerLimit) multiplied by the number of threads per process (ThreadsPerChild) up to the limits imposed by the MaxRequestWorkers directive. Also bear in mind the ceiling on the number of threads imposed by the ThreadLimit setting.
In the Web Gateway configuration, the limit on the number of connections imposed by the Maximum Server Connections directive applies to each individual Apache worker process. A value higher than the maximum number of threads allowed per process (ThreadsPerChild) has little effect since Apache cannot allocate more concurrent work than can be accommodated by the number of threads available in each process. Setting Maximum Server Connections to a value lower than the number of threads allowed per process (ThreadsPerChild) potentially leads to queuing in the Web Gateway modules since Apache can potentially allocate more work to each worker process than can be handled by the number of allowed connections to InterSystems IRIS. Such a configuration can potentially lead to congestion in the Apache environment so care should therefore be taken.
For installations where most of the Apache workload is made up of InterSystems file types, it is better to not assign a value to the Web Gateway’s Maximum Server Connections directive and control the amount of concurrent work that can be done (and by implication the number of connections to InterSystems IRIS) with the corresponding Apache configuration parameters.
Setting an independent value for the Web Gateway’s Maximum Server Connections directive would, however, make sense in installations where InterSystems file types represent only part of the workload for the Apache installation as a whole.
State-Aware Sessions (Preserve mode 1)
Support for state-aware sessions in a web server that distributes load over multiple worker processes relies on InterProcess Communications (IPC) protocols to manage the routing of requests between individual worker processes. Operating in this web server architecture, the Web Gateway has no control over which worker process handles any particular request.
The Web Gateway uses UNIX® domain sockets for its IPC protocol and the method for supporting state-aware sessions is described below.
As an example, consider a web server installation that distributes its load over 3 worker processes: P1, P2 and P3. Each worker process can potentially start any number of threads (T1, T2 … Tn) according to the web server MPM and configuration in use.
Suppose an application makes a request to mark its session as state-aware (preserve mode 1) and the Web Gateway acknowledges this instruction in process P2. The connection and (security context) to the now private InterSystems IRIS process is hosted by web server worker process P2. All further requests for that user/session must now be processed by worker process P2. However, the Web Gateway has no control over which worker process the web server routes subsequent requests to, so the Web Gateway must establish an IPC channel between P2 and (potentially) any other worker process in the set.
When the Web Gateway marks the connection as state-aware in P2, it starts a listening service in a separate, detached, thread. For log level v2, a message similar to the one shown below is written to the Event Log.
IPC Server Process ID: 28457 Listening on Domain Socket: /tmp/csp28457.str
Now, say a further request for the same session is processed by worker process P3. The Web Gateway forwards that request to process P2 via the IPC channel previously established and waits for the response. For log level v2, a message similar to the one shown below is recorded:
Route request over IPC to another web server process PrivateSession=2; pid_self=28456; ipc_to_pid=28457;
Of course, if a request for the session happens to be routed by the web server directly to P2, then no further routing is necessary in the Web Gateway environment, since P2 is hosting the session’s private connection.
If the Web Gateway is unable to connect and forward a request to a previously created IPC channel, one of the following messages is recorded depending on the context in which the error was raised:
IPC CLIENT: Error Cannot connect
IPC CLIENT: Error Cannot send request
The most common reason for problems in this area is if Apache has closed (or recycled) a worker process (in the case of the example, P2). Of course, a process can crash (for example, with an access violation/SIGSEGV error) and, in this case, an error message is probably reported in the Apache error log.
Apache also, by default, periodically recycles worker processes.
If you use state-aware sessions, configure Apache such that it doesn’t recycle worker processes by configuring the installation as follows.
Set the value of MaxConnectionsPerChild to zero
Set the value of MaxSpareThreads to the same value as MaxRequestWorkers
If it is not possible to prevent Apache periodically recycling processes (perhaps as a result of a malfunctioning module) and state-aware sessions must be used, then an NSD based Gateway configuration can be used. An NSD-based architecture avoids the problems discussed above because it effectively separates the process management of the Web Gateway from the web server. Options for using the Web Gateway’s network service daemon (NSD) are covered in Using the NSD on Microsoft Windows and Using the NSD on UNIX®, Linux, and macOS.