How is it that, with as much caffieine as we ingest on a daily basis, sleepy still wins in a Google Fight?

The Ann Arbor Perl Mongers are a great, and still building steam, group of guys (no girls yet) that love perl. Dan and I went down to Ann Arbor for April’s meeting, and had a great time.

In April:

* Tommy Staton gave a presentation on ‘Syncing your ~ with Git’. It was well done, and somewhat controversial, and I learned a TON about Git, which is kind of awesome. Too, he used Voom (which allows you to write and show presentations in Vim) to write his presentation, and with my love of Vim, it was easy to like him.

* Jamie Pitts – gave a presentation about “Boilerplate Web Apps” (downloads his presentation) about Catalyst::Helper, which was incredibly detailed and quite revolutionary for me but… went immediately over my head. As a side note, I really REALLY need to learn Perl (which, incidentally, apparently is three classes deep in the Computer Sciences department at LCC… JUST GIVE ME A DAMN CLASS ON PERL)

This month Liquid Web (by which I mean me) offered to host the meeting. I asked our devteam if they wanted to present anything, and, to put it simply, they are far too busy with things I want them to be doing for LW. It was coming down to it, and it was becoming bloody apparent: I was going to have to present something. To the Perl Mongers.

Me. Who doesn’t know perl. ….shiiiiiiii.

But then my (quickly becoming good) buddy Mark suggested I talk about what I do know: why Perl Enthusiasts need to be their own Evangelists. I was 3/4 of the way through building that when Jamie (from the last meeting, remember?) and Mike Mol both offered to present, and my lead dev tells me that this topic was covered three months ago, at the same group.

SAVED!

Organize the event?! I can do that in my sleep.

Pizza, soda, technology, and places to sit. That’s all I had to come up with.

This Month:

* Jamie presented on the Architecture of a well written, multi-layer (and sometimes multi-application) perl application. It definitely engaged the LW devs and Engineering folk that made it out. This time I actually followed along with the concepts, but as soon as he started giving examples he lost me (have I mentioned how it would be a good idea to learn perl?). It was a good presentation, though.

* Mike Mol talked about his extremely cool project: Rosetta Code, which is exactly what you’d think it was. The base idea: write the same function/program/concept in as many different languages as you can. While relatively simple in concept, it is a huge undertaking. He’s grown his own community, and faced his challenges, and it was a joy watching him interact with my beloved engineering geeks.

After we broke (best two hours ever), I took the four out-of-towners on a tour of LW, showing them the one section of servers, and the huge generators, and the Awe-Inspiring that is the Datacenter I typically take for granted. They had questions about all sorts of statistics I haven’t committed to memory yet (Mental note: learn about the electrical infrastructure), but it was a spectacular end to the night.

All in all, it was an inspiring, tiring, and overall … experience. I’m amazed at, no matter how many times I do things like this, I always learn something new, or think of something I should have done differently or better.

A++ would do again.

Note: This is part 1 in a series. Currently, I think there are likely to be three parts (configuration explained, performance tweaks, and troubleshooting), but we will have to see as I write them. I really expected this to only be a single entry.

First and foremost, I have to give credit to Matt Aurand. He took the time to read the Apache docs, and distill them into something I actually read, comprehended, and can now spit back out to you. There are some large portions of this post copied directly from what he wrote, because I could not come up with a better way to say it.  The other huge resource is the Apache Docs (1). Huge, and dry. I tried to make it less dry, and more condensed.

---

My background with Apache is almost entirely in LAMP servers that have cPanel/WHM installed. While I have attempted to remove any and all assumptions based on that configuration, I’m sure I’ve missed some. Feel free to point them out, if you find them.

---

Apache is the most common and most popular web server out there(2). It’s free, stable, and easy to support. While it does work in a sort of out-of-the-box way, there are lots of things you can do to make your server (and the apache service) more stable.

What we’ll cover in this post is all contained here. This is a copy/paste of my own VPS’s apache directives.

KeepAlive On
MaxKeepAliveRequests 150
KeepAliveTimeout 5
<IfModule prefork.c>
StartServers 10
MinSpareServers 10
MaxSpareServers 20
MaxClients 150
MaxRequestsPerChild 500
</IfModule>
<IfModule worker.c>
ServerLimit 5
StartServers 2
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxClients 150
MaxRequestsPerChild 500
</IfModule>

Timeout 300

MPMs:

The two most common MPMs (Multi-process modules) used with apache are mpm-prefork and mpm-worker. They’re both stable, and easy to use, but are good for very different things. Briefly: Prefork is non-threaded, and runs a lot like apache 1.3. Worker combines threading and non-threading and is extremely powerful.

According to the Apache docs, Prefork is:

This Multi-Processing Module (MPM) implements a non-threaded, pre-forking web server that handles requests in a manner similar to Apache 1.3. It is appropriate for sites that need to avoid threading for compatibility with non-thread-safe libraries. It is also the best MPM for isolating each request, so that a problem with a single request will not affect any other.

Worker‘s long description:

This Multi-Processing Module (MPM) implements a hybrid multi-process multi-threaded server. By using threads to serve requests, it is able to serve a large number of requests with fewer system resources than a process-based server. However, it retains much of the stability of a process-based server by keeping multiple processes available, each with many threads.

Follow the links for a complete description. For the remainder of this post, I will assume that you have a base understanding of Apache 2.0 and the two above MPM’s (hereafter called Prefork and Worker).

Which MPM should you use?

Apache comes by default with Prefork. If you’re using any modules that do not work correctly threaded, you should stick with prefork. Prefork is more stable, and built directly on known technology (Apache 1.3).

Prefork, however, also uses more resources on a normal server, and therefore does not always gracefully handle spikes or growth. Servers seeing bottlenecks at CPU or Memory should consider switching to Worker.

Worker shouldn’t be installed, though, if you have a compelling reason for each Apache request to be isolated. Since Worker is threaded, each child process handles multiple requests and may not be appropriate for what you need. It should be noted; a huge drawback to Worker is that if PHP has been compiled and installed to be used by Apache as a DSO, Worker will not work.

Directives

Each Module has 20+ directives that can be defined to alter the performance of apache. For the sake of brevity, I will give a brief description of each as we arrive at it, and link to where you can read more about it.

Common to both MPMs

For the full list, you can head over to the apache docs. I’ll just cover some of the most common.

Timeout $integer (Default 300) Apache#Timeout

Timeout can be set in the Server Config, or in individual vhosts.

Timeout, in seconds, controls how much time Apache will wait for the follow three conditions:
The total ammount of time it takes to recieve a GET request
The amount of time between receipt of TCP packets on a POST or PUT request
The amount of time between ACKs on transmissions of TCP packets in responses.

KeepAlive $(on|off) (Default on) Apache#KeepAlive

KeepAlive can be set in the Server Config and in individual vhosts.

The KeepAlive directive decides whether or not the connection is kept open to that client for further requests, after it serves its request. On mean yes, Off means no. “In some cases this has been shown to result in an almost 50% speedup in latency times for HTML documents with many images. To enable Keep-Alive connections, set KeepAlive On.”

MaxKeepAliveRequests $integer (Default 100) Apache#MaxKeepAliveRequests

MaxKeepAliveRequests can be set in the Server Config or in individual vhosts.

MaxKeepAliveRequests limits the number of requests that are allowed per connection when KeepAlive is on. 0 means unlimited. A good rule of thumb in a shared environment is 150% of your MaxClients. If your Site is static, stable, and on its own servers, this number can be safely increased to 1000 or more.

KeepAliveTimeout $integer (Apache Default 5, Cpanel Default 15) Apache#KeepAliveTimeout

This value can be set in the server config and in individual vhosts.

KeepAliveTimeout, in seconds, controls how long Apache will wait for a subsequent request from a specific client before closing the connection to that client. Having this value too high will cause performance problems on high traffic servers, because server processes will be occupied waiting on connections with idle clients. If KeepAlive is off, this value will not be read.

MaxClients $integer (Apache Prefork Default 256, Apache Worker Default ServerLimit X ThreadsPerChild, Cpanel Default 150) Apache#MaxClients

MaxClients can only be set in the server config.

MaxClients sets the limit on the simultaneous requests that will be served. Connections over this value will be queued, which is controlled by the ListenBacklog variable. Worker’s MaxClients must be equal to no more than ServerLimit X ThreadsPerChild.

MaxClients can be increased if a server is having trouble keeping up with legitimate traffic, and can vary greatly, depending on the type of content being served, the kind of server it is, and the amount of memory the server has. Increasing this value should be done cautiously, because if it is set too high you will see an increase in server instability, as each additional MaxClient causes apache to use more resources.

If you set prefork’s MaxClient above 256, the ServerLimit may also be also need to be set.

ServerLimit (Apache Prefork Default 256, Apache Worker Default 16) Apache#ServerLimit

This value is set in the server config.

With PreFork, ServerLimit sets the MaxClients for the lifetime of the Apache process. With worker, ServerLimit combined with ThreadLimit determines the server’s MaxClients. Please note that the ServerLimit cannot be changed via a restart. It requires that you stop and start apache. This Value does need only be set if you need to increase MaxClients above the server’s hard limit and has a hard-coded limit of 20,000 for worker and 200,000 for prefork. This is intended to avoid nastyness caused by typos.

MaxRequestsPerChild $interger (Apache Default 10000, Cpanel Default 0) Apache#MaxRequestsPerChild

This value is set in the server config.

MaxRequestsPerChild sets a limit on the number of requests each individual Apache process will handle before the child process is stopped. A value of 0 (infinity) will prevent any processes from expiring. If the content being served by Apache is stable (even if not static) this directive can be set quite high, to allow processes to serve more content.

StartServers $interger (prefork default 5, worker default 3) Apache#StartServers

StartServers is set in the server config.

StartServers controls how many child processes are spawned at startup. Please note that after start, this directive isn’t used. What value this should be set to is dependent on the type and size of CPU, as well as the amount of memory the server has. Due to the way apache works, setting this value lower than MinSpareServers or more than MaxSpareServers is counterproductive as it generally only puts extra load on the server itself.

RLimitCPU $interger|$interger (set in seconds) (Not set by default) Apache#RLimitCPU

RLimitCPU can be set in the server config, an individual domain’s vhost, a directory block or a .htaccess file in the domain’s docroot.

RLimitCPU determines the CPU limit for all *child* processes such as PHP or CGI processes, not the Apache children themselves. The first value is a soft limit and the second value is a hard limit. Either value can be set to a number or the word “max”. If you are on a VPS, it is strongly advised that you do not set this value, as it could cause significant problems for your host.

RLimitMEM $interger|$interger (set in seconds) (Not set by default) Apache#RLimitMEM

RLimitMEM can be set in the server config, an individual domain’s vhost, a directory block or a .htaccess file in the domain’s docroot.

RLimitMEM determines the MEM limit for all *child* processes such as PHP or CGI processes, not the Apache children themselves. The first value is a soft limit and the second value is a hard limit. Either value can be set to a number or the word “max”. If you are on a VPS, it is strongly advised that you do not set this value, as it could cause significant problems for your host.

---

If you’ve made it this far, congratulations. You now have a much more firm understanding of what Apache directive does, and how you can leverage it for your own server-worldly good.

Next week I’ll talk about the tricks I’ve picked up as a SysAdmin at a Web Host, and how they can help you.

__________________________________________________________________

References:

http://httpd.apache.org/docs/2.0/

2. http://news.netcraft.com/archives/2010/02/22/february_2010_web_server_survey.html

These are the guys (and gals) that I spend my time humbling, supporting, loving, and keeping in line.

I don’t talk much about the job on here, but I really do work with some of the most amazing people on the planet. They’re smart, motivated, and very very good at what they do. I am challenged daily, and almost never feel like I’m the smartest person in the room. I am humbled, and, supported, and loved, and kept in line by these guys. I can’t imagine anyplace else I’d rather be.

I keep being asked what I’ve been up to, and where I’ve been hiding.  Truth is, I haven’t really been hiding, just … being picky about where I spend my time.  I’ve been doing anything that makes me happy, and avoiding anything that doesn’t.

I’ve been seeing live music a lot (Gogol Bordello, Porcupine Tree, Ultraviolet Hippopotamus), and spending time contemplating this:

I’ve been spending a lot of time with these guys:

And some time with [The Kings] because I cannot resist them, for they are adorable and squishy and mostly smell pretty good.

Jane has become the devil, and Marley plays with ghosts.

Too, these have been in my dining room for two weeks, but will be returned to my mom tomorrow when she gets home from Amsterdam.

In short: I’ve been here, and having fun.  I really, truly, and sincerely hope you are doing the same.

Friday night was short, but completely worth the lack of sleep.

I’d emailed the GLLUG earlier this week to see if they were looking for a food sponsor for either of the nights of PenguiCon (I like to give away food), and they (not surprisingly) obliged. I agreed to buy some pizzas for the Friday night room party, and they agreed to pay me in beer.

I left lansing around 8:15pm (a full hour after I’d hoped to).  As I passed Okemos, I sent am email to the GLLUG group to ask simply: “Who’s got an opinion about where to get the best pizza in the Troy area?”  By 8:30, I had the name, address, and phone number of Alibi. By 8:45 I’d called, had 9 pizzas ordered, and was following the directions that my phone was using to get me there.

(have I said how I love living in the future lately?)

I emailed the group to let them know that I was on my way with the pizza, and they talked someone into meeting me at the door to the hotel. The man who helped me carry the pizza in was awesome, and the party went off with a bang. The beer and the company were amazing, and it was a blast.

Saturday was a whirl of panels, meeting and talking to new people, and seeing old friends. Too much fun to name it all.

__

One of the things that stuck me, at this conference in specific, was a statement made by one of the attendees.

Nerds in Media:

“We don’t need validation, here, for being Nerds. Do we care what main stream media thinks?” — Jeff DeLazio

That, ladies and gentlemen, is why I love PenguiCon.