For the second year in a row Liquid Web submitted two chilis for the Lansing Board of Water and Light sponsored Chili Cookoff. This year, placed in b oth categories that we entered.
Chris and Jerry, with their awards.
That’s bloody RIGHT. We do hosting AND chili. You can see the rest of the set at flickr.
:D
Note: This is part 2 in a series on Apache in a LAMP environment. You can read part 1 here.
Now that we’ve covered the common directives, and how they relate to each other, let’s talk about some tweaks to make Apache hum in your environment. We will assume that you know where to find the referenced files.
What we’ll cover in this post is all contained here (a copy of my own VPS’s Apache directives).
KeepAlive On
MaxKeepAliveRequests 150
KeepAliveTimeout 5
<IfModule prefork.c>
StartServers 10
MinSpareServers 10
MaxSpareServers 20
MaxClients 150
MaxRequestsPerChild 500
</IfModule>
<IfModule worker.c>
ServerLimit 5
StartServers 2
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxClients 150
MaxRequestsPerChild 500
Timeout 300
Understanding the implications of any changes you make to your configuration is essential, but a lot of that knowledge will come with experience. Here is my take on three of the directives, and some of the things you can do with them.
1: MaxClients.
This directive defines how many simultaneous and independent requests will be served by Apache. There are two things to consider before you change this setting.
By default Apache uses a setting of 150. This is a good starting point but it’s probably low for dedicated servers. Slowly ncrease it in increments of 25-50. If it’s already at 500 or 1000 then increasing it more may not be the right answer.
You want to allow pages to be served. You don’t want to crash the server. If your site is normally memory or CPU intensive and you don’t have much to spare you will want to think carefully before increasing it.
If MaxClients is reached new requests are queued rather than being served immediately. This will appear to clients like a really long load time. Check your error log for this error:
[Sun Jun 06 18:00:15 2010] [error] server reached MaxClients setting, consider raising the MaxClients setting
There are a couple ways to handle this. The most obvious is increasing that number. You can also use a command like netstat to determine if there is a single IP address that has a lot of connections to port 80 on your server. My favorite piece of bash-fu:
netstat -tn 2>/dev/null | grep :80 | awk '{print $5}' | cut -f1 -d: | sort | uniq -c | sort -rn | head
That will collect a list of all of the open connections to port 80 and sorts the 5th field in that string. Depending on your system you might need to tweak the line to sort by the correct field. The ‘Foreign Address’ field is the one you want.
nestat -lnp | head
2. KeepAlive
KeepAlive is an on/off setting. It determines if a connection to the client will remain open after Apache serves its request. If your server is high traffic or is serving a lot of small and/or flat files then turning KeepAlive off can significantly reduce resource use by Apache.
Beware, though, that changing this directive is not as predictable as with the others. If you decide to change it on an a server that is experiencing problems at the time you should keep a close eye on your server’s load and your finger on the kill -9 trigger.
3: MaxRequestsPerChild
MaxRequestsPerChild defines how many requests a single Apache child would serve before it was killed and a new child spawned. In a shared environment this can be very helpful to negate the effects of memory leaks in the web server. By default this setting is 1000. A good value for this setting in a shared environment is 150% of your MaxClients directive.
If you are serving the same static, predictable content you can safely pump this directive up pretty high. This will decrease the amount of times that Apache has to kill and spawn new children. 10,000 is a good setting for higher traffic servers with stable/static content. If you like to live dangerously you can set it to ’0′ (unlimited). If you increase it beyond 150% you might consider writing a script that will monitor your server’s memory use and alert you if it gets too high.
The Ann Arbor Perl Mongers are a great, and still building steam, group of guys (no girls yet) that love perl. Dan and I went down to Ann Arbor for April’s meeting, and had a great time.
In April:
* Tommy Staton gave a presentation on ‘Syncing your ~ with Git’. It was well done, and somewhat controversial, and I learned a TON about Git, which is kind of awesome. Too, he used Voom (which allows you to write and show presentations in Vim) to write his presentation, and with my love of Vim, it was easy to like him.
* Jamie Pitts – gave a presentation about “Boilerplate Web Apps” (downloads his presentation) about Catalyst::Helper, which was incredibly detailed and quite revolutionary for me but… went immediately over my head. As a side note, I really REALLY need to learn Perl (which, incidentally, apparently is three classes deep in the Computer Sciences department at LCC… JUST GIVE ME A DAMN CLASS ON PERL)
This month Liquid Web (by which I mean me) offered to host the meeting. I asked our devteam if they wanted to present anything, and, to put it simply, they are far too busy with things I want them to be doing for LW. It was coming down to it, and it was becoming bloody apparent: I was going to have to present something. To the Perl Mongers.
Me. Who doesn’t know perl. ….shiiiiiiii.
But then my (quickly becoming good) buddy Mark suggested I talk about what I do know: why Perl Enthusiasts need to be their own Evangelists. I was 3/4 of the way through building that when Jamie (from the last meeting, remember?) and Mike Mol both offered to present, and my lead dev tells me that this topic was covered three months ago, at the same group.
SAVED!
Organize the event?! I can do that in my sleep.
Pizza, soda, technology, and places to sit. That’s all I had to come up with.
This Month:
* Jamie presented on the Architecture of a well written, multi-layer (and sometimes multi-application) perl application. It definitely engaged the LW devs and Engineering folk that made it out. This time I actually followed along with the concepts, but as soon as he started giving examples he lost me (have I mentioned how it would be a good idea to learn perl?). It was a good presentation, though.
* Mike Mol talked about his extremely cool project: Rosetta Code, which is exactly what you’d think it was. The base idea: write the same function/program/concept in as many different languages as you can. While relatively simple in concept, it is a huge undertaking. He’s grown his own community, and faced his challenges, and it was a joy watching him interact with my beloved engineering geeks.
After we broke (best two hours ever), I took the four out-of-towners on a tour of LW, showing them the one section of servers, and the huge generators, and the Awe-Inspiring that is the Datacenter I typically take for granted. They had questions about all sorts of statistics I haven’t committed to memory yet (Mental note: learn about the electrical infrastructure), but it was a spectacular end to the night.
All in all, it was an inspiring, tiring, and overall … experience. I’m amazed at, no matter how many times I do things like this, I always learn something new, or think of something I should have done differently or better.
A++ would do again.
Note: This is part 1 in a series. Currently, I think there are likely to be three parts (configuration explained, performance tweaks, and troubleshooting), but we will have to see as I write them. I really expected this to only be a single entry.
First and foremost, I have to give credit to Matt Aurand. He took the time to read the Apache docs, and distill them into something I actually read, comprehended, and can now spit back out to you. There are some large portions of this post copied directly from what he wrote, because I could not come up with a better way to say it. The other huge resource is the Apache Docs (1). Huge, and dry. I tried to make it less dry, and more condensed.
What we’ll cover in this post is all contained here. This is a copy/paste of my own VPS’s apache directives.
KeepAlive On
MaxKeepAliveRequests 150
KeepAliveTimeout 5
<IfModule prefork.c>
StartServers 10
MinSpareServers 10
MaxSpareServers 20
MaxClients 150
MaxRequestsPerChild 500
</IfModule>
<IfModule worker.c>
ServerLimit 5
StartServers 2
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxClients 150
MaxRequestsPerChild 500
</IfModule>
Timeout 300
The two most common MPMs (Multi-process modules) used with apache are mpm-prefork and mpm-worker. They’re both stable, and easy to use, but are good for very different things. Briefly: Prefork is non-threaded, and runs a lot like apache 1.3. Worker combines threading and non-threading and is extremely powerful.
According to the Apache docs, Prefork is:
This Multi-Processing Module (MPM) implements a non-threaded, pre-forking web server that handles requests in a manner similar to Apache 1.3. It is appropriate for sites that need to avoid threading for compatibility with non-thread-safe libraries. It is also the best MPM for isolating each request, so that a problem with a single request will not affect any other.
Worker‘s long description:
This Multi-Processing Module (MPM) implements a hybrid multi-process multi-threaded server. By using threads to serve requests, it is able to serve a large number of requests with fewer system resources than a process-based server. However, it retains much of the stability of a process-based server by keeping multiple processes available, each with many threads.
Follow the links for a complete description. For the remainder of this post, I will assume that you have a base understanding of Apache 2.0 and the two above MPM’s (hereafter called Prefork and Worker).
Apache comes by default with Prefork. If you’re using any modules that do not work correctly threaded, you should stick with prefork. Prefork is more stable, and built directly on known technology (Apache 1.3).
Prefork, however, also uses more resources on a normal server, and therefore does not always gracefully handle spikes or growth. Servers seeing bottlenecks at CPU or Memory should consider switching to Worker.
Worker shouldn’t be installed, though, if you have a compelling reason for each Apache request to be isolated. Since Worker is threaded, each child process handles multiple requests and may not be appropriate for what you need. It should be noted; a huge drawback to Worker is that if PHP has been compiled and installed to be used by Apache as a DSO, Worker will not work.
Each Module has 20+ directives that can be defined to alter the performance of apache. For the sake of brevity, I will give a brief description of each as we arrive at it, and link to where you can read more about it.
For the full list, you can head over to the apache docs. I’ll just cover some of the most common.
Timeout $integer (Default 300)
Apache#Timeout
Timeout can be set in the Server Config, or in individual vhosts.
Timeout, in seconds, controls how much time Apache will wait for the follow three conditions:
The total ammount of time it takes to recieve a GET request
The amount of time between receipt of TCP packets on a POST or PUT request
The amount of time between ACKs on transmissions of TCP packets in responses.
KeepAlive $(on|off) (Default on)
Apache#KeepAlive
KeepAlive can be set in the Server Config and in individual vhosts.
The KeepAlive directive decides whether or not the connection is kept open to that client for further requests, after it serves its request. On mean yes, Off means no. “In some cases this has been shown to result in an almost 50% speedup in latency times for HTML documents with many images. To enable Keep-Alive connections, set KeepAlive On.”
MaxKeepAliveRequests $integer (Default 100)
Apache#MaxKeepAliveRequests
MaxKeepAliveRequests can be set in the Server Config or in individual vhosts.
MaxKeepAliveRequests limits the number of requests that are allowed per connection when KeepAlive is on. 0 means unlimited. A good rule of thumb in a shared environment is 150% of your MaxClients. If your Site is static, stable, and on its own servers, this number can be safely increased to 1000 or more.
KeepAliveTimeout $integer (Apache Default 5, Cpanel Default 15)
Apache#KeepAliveTimeout
This value can be set in the server config and in individual vhosts.
KeepAliveTimeout, in seconds, controls how long Apache will wait for a subsequent request from a specific client before closing the connection to that client. Having this value too high will cause performance problems on high traffic servers, because server processes will be occupied waiting on connections with idle clients. If KeepAlive is off, this value will not be read.
MaxClients $integer (Apache Prefork Default 256, Apache Worker Default ServerLimit X ThreadsPerChild, Cpanel Default 150)
Apache#MaxClients
MaxClients can only be set in the server config.
MaxClients sets the limit on the simultaneous requests that will be served. Connections over this value will be queued, which is controlled by the ListenBacklog variable. Worker’s MaxClients must be equal to no more than ServerLimit X ThreadsPerChild.
MaxClients can be increased if a server is having trouble keeping up with legitimate traffic, and can vary greatly, depending on the type of content being served, the kind of server it is, and the amount of memory the server has. Increasing this value should be done cautiously, because if it is set too high you will see an increase in server instability, as each additional MaxClient causes apache to use more resources.
If you set prefork’s MaxClient above 256, the ServerLimit may also be also need to be set.
ServerLimit (Apache Prefork Default 256, Apache Worker Default 16)
Apache#ServerLimit
This value is set in the server config.
With PreFork, ServerLimit sets the MaxClients for the lifetime of the Apache process. With worker, ServerLimit combined with ThreadLimit determines the server’s MaxClients. Please note that the ServerLimit cannot be changed via a restart. It requires that you stop and start apache. This Value does need only be set if you need to increase MaxClients above the server’s hard limit and has a hard-coded limit of 20,000 for worker and 200,000 for prefork. This is intended to avoid nastyness caused by typos.
MaxRequestsPerChild $interger (Apache Default 10000, Cpanel Default 0)
Apache#MaxRequestsPerChild
This value is set in the server config.
MaxRequestsPerChild sets a limit on the number of requests each individual Apache process will handle before the child process is stopped. A value of 0 (infinity) will prevent any processes from expiring. If the content being served by Apache is stable (even if not static) this directive can be set quite high, to allow processes to serve more content.
StartServers $interger (prefork default 5, worker default 3)
Apache#StartServers
StartServers is set in the server config.
StartServers controls how many child processes are spawned at startup. Please note that after start, this directive isn’t used. What value this should be set to is dependent on the type and size of CPU, as well as the amount of memory the server has. Due to the way apache works, setting this value lower than MinSpareServers or more than MaxSpareServers is counterproductive as it generally only puts extra load on the server itself.
RLimitCPU $interger|$interger (set in seconds) (Not set by default)
Apache#RLimitCPU
RLimitCPU can be set in the server config, an individual domain’s vhost, a directory block or a .htaccess file in the domain’s docroot.
RLimitCPU determines the CPU limit for all *child* processes such as PHP or CGI processes, not the Apache children themselves. The first value is a soft limit and the second value is a hard limit. Either value can be set to a number or the word “max”. If you are on a VPS, it is strongly advised that you do not set this value, as it could cause significant problems for your host.
RLimitMEM $interger|$interger (set in seconds) (Not set by default)
Apache#RLimitMEM
RLimitMEM can be set in the server config, an individual domain’s vhost, a directory block or a .htaccess file in the domain’s docroot.
RLimitMEM determines the MEM limit for all *child* processes such as PHP or CGI processes, not the Apache children themselves. The first value is a soft limit and the second value is a hard limit. Either value can be set to a number or the word “max”. If you are on a VPS, it is strongly advised that you do not set this value, as it could cause significant problems for your host.
Next week I’ll talk about the tricks I’ve picked up as a SysAdmin at a Web Host, and how they can help you.
__________________________________________________________________
References:
http://httpd.apache.org/docs/2.0/
2. http://news.netcraft.com/archives/2010/02/22/february_2010_web_server_survey.html
These are the guys (and gals) that I spend my time humbling, supporting, loving, and keeping in line.
I don’t talk much about the job on here, but I really do work with some of the most amazing people on the planet. They’re smart, motivated, and very very good at what they do. I am challenged daily, and almost never feel like I’m the smartest person in the room. I am humbled, and, supported, and loved, and kept in line by these guys. I can’t imagine anyplace else I’d rather be.
This is exactly what I think sets the leaders of Liquid Web apart, I think. We’re full of ‘can’, instead of the ‘can’t’ that he talks about.
“Just about everything worth doing is worth doing because it’s important and because the odds are against you. If they weren’t, then anyone could do it, so don’t bother.”
We, too, get caught up in the details of things and use them to prevent forward motion, instead of motivating us. We have to remind ourselves: everything has details. They are just as important as the big picture. Conquer them, complete them, and move on.
I am occasionally overwhelmed, just like everyone else. I want to go hide in my hole, and not come out. Ignore all of my responsibilities, stop training my guys, stop pushing them to be as good as they are, stop pushing myself and the company, and let the email pile up.
The thing that keeps me going? The 12 guys that are counting on me being their motivation, their light, their direction.
It may be corny, but I know I’m the leader, the pusher, the fire under their asses.
For saving me in my most self destructive times, and for just plain being awesome, I talked the bobs into letting me take the guys out for a meal. Each shift has picked a place (or had one assigned, if they weren’t vocal enough). Each shift will meet a time that works well for the shift, have a couple drinks and some food, paid for by the company.
It’s not a small amount of work to organize, but damn is it a good time, and makes a HUGE difference in the morale, and in the interaction of the team.
This is gonna be awesome. I love it.
rpm is freaking powerful. The two flagsets that I always wish I could remember, but seem to forget:
This one seems to show you what directories are touched by a given package:
rpm -lq $packagename
If you have a file, instead, and want to know what package put it there:
rpm -qf $/path/to/file/filename
So handy.
Noti: you never replied to my email. Where’re we meeting tonight.
me: THATS BECAUSE I DONT KNOW HOW TO USE A COMPUTER LETS MEET AT UR MOMS HOUSE HAHA I MEAN LETS MEET AT THE BIGGBY ON WAVERLY AT LANSING RD SEE YOU AT SIX
Noti: lol wow ur crazy, but I’ll be there.
me: I really do crack my shit up. and k.
Once or twice every year, a phenomenon that we (not so) affectionately call the plague sweeps through LiquidWeb. I’m sure it happens other places too, but it’s kind of insane to watch a single person walk in the door sick on a monday, and watch a third of the company drop by friday.
The problem? Dedication. Either to the company, or to the paycheck. Folks come in sick, and take other people down with them. I’ve been guilty of it before, and now I just try to avoid getting sick. I was only sick once last winter, not at all during the summer, and have avoided two rounds of the plague already this fall.
The secrets!
Flickr
Twitter
Last.fm
