View Full Version : Apache - Segmentation fault
Hello,
Last week I brought up a new Hsphere cluster consisting of 10 servers ... CP, NS1, NS2, Mail, MySQL, Urchin and 4 Web servers at this time. All servers are running CentOS 4.x Final with Hsphere 2.5 Patch 6.
On all servers that support web services (the Web, Mail and MySQL servers), the Apache log shows:
[Sat Oct 14 09:44:53 2006] [notice] Accept mutex: sysvsem (Default: sysvsem)
[Sat Oct 14 09:45:20 2006] [notice] child pid 13704 exit signal Segmentation fault (11)
[Sat Oct 14 09:45:42 2006] [notice] child pid 13714 exit signal Segmentation fault (11)
with the bulk of these messages showing up on the Web servers. Anybody having the same experience?
I understand the bulk of the Apache modules are now DSO compiled ... I would like to recompile Apache with the bear minimium of modules needed to support Hsphere ... in other words, "hard compiling" just the modules need for Hsphere into Apache and getting rid of DSO (personally, I never did like the idea).
The instructions on
http://www.psoft.net/HSdocumentation/sysadmin/apache_compilation.html
are pretty much useless to me at this time. The lastest bundle of source in http://www.psoft.net/shiv/HS/bundles/ is Apache 1.3.36 ... am I missing something on this apache_compilation page?
I would appreciate a list of just the required modules for Hsphere 2.5.x and a common repository where the current modules can be found. And an easy way to distribute "my Apache" across 100s of servers would be nice too.
Thank you for you time.
Craig
dynamicnet
10-14-06, 12:48
Greetings Craig:
In your hosting plans, turn off mnogosearch and mod_throttle. Also make sure they are off in the end user plans.
Thank you.
Thank you Peter however that ain't the problem ... these have never been turned on within the hosting plans under 2.5:
Throttle Resource
mnogoSearch
RLimitCPU
RLimitMEM
RLimitNPROC
Craig
dynamicnet
10-14-06, 16:41
Greetings Craig:
For the said error you posted, in the past https://support.comodo.com/?group=psoft provided the solution I posted.
If that is not working, then please put in a ticket to https://support.comodo.com/?group=psoft
Thank you.
Craig,
I had a ticket in on this .. It appears that Apache is compiled with a non compatible version of GCC in some cases. Peters suggestion to put in a ticket is a good option since that Apache compilation page is so annoyingly out of date.
Francesca
So what was the outcome of your Support Ticket? Seems like it is still broken.
I need to re-compile Apache anyway to get a custom version of mod_auth installed. I personally don't like running Apache with DSO modules ... years ago, I read something about DSO effecting performance so I have always just compiled Apache with all the need modules.
dynamicnet
10-15-06, 07:26
Greetings Craig:
Your custom compilation may be the root of your other problems you posted.
i.e. http://forum.psoft.net/showthread.php?t=17150
While customizations can be done to httpd.conf, custom compiling Apache does open up doors to potential problems during upgrades.
Thank you.
Hello,
[Sat Oct 14 09:44:53 2006] [notice] Accept mutex: sysvsem (Default: sysvsem)
[Sat Oct 14 09:45:20 2006] [notice] child pid 13704 exit signal Segmentation fault (11)
[Sat Oct 14 09:45:42 2006] [notice] child pid 13714 exit signal Segmentation fault (11)
Craig
Hi,
I have the same problem since P5.
I did not find a solution yet and Psoft said it was my mod_evasie but with or without it I get the problem.
I had to turn off core generation to prevent disk filling but I had hopes that the P6 version would solve it.
I need to spend time on this to troubleshoot the faulty module if any.
I am on CentOS 4-4 in a Xen environment and I thought it might have been the problem because of the new version of Glibc but I did not updated yet my Hypervisor for it, hopefully this week I'll have more info.
++
Hello,
I have put in another reminder that this needs to be fixed with a patch two days ago .. I have not heard anything back ..
Full thread of my conversation with PSOFT Developer .. Newest is at top ..
"Recent Message Regarding the 20 or so Centos 4.X servers I am seeing this on regular now"
>Any plans on fixing this with a patch perhaps ??..
>It appears on most all Centos 4.4 boxes I maintain.
"His Answer I got about three weeks ago ... Server is running fine since this custom compile"
>
> I believe I was wrong.
>
> The same error appeared during compilation of php 4.4.0: during apache
> restart we received Segmentation fault message.
>
> When reviewing core dumps we received the same output:
> > #0 0x007ce4a6 in do_lookup_x () from /lib/ld-linux.so.2
> > #1 0x007ce8b2 in _dl_lookup_symbol_x () from /lib/ld-linux.so.2
> > #2 0x007d1c66 in fixup () from /lib/ld-linux.so.2
> > #3 0x007d1ba0 in _dl_runtime_resolve () from /lib/ld-linux.so.2
> > #4 0x009a9126 in __do_global_dtors_aux () from /lib/tls/librt.so.1
> > #5 0x009ae706 in _fini () from /lib/tls/librt.so.1
> > #6 0x007d2907 in _dl_fini () from /lib/ld-linux.so.2
>
> It was peculiar only to the systems with gcc 3.4. After php
> recompilation on gcc 3.2.3 the problem disappeared.
>
> Speaking of your problem with apache:
> Apache worked fine until October 7, but after 8 p.m. the Segmentation
> fault message appeared in error_log.
> According to /var/log/yum.log, the system upgrade took place:
> Oct 07 11:14:36 Updated: glibc-common.i386 2.3.4-2.25
> Oct 07 11:14:41 Updated: glibc.i686 2.3.4-2.25
>
> Then, upon running /hsphere/shared/apache/bin/apachectl graceful,
> a number of Segmentation fault messages always appeared in error_log,
> which corresponded the number of processes.
> Therefore, I hurried to make a conclusion that the problem was in new
> glibc, but...
>
> We tried to check if previous versions of apache+php would work.
> But after we installed the last version back again - apache started
> working faultlessly.
>
> Unfortunately, I don't really know how to re-produce the error to
> determine the prime cause of the problem.
>
> gdb was un-installed as unnecessary.
>
> best regards,
> Andrew
My Answer to downgrading GLIBC ..
>
> > I appreciate your logic here. But this box was updated as part of
> > Centos/RHES update # 4. That came out over two months ago and had a Bunch
> > of security updates. (About 170 or so packages on a minimal install
> > system)
> >
> > I don't think that downgrading GLIBC would be good for the rest of the
> > systems stability (If its even Possible given dependencies).
> >
> > Furthermore this update has and is being applied to quite a few Servers
> > running Hsphere. So the best choice here IMHO is to fix this Apache/PHP
> > issue now with this version of glibc.
"First Stab at it .. telling me to downgrade glibc"
>
> > > > On Tuesday 10 October 2006 06:06, afterstep wrote:
> > > > Hello,
> > >
> > > You have updated your box on October 7. Can you downgrade glibc to
> > > previous version?
> > >
> > > From yum logs - currently installed:
> > > Oct 07 11:14:36 Updated: glibc-common.i386 2.3.4-2.25
> > > Oct 07 11:14:41 Updated: glibc.i686 2.3.4-2.25
> > >
> > > but before Oct 7 was
> > > Apr 08 05:44:18 Updated: glibc-common.i386 2.3.4-2.19
> > > Apr 08 05:44:23 Updated: glibc.i686 2.3.4-2.19
> > >
> > > Example of core dump backtracing
> > > (gdb) bt
> > > #0 0x007ce4a6 in do_lookup_x () from /lib/ld-linux.so.2
> > > #1 0x007ce8b2 in _dl_lookup_symbol_x () from /lib/ld-linux.so.2
> > > #2 0x007d1c66 in fixup () from /lib/ld-linux.so.2
> > > #3 0x007d1ba0 in _dl_runtime_resolve () from /lib/ld-linux.so.2
> > > #4 0x009a9126 in __do_global_dtors_aux () from /lib/tls/librt.so.1
> > > #5 0x009ae706 in _fini () from /lib/tls/librt.so.1
> > > #6 0x007d2907 in _dl_fini () from /lib/ld-linux.so.2
> > > #7 0x00809527 in exit () from /lib/tls/libc.so.6
> > > #8 0x0805e464 in ap_add_config_define ()
> > > #9 0x08060143 in ap_add_fatal_exception_hook ()
> > > #10 <signal handler called>
> > > #11 0x007c67a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
> > > #12 0x0089fa0d in ___newselect_nocancel () from /lib/tls/libc.so.6
> > > #13 0x0805f217 in ap_kill_timeout ()
> > > #14 0x08061a5b in ap_child_terminate ()
> > > #15 0x08061bcd in ap_child_terminate ()
> > > #16 0x08061c73 in ap_child_terminate ()
> > > #17 0x0806230a in ap_child_terminate ()
> > > #18 0x08062b42 in main ()
> > >
> > > We was review many core's and all is same.
> > > Possible problem related with
> > > [root@lnxweb1-002 logs]# rpm -qf /lib/ld-linux.so.2
> > > glibc-2.3.4-2.25
> > >
> > > Sorry for inconvenience
> > >
> > > --
> > > Andrew
Francesca
Hello All,
This is rather frustrating ... I now have 30+ servers managed by Hsphere 2.5 Patch 6. All servers hardware is basically configured the same way with the same build of CentOS release 4.4 (Final) (Linux version 2.6.9-42.0.3.EL).
I would say half of them are getting the Seg Fault and the other half is good. I have taken the Apache/PHP files from a "good server" and placed them on a "bad server" ... to no avail ... still getting the Segmentation Fault.
Sure wish someone would take the time to fix this one and for all or provide COMPLETE instructions on how to compile Apache / PHP under Hsphere.
Craig
Hello,
I am glad to say this idiotic Bug still exists with latest Apache Update and 2.5 patch 7.
Lets see .. About 10K seg faults each 24 hrs ..
I give up .. this developer has no interest in fixing this at all.
Francesca
Hello,
try to disable php mnogosearch extension
(remove file ~httpd/conf/php4/php.d/mnogosearch.ini or comment line extension=mnogosearch.so in this file).
This is possible cause of segfaults, we will be fix it in next php4 distribution.
Hello,
try to disable php mnogosearch extension
(remove file ~httpd/conf/php4/php.d/mnogosearch.ini or comment line extension=mnogosearch.so in this file).
This is possible cause of segfaults, we will be fix it in next php4 distribution.
Yes. This work for me. Have the same issue with tons of segmentation fault in one of our web server. Have not have them after disabling mnogosearch...
Hello,
Hello,
try to disable php mnogosearch extension
(remove file ~httpd/conf/php4/php.d/mnogosearch.ini or comment line extension=mnogosearch.so in this file).
This is possible cause of segfaults, we will be fix it in next php4 distribution.
Hello .. I appreciate this .. But that is NOT how you fixed my issue before. Now additionally I sent in over a MONTH ago a recommendation to update mnogosearch. And you guys sat on that. Now its way past due that we get a fix for this annoying "Bug".
Francesca
stevewest15
11-29-06, 19:12
If I may ask...how many customers really use mnogosearch? In our case at least, it seems like another none needed component that had it's time and now just causes a ton of headaches...
Just my opinion...would love to hear from others if they really use it rather than another more stable tool.
SW
Steve,
I have quite a few customers that use it. Its well maintained also.
Also I am throwing rocks at this mnogosearch being the issue. Considering the version hsphere is using is so far out of date.
Francesca
dynamicnet
12-28-06, 09:11
Greetings:
Is this still an issue for H-Sphere 2.5.0 P8?
Thank you.
Hello,
I do not know for I am running 2.5 P6 ... I think. My CP is P6 however I have added a bunch of new servers with the recent one just the other day. Therefore, I have no idea which "P" version of Apache I am running.
I do know by "disabling" mnogosearch by rename the .so and .ini files, all Segmentation faults have disappearred.
On another note, last night I ran into a problem with /hsphere/shared/apache/bin/htpasswd where on an "add" it "0-ed" out an existing .htpasswd. I could not get it to add a new username:password, it kept creating a zero length (empty) .htpasswd file. I solved the problem by coping over a "good" htpasswd from another server.
This brings up the question is ANYBODY at Psoft doing QA on new releases/patches or are we, the customer base, the QA team?
Craig
dynamicnet
12-28-06, 10:02
Greetings Craig:
Almost all H-Sphere updates are through the control panel physical server.
If you did not update to http://www.psoft.net/HSdocumentation/versions/index.php?v=250p8&p=r ,then all new servers you add still running 2.5.0 P6.
Thank you for the note on mnogosearch; that helps.
Hello,
I am not going to argue with you about what version of what I am running ... I don't have the time or the desire to.
However, I will tell you that I add all new servers the same way ... according to the documentation. I first add the P-server and then the L-server via the CP interface. After that, I log into the CP server and run sh U25.0P6 entering
hspackages ips=x.x.x.x
at the prompt. The same applies to all Hsphere patch updates.
By looking at the directories and files, I can tell that each server is somewhat different ... examples:
1) the file length of /hsphere/shared/apache/bin/htpasswd ... one at least one server, the htpasswd command does not work, see previous post.
2) do a search on mnogosearch* ... on some servers, a .ini file is found and on others, it's not.
It would seem to me, if I follow all the documented procedures to add servers and to update the patch releases, all servers would be the SAME or am I missing something here?
Craig
dynamicnet
12-28-06, 11:50
Greetings Craig:
Whenever you run "hspackages," the latest stable packages will be applied; this does not change your H-Sphere version, just the package version.
Given http://www.psoft.net/misc/hs_pkg_update_14.html was released as stable (also see http://forum.psoft.net/showthread.php?t=18051 ), you may want to run hspackages for all of your IP's. See http://www.psoft.net/misc/hs_pkg_update_14.html for more information.
You may also want to consider upgrading to H-Sphere 2.5.0 P8 as it has bug fixes that exist in P6, P7 and prior versions.
Thank you.
Peter,
I didn't mean to rant ... it just gets a little frustrating. I have been at this a long long time. Back in "my mainframe days", we were always 6 mouths to 1 year behind on our upgrades.
It might be true that P8 fixes the bugs that were found in P6 and P7, but, can you honestly tell me that P8 will not introduce new bugs?
There is a thread in the Bug section about errors in bandwidth calculations and it was addressed with hsphere-apache-6 ... seems 4 and 5 were in error ... great ... now that that has been fixed, who is going to fix the credibility of future invoices to my client base? Once in error, always in error?
But, I will consider taking your advice and upgrading to P8.
Again, I appologize for ranting.
Craig
Hello,
Still a problem in 2.5 Patch 8.
EDIT:: This is only a issue apparently with Linux. This does not occur anywhere as I can see with Freebsd.
Francesca
dynamicnet
12-28-06, 12:26
Greetings Craig:
I guess for us, it is a journey towards less bugs even though the journey may have new bugs. That is, if the end result is less bugs, and the new bugs don't overshadow the less bugs benefit, then it is ok.
Thus stated, the approach psoft took for hspackages is that anytime you run it, you get the latest released even if you wanted to stay with the same version from x months ago.
Thank you.
Hello,
Just to update this thread .. Afterstep the developer comes in here and says disable mnogosearch. Thats cool since it can't work with Centos 4.4 and they tried everything at PSOFT to fix this. Right ??
Well we have a different story if one looks a bit deeper at this issue.
Here is what I know about this issue as of tonight ..
I griped that they should update mnogosearch to latest and they did a package
update or so ago.
But they only updated the system library's and binaries ..
They DID NOT recompile php with the new mnogosearch to create a new compatible php module.
Take a look at a fully patched box php environment ..
You will see
mnogosearch library version 30235
The above equals 3.2.35 (23 November 2005)
rpm -qa hsphere-mnogosearch
hsphere-mnogosearch-3.2.40-0
Comment out entry in
/hsphere/shared/apache/conf/php4/php.d/mnogosearch.ini
; Enable mnogosearch extension module
;extension=mnogosearch.so
Restart apache and whola .. no more segfaults ..
Of course mnogosearch does not work ..
But these segfaults happen whether or not that one actually uses
mnogosearch ..
Whats the smoking gun here ??
I told afterstep at PSOFT that he needed to recompile apache/php for Newer Centos 4.X and RHEL 4.X .. something to do with new glibc/gcc
He did not and still has not ..
I give .. As long as we have this type of sloppiness hanging around we will NEVER have stable apache with Hsphere.
Francesca
stevewest15
02-17-07, 13:13
Hi Francesca,
Thanks for the info and all of your hard work trying to get Psoft to listen!
SW
We should all be grateful to Francesca for taking the time to keep on these items.
We shouldn't be seeing segmentation faults from incomplete builds ... h-sphere is a web hosting platform after all ...
vBulletin® v3.7.2, Copyright ©2000-2009, Jelsoft Enterprises Ltd.