Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Apache graceful causes zombie processes when vips module loaded #26

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jesseforrest opened this issue Dec 20, 2016 · 11 comments
Closed
Labels

Comments

@jesseforrest
Copy link

jesseforrest commented Dec 20, 2016

I'm not sure if this php-vips project is the most appropriate forum to writeup this bug report, but I'm not sure where else would be appropriate.

We're trying to get Vips to work with:

  • Ubuntu 14.04.1 LTS
  • Apache 2.4
  • PHP 7.0

We tried to follow the guidelines as instructed in the main readme here:
https://github.com/jcupitt/php-vips

In doing so, we were able to get Apache to run appropriately with the Vips C executable, the PECL module, and your php-vips composer package:

- name: Stop Apache
  shell: service apache2 stop

- name: VIPS Requirements
  apt: name="{{ item }}" state=latest
  with_items:
  - build-essential
  - pkg-config
  - libglib2.0-dev
  - libxml2-dev
  - libtiff5-dev
  - libjpeg-turbo8-dev
  - libgsf-1-dev
  - libvips-dev

- name: Download libvips package
  command: aws s3 cp s3://our-3rd-party-software-repo/vips/vips-8.3.3.orig.tar.gz /tmp/vips-8.3.3.tar.gz
  sudo: yes
  sudo_user: "{{ s3_user }}"

- name: Untar libvips package
  unarchive: src=/tmp/vips-8.3.3.tar.gz dest=/tmp/ copy=no
  sudo: no

- name: Make libvips
  shell: cd /tmp/vips-8.3.3 && ./configure && make

- name: Make install libvips
  shell: cd /tmp/vips-8.3.3 && make install
  sudo: yes

- file: path=/tmp/vips-8.3.3 state=absent

- file: path=/tmp/vips-8.3.3.tar.gz state=absent

- name: Install the binary PHP extension
  shell: /usr/bin/yes '' | pecl install vips
  sudo: yes

- name: Include extension in php.ini
  lineinfile: dest={{ php_ini_apache_path }} regexp='^\[vips\]' line='[vips]'
- lineinfile: dest={{ php_ini_apache_path }} insertafter='^\[vips\]' regexp='^extension=vips.so' line='extension=vips.so'
- lineinfile: dest={{ php_ini_cli_path }} regexp='^\[vips\]' line='[vips]'
- lineinfile: dest={{ php_ini_cli_path }} insertafter='^\[vips\]' regexp='^extension=vips.so' line='extension=vips.so'

- name: Start Apache
  shell: service apache2 start

Everything works beautifully, except when we run an apache2ctl graceful. As soon as we run a graceful, Apache no longer works and we end up with zombie processes.

For example, after provisioning completes, it works and looks healthy:

05:38 PM vagrant@dev:[~]: ps aux | grep apache
root     28121  0.2  0.6 581932 49412 ?        Ss   17:38   0:00 /usr/sbin/apache2 -k start
www-data 28468  0.5  0.3 584504 29472 ?        S    17:38   0:00 /usr/sbin/apache2 -k start
www-data 28469  0.3  0.3 583900 31360 ?        S    17:38   0:00 /usr/sbin/apache2 -k start
www-data 28470  0.8  0.3 583764 29988 ?        S    17:38   0:00 /usr/sbin/apache2 -k start
www-data 28471  0.0  0.1 581964 13344 ?        S    17:38   0:00 /usr/sbin/apache2 -k start
www-data 28472  0.0  0.1 582164 16088 ?        S    17:38   0:00 /usr/sbin/apache2 -k start
www-data 28542  0.0  0.1 581964 13344 ?        S    17:38   0:00 /usr/sbin/apache2 -k start
www-data 28546  0.0  0.1 581964 13344 ?        S    17:38   0:00 /usr/sbin/apache2 -k start
www-data 28549  0.0  0.1 581964 13344 ?        S    17:38   0:00 /usr/sbin/apache2 -k start
www-data 28610  0.0  0.1 581964 13344 ?        S    17:38   0:00 /usr/sbin/apache2 -k start
vagrant  28847  0.0  0.0  14120  2304 pts/1    R+   17:38   0:00 grep --color=auto apache

Then when we run apache2ctl graceful, Apache becomes unresponsive and we end up with zombie processes:

05:38 PM vagrant@dev:[~]: sudo apache2ctl graceful
05:39 PM vagrant@dev:[~]: ps aux | grep apache
root     28121  0.2  0.4 483124 34008 ?        Ss   17:38   0:00 /usr/sbin/apache2 -k start
www-data 28468  0.4  0.3 584504 29472 ?        S    17:38   0:00 /usr/sbin/apache2 -k start
www-data 28469  0.2  0.3 583900 31360 ?        S    17:38   0:00 /usr/sbin/apache2 -k start
www-data 28470  0.6  0.3 583764 29988 ?        S    17:38   0:00 /usr/sbin/apache2 -k start
www-data 28471  0.0  0.0      0     0 ?        Z    17:38   0:00 [apache2] <defunct>
www-data 28472  0.0  0.1 582164 16088 ?        S    17:38   0:00 /usr/sbin/apache2 -k start
www-data 28542  0.0  0.0      0     0 ?        Z    17:38   0:00 [apache2] <defunct>
www-data 28546  0.0  0.0      0     0 ?        Z    17:38   0:00 [apache2] <defunct>
www-data 28549  0.0  0.0      0     0 ?        Z    17:38   0:00 [apache2] <defunct>
www-data 28610  0.0  0.0      0     0 ?        Z    17:38   0:00 [apache2] <defunct>
vagrant  29023  0.0  0.0  14120  2304 pts/1    R+   17:39   0:00 grep --color=auto apache

What's stranger, is that if we simply run apache2ctl stop and then apache2ctl start, everything looks fine. However, when we deploy to production we need to run a graceful to make sure we don't kill live requests.

When we look at apache error logs, this is the error that appears:

05:39 PM vagrant@dev:[~]: tail -f /var/log/apache2/error.log
[Mon Dec 19 17:38:32.002616 2016] [core:notice] [pid 28121] AH00094: Command line: '/usr/sbin/apache2'
[Mon Dec 19 17:39:01.671918 2016] [mpm_prefork:notice] [pid 28121] AH00171: Graceful restart requested, doing restart

(banana:28121): GLib-GObject-WARNING **: cannot register existing type 'VipsObject'

(banana:28121): GLib-CRITICAL **: g_once_init_leave: assertion 'result != 0' failed

(banana:28121): GLib-GObject-CRITICAL **: g_type_register_static: assertion 'parent_type > 0' failed

(banana:28121): GLib-CRITICAL **: g_once_init_leave: assertion 'result != 0' failed

Any help would be appreciated. We would really like to use Vips with PHP 7 on Apache, but unless we can solve this problem it looks to be a deal breaker.

The only other thing we were confused by was whether or not we needed to install libvips by apt-get install libvips-dev and/or vips-8.3.3.tar.gz (from source). The Ansible playbook above does both.

Thanks in advance!

@ehongyu
Copy link

ehongyu commented Dec 20, 2016

Jesse,

I think the apt-get install libvips-dev should have installed the libvips C library, so there is no need to compile and install the vips-8.3.3.tar.gz source code again for the same library. You can do either one of them, but there is no need to do both for the same task.

I have no idea on the cause of the GLib error in apache restart, however.

@jcupitt
Copy link
Member

jcupitt commented Dec 20, 2016

Hey @jesseforrest, thanks for the bug report, that's very detailed.

The critical thing looks like the gobject error. It looks as if php-vips is being unloaded, but the gobject library under that is not. When php-vips is reloaded it tries to init gobject again, and fails. There's supposed to be logic to prevent double-inits of gobject, but it's obviously not working in this case. I'll see if I can reproduce your error here.

@ehongyu, you're right, you only need one libvips. php-vips needs vips 8.2 or later and I think the one in Ubuntu 14.04 is 7.38, or maybe even earlier. I would remove the Ubuntu libvips package and just use the one built from source.

Have two installed should be fine, but unless you are careful with the paths they can get confused, and that will lead to a range of mysterious crashes.

@jcupitt
Copy link
Member

jcupitt commented Dec 20, 2016

I tried on my Ubuntu 16.10 laptop. I did:

$ sudo apt-get install libvips-dev php-dev
$ sudo pecl install vips

I added extension=vips.so to /etc/php/7.0/apache2/php.ini, did sudo apachectl restart and php appeared to be working and there were no obvious problems in the error log.

I wrote this tiny test page:

<?php

    echo "Hello world!<br>\n";

    $filename = "k2.jpg";

    $x = vips_image_new_from_file($filename)["out"];
    if(!$x) {
        echo "failed to open image<br>\n";
        echo "vips error log is " . vips_error_buffer() . "\n<br>";
    }
    else {
        $width = vips_image_get($x, "width")["out"];

        echo "image " . $filename . " is " . $width . " pixels across\n";
    }
?>

And I can view the page in my browser:

Hello world!
image k2.jpg is 1450 pixels across 

k2.jpg is a test image in the same directory.

If I now do apachectl graceful, apache dies horribly, as you said. I see:

$ sudo apachectl graceful
$ ps aux | grep apa
root      2307  0.1  0.3 401068 29420 ?        Ss   10:03   0:00 /usr/sbin/apache2 -k start
www-data  2312  0.0  0.0      0     0 ?        Z    10:03   0:00 [apache2] <defunct>
www-data  2313  0.0  0.0      0     0 ?        Z    10:03   0:00 [apache2] <defunct>
www-data  2314  0.0  0.0      0     0 ?        Z    10:03   0:00 [apache2] <defunct>
www-data  2315  0.0  0.0      0     0 ?        Z    10:03   0:00 [apache2] <defunct>
www-data  2316  0.0  0.0      0     0 ?        Z    10:03   0:00 [apache2] <defunct>
www-data  2331  0.0  0.0      0     0 ?        Z    10:03   0:00 [apache2] <defunct>

Now apachectl stop doesn't work, I have to kill -9 the root apache process before I can restart. I don't see your helpful message about VipsObject reinit, which is a bit odd.

Anyway, I agree this is a problem and I can reproduce it. I'll investigate.

@jcupitt jcupitt added the bug label Dec 20, 2016
@ehongyu
Copy link

ehongyu commented Dec 20, 2016

Thanks, @jcupitt,

I can see the error message in the Apache error log file :

==> /var/log/apache2/error.log <==
[Mon Dec 19 23:28:51.263866 2016] [mpm_prefork:notice] [pid 12639] AH00171: Graceful restart requested, doing restart

(banana:12639): GLib-GObject-WARNING **: cannot register existing type 'VipsObject'

(banana:12639): GLib-CRITICAL **: g_once_init_leave: assertion 'result != 0' failed

(banana:12639): GLib-GObject-CRITICAL **: g_type_register_static: assertion 'parent_type > 0' failed

(banana:12639): GLib-CRITICAL **: g_once_init_leave: assertion 'result != 0' failed

@jcupitt
Copy link
Member

jcupitt commented Dec 22, 2016

I've been doing some more digging, and it seems to be related to prefork.

The stack is apache loading mod_php, which in turn loads the php-vips-ext extension, which then links to libvips, which then links to glib. php-vips-ext also calls a few glib functions.

As apache starts up, it loads mod_php, which in turn loads and calls MINIT (module init) for all php extensions. The MINIT for vips calls vips_init(), which in turn starts up glib and creates all the base types that vips needs. The main process then fork()s four times to create the four apache workers. This means that MINIT is executed only once, and then copied four times by fork() into the child processes.

On graceful, the four children call MSHUTDOWN (module shutdown) for the vips module and terminate, awaiting the collection of their exit status by the main process.

The main process also does MSHUTDOWN, then unloads mod_php, then reloads it. It seems it's smart enough to see that vips needs to be unloaded and reloaded, since vips appears to be reset at this point, but it doesn't unload and reload glib, which seems to see stuff from the previous run.

Now when libvips starts up again, it thinks it is starting afresh and tries to rebuild everything. It all clashes and breaks, and then the main process locks. With the main process locked, nothing can collect the exit status from the previous children, and they become defunct.

I've tried various combinations of fixes, but not found a working one yet :( I'll keep looking.

@jesseforrest
Copy link
Author

Thanks for the update @jcupitt and for the detailed response. Please let us know if we can help on our end too.

jcupitt added a commit to libvips/php-vips-ext that referenced this issue Dec 27, 2016
"apachctl graceful" was failing: we'd lock during restart.

This extension was being unloaded, which in turn unloaded libvips (since
only we referenced it), but glib was not being unloaded, since other
things used by apache were still using it.

As a result, on reload of this extension, libvips would try to init, and
discover that parts of its previous init were still there, such as the
VipsObject type.

Fix: lock libvips in memory to prevent unload.

See libvips/php-vips#26
@jcupitt
Copy link
Member

jcupitt commented Dec 27, 2016

Hi again, I think I've found a simple fix: it now just locks libvips in memory to block unload on MSHUTDOWN. It seems to work for me, anyway. The actual fix is just this:

libvips/php-vips-ext@d5e34e8#diff-b1cc89f7ef630413232cfad4ba9bdbf2R1624

Would you be able to test? There's a vips-1.0.3.tgz you should be able to install with pecl in git master:

https://github.com/jcupitt/php-vips-ext

@jesseforrest
Copy link
Author

Thanks @jcupitt -- yeah, we can definitely test this out! Will post back after.

@jesseforrest
Copy link
Author

@jcupitt : I was able to build and install the 1.0.3 package. Everything seems to be working now.

02:23 PM root@dev:[/etc/php/7.0/apache2]: service apache2 graceful
 * Reloading web server apache2                                                                                                                                                                        *
02:23 PM root@dev:[/etc/php/7.0/apache2]: ps aux | grep apache2
root     18815  0.1  0.5 541936 46596 ?        Ss   14:22   0:00 /usr/sbin/apache2 -k start
www-data 18818  1.2  0.5 625680 43248 ?        S    14:22   0:00 /usr/sbin/apache2 -k start
www-data 18820  0.4  0.5 772984 40940 ?        Sl   14:22   0:00 /usr/sbin/apache2 -k start
www-data 18966  0.3  0.3 547264 30728 ?        S    14:22   0:00 /usr/sbin/apache2 -k start
www-data 18973  0.5  0.4 546896 33156 ?        S    14:22   0:00 /usr/sbin/apache2 -k start
www-data 18974  1.4  0.5 627888 45836 ?        S    14:22   0:00 /usr/sbin/apache2 -k start
www-data 18977  0.0  0.1 542368 15736 ?        S    14:22   0:00 /usr/sbin/apache2 -k start
www-data 19681  0.0  0.1 542000 12784 ?        S    14:23   0:00 /usr/sbin/apache2 -k start
www-data 19682  0.0  0.1 541968 12784 ?        S    14:23   0:00 /usr/sbin/apache2 -k start
www-data 19683  0.0  0.1 541968 12784 ?        S    14:23   0:00 /usr/sbin/apache2 -k start
www-data 19684  0.0  0.1 541968 12784 ?        S    14:23   0:00 /usr/sbin/apache2 -k start
www-data 19685  0.0  0.1 541968 12784 ?        S    14:23   0:00 /usr/sbin/apache2 -k start
root     19699  0.0  0.0  14116  2392 pts/1    R+   14:23   0:00 grep --color=auto apache2

Thank you so much for putting the time into researching/resolving the problem. Is it possible for you to promote 1.0.3 so that when we do a pecl install vips it will utilize the latest 1.0.3 build?

@jcupitt
Copy link
Member

jcupitt commented Dec 29, 2016

Hi, yes, I was waiting for confirmation before making a new version. I'll update the main pecl repository now.

Thanks for reporting this issue!

@jcupitt
Copy link
Member

jcupitt commented Dec 29, 2016

pecl updated, I'll close.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants