pip 1.4 doesn't upgrade setuptools properly

[comandrei]

I tried upgrading to setuptools as per #1064
pip install --use-mirrors setuptools --upgrade

Downloading/unpacking distribute from https://pypi.python.org/packages/source/d/distribute/distribute-0.7.3.zip#md5=c6c59594a7b180af57af8a0cc0cf5b4a
  Downloading distribute-0.7.3.zip (145kB): 145kB downloaded
  Running setup.py egg_info for package distribute

Downloading/unpacking setuptools>=0.7 (from distribute)
  Downloading setuptools-0.9.8.tar.gz (766kB): 766kB downloaded
  Running setup.py egg_info for package setuptools

Installing collected packages: setuptools, distribute
  Found existing installation: distribute 0.6.27
    Uninstalling distribute:
      Successfully uninstalled distribute
  Running setup.py install for setuptools

    Installing easy_install script to /usr/bin
    Installing easy_install-2.7 script to /usr/bin
  Found existing installation: distribute 0.6.27
    Can't uninstall 'distribute'. No files were found to uninstall.
  Running setup.py install for distribute

Successfully installed setuptools distribute
Cleaning up...

pip install --use-mirrors --upgrade virtualenv==1.10

Downloading/unpacking virtualenv==1.10
  Downloading virtualenv-1.10.tar.gz (1.3MB): 1.3MB downloaded
  Running setup.py egg_info for package virtualenv
    /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'entry_points'
      warnings.warn(msg)
    /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'zip_safe'
      warnings.warn(msg)
    /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'test_suite'
      warnings.warn(msg)
    /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'tests_require'
      warnings.warn(msg)
    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: -c --help [cmd1 cmd2 ...]
       or: -c --help-commands
       or: -c cmd --help

    error: invalid command 'egg_info'
    Complete output from command python setup.py egg_info:
    /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'entry_points'

  warnings.warn(msg)

/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'zip_safe'

  warnings.warn(msg)

/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'test_suite'

  warnings.warn(msg)

/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'tests_require'

  warnings.warn(msg)

usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]

   or: -c --help [cmd1 cmd2 ...]

   or: -c --help-commands

   or: -c cmd --help



error: invalid command 'egg_info'

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /root/.pip/build/virtualenv
Storing complete log in /root/.pip/pip.log

This happens on a fresh install of CentOS 6.4 x86_64 (Python 2.7.3)

Also any subsequent attempts to install other packages fail just like installing virtualenv.

[comandrei]

After installing setuptools with easy_install setuptools -U everything seems to be working, so this looks like pip isn't actually installing setuptools.

[qwcode]

the upgrade has been validated to work many times (with older pip's and pip-1.4), so there must be something different in this case.

do you mind:

1. running pip list after the upgrade is done
2. ls your resulting site-packages dir
3. ls the directory you're running pip from when running the failing virtualenv install

this error looks familiar in that I've seen that it when running "egg_info" in the distribute-0.7.3 build dir, which won't work.

[sdake]

Apologies to necromance this bug, but python-setuptools-0.9.8 shipped with CentOS 7.2 does not properly upgrade to setuptools-20-y.z. when pip is used. The workaround used in this document (using easyinstall) gets the job done, but seems wrong. After I installed setuptools-20.y.z, I removed all of the python noarch packages, and then pip properly install setuptools-20.y.z. I think the issue is related to some type of pip database on teh system, but I'm not really sure not being a python expert.

[sdake]

Our upstream solution to this problem
https://review.openstack.org/#/c/298479/

[Ivoz]

@sdake you could also use the get-pip.py script to bootstrap installation of the latest tools.

You could even only install the latest pip:

$ python get-pip.py --no-setuptools --no-wheel
$ pip install -U setuptools wheel

[Ivoz]

I've also seen this as a part of using Docker with overlayfs, actually completely unrelated to pip. overlayfs somehow can't deal with a command doing both an uninstall & install operation (as an upgrade implies), although I am fuzzy on the details.

#2953

[sdake]

@Ivoz We do that. Our full bootstrap is here:
https://github.com/openstack/kolla/blob/master/docker/kolla-toolbox/Dockerfile.j2#L36

The issue is in a parent container (base), jinja2 is installed which has a hard python dependency on setuptools:
https://github.com/openstack/kolla/blob/master/docker/base/Dockerfile.j2#L142

Why not just use get-pip here and install these packages from source? I guess we could but we ultimately decided to hold off on doing that and install source packages in a virtualenv in our base package as is done in our openstack-base. That work was never done and I guess we should do it now. The virtualenv is here:
https://github.com/openstack/kolla/blob/master/docker/openstac-base/Dockerfile.j2#L228

The reflection on this bug from my standpoint is you can't mix distro packaging with pip that involves setuptools or everything goes south.

Regards
-steve