Plumbing Life's Depths - easy_install no longer working with SourceForge-hosted projects?

It seems like all of the projects I host on SourceForge are no longer easy-installable. The only ones that can install are the ones that have been uploaded to PyPI, and then only the versions on PyPI install (i.e. a later version on SF doesn't install or prevent an earlier version from installing). I'm guessing this is a further iteration in the "war" of hacks that involves downloading HTML and parsing it in order to find links; SF's recent restructuring of their files-for-download stuff probably just broke the parser-based stuff (likely SF is using AJAX to populate the sets on the pages).

So maybe I just switch to PyPI upload for everything. I'm guessing that some PSF donor is paying for the bandwidth/storage for PyPI. I don't have a huge bandwidth footprint for the set of projects I maintain. Or maybe I should set up a simple "downloads" page that can be easily parsed by easy_install with direct links for each version of each project and update the download links in the project metadata to point to that (ick). Or maybe use Launchpad's file-releases?

Ah well, not going to solve this this afternoon. Client projects beckon.

Comments

Martijn Faassen on 08/18/2009 3:11 p.m. #

I think hosting on PyPI is the best way to go. Anything that relies on going off to some other URL introduces more chances of something going wrong. I personally think we'd all have been better off if easy_install (setuptools) didn't try to be so clever at finding stuff, though I can see that it was needed for bootstrapping as well.
Marius Gedminas on 08/18/2009 4:38 p.m. #

PyPI sdist uploads == goodness.
Richard Jones on 08/18/2009 7:56 p.m. #

This is interesting - I wonder how many other people are wary of uploading to PyPI because of perceived hosting limitations? And how do we dispel those myths?
Dave Brondsema on 08/19/2009 9:04 a.m. #

Hi, I work at SourceForge, Inc. We didn't intend to break pypi downloads from SourceForge during the recent changes, obviously. And while I concur with http://simonwillison.net/2009/Aug/19/easyinstall/ if its not too difficult we should be able to adjust the downloading page to make it work again (AJAX isn't used for anything required for downloading). I'm not familiar with how easy_install scrapes and decides which URL to use, so if someone could point me where to look for that or what should change, that'd be great.

URLs like these (for example) should work:

http://sourceforge.net/projects/pytidylib/files/pytidylib-0.1.2.tar.gz/download

http://downloads.sourceforge.net/project/pytidylib/pytidylib-0.1.2.tar.gz

For automated scripts (no referer) those redirect straight to the file.

Note: for official support from SourceForge, please see http://sourceforge.net/support
Mike C. Fletcher on 08/21/2009 1:52 p.m. #

Hi Dave,

Thanks for replying. I *also* happen to agree that link-parsing is a hacky way to handle the situation, but whatever, the "boot strap" solution has become accepted practice.

I honestly don't know how easy_install processes the pages. I'm "off" on thesis work this weekend, but I'll try to look into it and get back to you next week.

I'm pretty much convinced what we should do is migrate all of the packages into the PyPI archives and then set up multiple mirrors for the archives, but getting SF hosted projects working in the short term would certainly help too.