I have an almost finished webapp that is used to scrape data from a website, and serve it up over an API. When it runs, more often than not it doesn't complete and gives an error message. I need someone who is comfortable with all the things in the project name to get this thing running reliably.
A few months ago I built a project just like this. Could I see your code? Selenium hub and celeryd sound like too much, maybe crossing critixal sections. Do you have unit tests, like with pyunit?