Better error handling in pipelines?

[JoeGermuska]

No doubt I should find better ways to notice what I did wrong in my code, but I found while writing a scraper that uncaught exceptions in tasks just ruins everything.

Can you catch my exceptions for me and tell me what a bozo I am?

Minor, sort of related, could you implement a ctrl-c handler?

[pudo]

Hm, they should really all be caught in the task handler, here:

https://github.com/pudo/scrapekit/blob/master/scrapekit/tasks.py#L132

Can you share a sample of where an exception escapes this impenetrable net?

As for the Ctrl-C thing, I tried to do that before, but it ended up in a way that meant the whole program didn't terminate except with Ctrl-C :( Will dig into it again!

[JoeGermuska]

maybe I just don't understand the pipeline syntax. I've had this working sometimes and then other times it just hangs. When I step through individual tasks in a REPL everything works.

https://github.com/JoeGermuska/pitchfork-scrape

[webmaven]

Is this still an active issue?