Prune Xpath or Xpath Target for Spider Function

[felipehertzer]

Hello @adbar,

I am currently testing the Spider function, but I encountered some difficulties in focusing the spider on a specific HTML container. Additionally, I'm exploring the possibility of excluding headers, footers, and other elements from the search. Do you think these adjustments could be feasible with the Spider function?

Thanks.

[adbar]

Hi @felipehertzer, the main functions are extract_links() and filter_links() in courlan.core. I see they are not really documented, I'm going to work on that.

They take a string (HTML) as input so you could pass the section you want to the link extraction function, you can also use Trafilatura's XPaths and then the extraction. Keep me updated, if you find a convenient way to do it we could work on a PR.

See also this issue: #290.

[felipehertzer]

Added feature on #684