Skip to content

Releases: Danceiny/parser_engine

several improvements

17 May 12:09
@Danceiny Danceiny
v0.1.6
7319f00
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
several improvements Pre-release
Pre-release

See CHANGELOG.

  • ItemClassloader的相关优化：

    • 增加load方法，支持绝对路径的类加载
    • PEParser实例化的时候即加载itemname参数对应的类

  • TaskRequest增加了url有效性检查。

  • utils增加了以下方法：

    • is_url(url)
    • item2dict(item)

  • 新增extract_all_keys参数，为真值时可直接提取JSON响应的所有字段（可以理解为，extract_keys=*）

  • 修复PEParser._parse_textbug：在没有parent参数且响应的json不是JSONArray状况下，支持的解析参数与API标准不一致

  • 修复context参数为空时crawled_time字段缺失

  • _parse_start_url调用时传入不存在的tpl_index_or_id时抛出异常，而非静默返回None

  • ItemClassLoader的settings使用普通get方法获取具体配置项，以支持dict类型的settings参数

Assets 4
Loading

fix several bugs; update examples

28 Mar 16:03
@Danceiny Danceiny
v0.1.2
4d90660
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
fix several bugs; update examples Pre-release
Pre-release
  • 修复了JSON解析器的bug
  • PEField新增以下特性：
    - mapper 用于支持将枚举类型的抓取字段映射为想要的值
    - join 用于支持将列表类型的抓取字段连接成字符串
  • 修复了其他若干bug
  • review并精简了示例项目的代码，优化了相关文档
Assets 3
Loading

builtin Clue Mechanism

22 Mar 08:57
@Danceiny Danceiny
v0.0.9
d22b307
Compare
Choose a tag to compare
builtin Clue Mechanism Pre-release
Pre-release

内建的线索机制

  • 完善的spider、item、model、pipeline支持
  • 作为独立的包存在
  • 支持在scrapy的settings.py中设置数据库连接

其他更新

  • 增强了scrapy_redis的redis，新增支持sentinel模式的连接
  • 修复了setup.py依赖无法安装的问题
Assets 3
Loading