parse_html.activity: about 30% speedup for html parsing #66
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
A few optimizations - using `.find` instead of `.select` is faster since it's not using CSS selectors - using `SoupStrainer` is faster since it only does partial parsing and avoids materializing parts of soup we don't actually use Measurements on a big `Chrome/MyActivity.html` file - before - parsing (up to for loop over `outer_divs`: 17s - processing (everything in for loop): 16s - after - parsing: 13s - procesing: 11s
karlicoss
commented
Sep 10, 2024
| @@ -88,7 +88,7 @@ def _serialize_default(obj: Any) -> Any: | |||
| if isinstance(obj, Exception): | |||
| return {"type": type(obj).__name__, "value": str(obj)} | |||
| elif dataclasses.is_dataclass(obj): | |||
| d = dataclasses.asdict(obj) | |||
| d = dataclasses.asdict(obj) # type: ignore[call-overload] # see https://github.com/python/mypy/issues/17550 | |||
There was a problem hiding this comment.
this is irrelevant to my changes, must be due to mypy upgrade (apparently is_dataclass refines type to union of dataclass instance and dataclass type...
|
thanks! will parse against my old takeouts just to make sure nothing breaks and merge in a bit |
|
I was checking against quite old ones, so hopefully should be ok, but thanks for double checking :) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I've been reviewing some older takeouts, so had cachew off and parsing was a bit painful... so had a quick look in a profiler and did some optimization
A few optimizations
.findinstead of.selectis faster since it's not using CSS selectorsSoupStraineris faster since it only does partial parsing and avoids materializing parts of soup we don't actually useMeasurements on a big
Chrome/MyActivity.htmlfilebefore
outer_divs: 17safter