4682 Updated cl_send_alerts to support V2 webhooks while maintaining compatibility with V1 webhooks

[albertisfu]

As planned in #4682, this PR updates the cl_send_alerts command to support V2 of the Opinion Search Alerts Webhooks.

The following changes have been applied:

• All Opinion Search Alerts emails now use the new email format, which displays nested opinions within the Opinion Cluster. This is achieved doing an ES parent-child query all the time as main query.
• If the user has a V1 Webhook enabled, an additional query is performed to retrieve plain results for the alert. This is executed via the MultiSearch API, allowing both the parent-child query and the plain query to be requested in a single request.
  • Nested results from the main query are used to render the email and V2 Webhook.
  • Plain results are used to send V1 Webhooks.
  • It's important to note that V1 Webhooks use a collapse query to group Opinions belonging to the same cluster. This aligns with V3 of the Opinions Search API running on ES, meaning that the same ES V3 API backwards incompatible changes for opinions apply to V1 Webhooks.
• These changes will take effect once we switch to ES for Opinions. Until that happens, we will continue sending only V1 Webhooks using Solr and the regular plain email, even if users have enabled a V2 Webhook endpoint for Search Alerts.
• Refactored the code used to merge child documents in nested queries to eliminate duplicate and redundant code across the codebase.
• Updated the Opinion Search Alerts tests accordingly.

Below are examples of the updated Opinion Search Alert Email and Webhooks:

*****************
CourtListener.com
*****************

-------------------------------------------------------
We have news regarding your alerts at CourtListener.com
-------------------------------------------------------

Your daily opinion alert -- Test Alert O Disabled -- had 1 hit:
-------------------------------------------------------
View Full Results / Edit this Alert: https://www.courtlistener.com/?type=o&stat_Unpublished=on&order_by=score+desc&filed_after=2024-11-14&edit_alert=23
Disable this Alert (one click): https://www.courtlistener.com/alert/disable/vAFtMQ10lUvXLwCqCqhxzTm4EBcWDDsTJFHinvyS/

1. California vs Lorem (Bankr. C.D. Cal. 2024)

Lead Opinion
   ...Lorem dolor sit amet, consectetur adipiscing elit hearing....


Combined Opinion
   ...Lorem dolor california sit amet, consectetur adipiscing elit....
    - Download original from the court: https://ca.flcourts.gov/
    - Download the original from our backup: https://storage.courtlistener.com/test/search/opinion_html.html

~~~~~
 - View this item on our site: https://www.courtlistener.com/opinion/11/california-vs-lorem/


************************
This alert brought to you by the 501(c)(3) non-profit Free Law Project
 - Blog: https://free.law
 - BlueSky: https://bsky.app/profile/free.law
 - X: https://x.com/freelawproject
 - Donate: https://donate.free.law/forms/supportflp
 - Become a Member: https://donate.free.law/forms/membership

Please donate to support our work.

Webhook V1

{
   "payload":{
      "alert":{
         "id":24,
         "name":"Test Alert 2 O Enabled",
         "rate":"dly",
         "user":14,
         "query":"q=California hearing&type=o&stat_Unpublished=on",
         "alert_type":"o",
         "secret_key":"LmdVe5PURP0mehQSmDdmdMn6OeaLBafHEdMFpMmB",
         "date_created":"2024-11-18T16:22:15.756423-08:00",
         "date_last_hit":"2024-11-14T16:22:15.765337-08:00",
         "date_modified":"2024-11-14T16:22:15.765337-08:00"
      },
      "results":[
         {
            "id":13,
            "type":"020lead",
            "cites":"None",
            "court":"Superior court of the Dirty Dishes",
            "judge":"Lorem Judge",
            "status":"Non-Precedential",
            "scdb_id":"",
            "snippet":"Lorem dolor sit amet, consectetur adipiscing elit <strong>hearing</strong>.",
            "attorney":"Attorney General of North Carolina",
            "caseName":"California vs Lorem",
            "citation":[
               "10 Yeates 4",
               "10 Neutral 4",
               "33 state 1"
            ],
            "court_id":"canb",
            "author_id":16,
            "citeCount":1,
            "dateFiled":"2024-11-14T00:00:00-08:00",
            "docket_id":13,
            "lexisCite":"10 Yeates 4",
            "panel_ids":[
               15
            ],
            "timestamp":"2024-11-14T16:22:15.765337-08:00",
            "cluster_id":11,
            "dateArgued":"1986-04-23T00:00:00-08:00",
            "local_path":"None",
            "per_curiam":false,
            "suitNature":"",
            "court_exact":"canb",
            "neutralCite":"10 Neutral 4",
            "sibling_ids":[
               13,
               14
            ],
            "absolute_url":"/opinion/11/california-vs-lorem/",
            "dateReargued":"2024-11-14T00:00:00-08:00",
            "date_created":"2024-11-14T16:22:15.765337-08:00",
            "docketNumber":"2:13-bk-70798",
            "download_url":"None",
            "joined_by_ids":[
               15
            ],
            "dateReargumentDenied":"2024-11-14T00:00:00-08:00",
            "court_citation_string":"Bankr. C.D. Cal."
         }
      ]
   },
   "webhook":{
      "version":1,
      "event_type":2,
      "date_created":"2024-11-19T00:22:15.753650+00:00",
      "deprecation_date":"2024-11-18T00:00:00+00:00"
   }
}

Webhook V2

{
   "payload":{
      "alert":{
         "id":17,
         "name":"Test Alert O",
         "rate":"dly",
         "user":11,
         "query":"q=California&type=o&stat_Unpublished=on",
         "alert_type":"o",
         "secret_key":"ivjg1J15qU8jgE70QHuI4XQkUciVuF7BBcmjAbDE",
         "date_created":"2024-11-18T16:22:15.707344-08:00",
         "date_last_hit":"2024-11-14T16:22:15.765337-08:00",
         "date_modified":"2024-11-14T16:22:15.765337-08:00"
      },
      "results":[
         {
            "meta":{
               "timestamp":"2024-11-15T00:22:15.765337Z",
               "date_created":"2024-11-15T00:22:15.765337Z"
            },
            "court":"Superior court of the Dirty Dishes",
            "judge":"Lorem Judge",
            "source":"CR",
            "status":"Unpublished",
            "posture":"",
            "scdb_id":"",
            "attorney":"Attorney General of North Carolina",
            "caseName":"<strong>California</strong> vs Lorem",
            "citation":[
               "10 Yeates 4",
               "10 Neutral 4",
               "33 state 1"
            ],
            "court_id":"canb",
            "opinions":[
               {
                  "id":14,
                  "meta":{
                     "timestamp":"2024-11-15T00:22:15.765337Z",
                     "date_created":"2024-11-15T00:22:15.765337Z"
                  },
                  "sha1":"d7bcc865b883abd70d74d9af7578d256ae62a973",
                  "type":"combined-opinion",
                  "cites":[
                     13
                  ],
                  "snippet":"Lorem dolor <strong>california</strong> sit amet, consectetur adipiscing elit.",
                  "author_id":17,
                  "local_path":"test/search/opinion_html.html",
                  "per_curiam":false,
                  "download_url":"https://ca.flcourts.gov/",
                  "joined_by_ids":[
                     
                  ]
               },
               {
                  "id":13,
                  "meta":{
                     "timestamp":"2024-11-15T00:22:15.765337Z",
                     "date_created":"2024-11-15T00:22:15.765337Z"
                  },
                  "sha1":"58ccc5a2c60c28bf04fe8efc78547270c360f208",
                  "type":"lead-opinion",
                  "cites":[
                     
                  ],
                  "snippet":"Lorem dolor sit amet, consectetur adipiscing elit hearing.",
                  "author_id":16,
                  "local_path":"None",
                  "per_curiam":false,
                  "download_url":"None",
                  "joined_by_ids":[
                     15
                  ]
               }
            ],
            "syllabus":"",
            "citeCount":1,
            "dateFiled":"2024-11-14",
            "docket_id":13,
            "lexisCite":"10 Yeates 4",
            "panel_ids":[
               15
            ],
            "cluster_id":11,
            "dateArgued":"1986-04-23",
            "suitNature":"",
            "neutralCite":"10 Neutral 4",
            "panel_names":[
               "Kristine Pena Mendez Jr."
            ],
            "sibling_ids":[
               13,
               14
            ],
            "absolute_url":"/opinion/11/california-vs-lorem/",
            "caseNameFull":"Todd Ballard, Julie Smith, Jeanne Peters, Ashley Jordan, and Andrew Thompson v. Johnson LLC, Walker-Bernard, Jones PLC, Bailey Group, and Lopez-Gonzales",
            "dateReargued":"2024-11-14",
            "docketNumber":"2:13-bk-70798",
            "procedural_history":"",
            "dateReargumentDenied":"2024-11-14",
            "court_citation_string":"Bankr. C.D. Cal.",
            "non_participating_judge_ids":[
               
            ]
         }
      ]
   },
   "webhook":{
      "version":2,
      "event_type":2,
      "date_created":"2024-11-19T00:22:15.511388+00:00",
      "deprecation_date":"None"
   }
}

Let me know what do you think.

[mlissner]

Looks right to me after a quick skim, thanks Alberto! Onward to Eduardo for full review!

[mlissner]

Process notes (@flooie and @s-taube, you might want to perk your ears, so we're all on the same page):

• I assigned to Eduardo so it shows up in his "By Assignee" tab.
• I set Eduardo as the reviewer, because he is.
• I moved it to the To Do column so it shows up in Eduardo's todo column.
• @ERosendo, when you finish your review, see if you can assign this back to Alberto, so it shows up in his queue, and then back and forth until you're done.

I think this should work, but I'm still experimenting. Definitely curious what we think.

[ERosendo]

@albertisfu LGTM, just one question.

[ERosendo]

We used this loop to combine child documents from nested queries into the parent. I noticed you removed the loop in a different file and used the new set_results_child_docs method instead. In this case, we're simply removing the logic. Won't this change impact the send_recap_alerts command? We're accessing the child_doc property as a dictionary key in the process_alert_hits helper function.

[albertisfu]

Good question! I removed this code because it was redundant. The do_es_sweep_alert_query method already calls limit_inner_hits, which sets the child_docs key in the results. Therefore, this code was no longer necessary. I realized that we only need to perform this conversion to use a defaultdict on child_docs when the results are an AttrDict instance, and they are going to be serialized for rendering an API response. That’s why it’s not required here.

This observation led me to realize that set_child_docs_and_score was more complex than it needed to be. If the results are already a dictionary, there’s no need to convert them again. As a result, I refactored that method and also moved the logic for assigning the BM25 score here.

[albertisfu]

Thanks, @ERosendo! I’ve updated the branch and resolved the merge conflicts.

• I also refactored the set_child_docs_and_score method.
• Removed the score field from Opinion Search Alerts webhooks.
• Updated some failing tests.