Build summary gprc #541

mudiagaobrikisil · 2024-11-18T07:34:38Z

This change is

… add_serval_release_version

codecov-commenter · 2024-11-18T07:55:51Z

Codecov Report

Attention: Patch coverage is 28.26087% with 33 lines in your changes missing coverage. Please review.

Project coverage is 56.94%. Comparing base (03e4716) to head (9e18ec5).

Files with missing lines	Patch %	Lines
...anslation/Services/TranslationPlatformServiceV1.cs	5.55%	17 Missing ⚠️
...l.Machine.Shared/Services/ServalPlatformService.cs	0.00%	11 Missing ⚠️
...red/Services/ServalPlatformOutboxMessageHandler.cs	0.00%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #541      +/-   ##
==========================================
- Coverage   57.03%   56.94%   -0.09%     
==========================================
  Files         302      302              
  Lines       15639    15683      +44     
  Branches     2155     2157       +2     
==========================================
+ Hits         8919     8930      +11     
- Misses       6076     6109      +33     
  Partials      644      644

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

johnml1135 · 2024-11-19T13:52:40Z

src/Machine/src/Serval.Machine.Shared/Models/Build.cs line 32 at r1 (raw file):

    public required BuildStage Stage { get; init; }
    public string? Options { get; set; }
    public Dictionary<string, int>? Statistics { get; set; }

@ddaspit - what do you think of the name "Statistics"? It is trying to capture any types of information created throughout the build and inference process, including (int the future) possibly warnings or other context.

johnml1135 · 2024-11-19T13:53:16Z

src/Machine/src/Serval.Machine.Shared/Services/PostprocessBuildJob.cs line 36 at r1 (raw file):

        }

        // here's where to add more data to send

This comment needs to be removed (or addressed).

johnml1135 · 2024-11-19T13:55:32Z

src/Serval/src/Serval.Translation/Controllers/TranslationEnginesController.cs line 1241 at r1 (raw file):

    }

    // private async Task<(int TrainCount, int PretranslateCount)> GetBuildCountsFromLogAsync(

All commented out code should be removed.

johnml1135 · 2024-11-19T13:56:35Z

src/Serval/src/Serval.Translation/Models/Build.cs line 21 at r1 (raw file):

    public string? DeploymentVersion { get; init; }
    public Dictionary<string, string> Statistics { get; init; } =
        new Dictionary<string, string>() { { "trainCount", "0" } };

I would initialize the statistics to an empty array.

johnml1135 · 2024-11-19T13:58:35Z

src/Serval/test/Serval.E2ETests/ServalApiTests.cs line 142 at r1 (raw file):

        TranslationBuild build = await _helperClient.TranslationEnginesClient.GetCurrentBuildAsync(engineId);
        Assert.That(build.Statistics, Is.Not.Null);
        Assert.That(build.Statistics, Contains.Key("trainCount"));

I think that a unit test it likely not needed.

johnml1135 · 2024-11-19T13:59:25Z

src/Serval/test/Serval.E2ETests/ServalApiTests.cs line 142 at r1 (raw file):

Previously, johnml1135 (John Lambert) wrote…

I think that a unit test it likely not needed.

But an E2E test is needed - add to an existing E2E test such as NmtBatch and verify that the build statistics is properly created.

johnml1135 · 2024-11-19T13:59:43Z

src/Serval/src/Serval.Client/Client.g.cs line 4552 at r1 (raw file):

        /// <returns>The build summary</returns>
        /// <exception cref="ServalApiException">A server side error occurred.</exception>
        System.Threading.Tasks.Task<TranslationBuildSummary> GetBuildSummaryAsync(string id, string buildId, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken));

See comment in controller.

johnml1135 · 2024-11-19T14:00:34Z

src/Serval/src/Serval.Translation/Models/TranslationBuildSummary.cs line 7 at r1 (raw file):

    public int LinesTrainedOn { get; set; }
    public int LinesPretranslated { get; set; }
    public TimeSpan TotalBuildTime { get; set; }

This data structure is not needed.

johnml1135 · 2024-11-19T14:03:00Z

src/Serval/src/Serval.Translation/Models/Build.cs line 16 at r1 (raw file):

    public int? QueueDepth { get; init; }
    public JobState State { get; init; } = JobState.Pending;
    public DateTime DateStarted { get; init; } = DateTime.UtcNow;

This can be fine - it allows the users to calculated the total time between creation and date finished. Still, I would name this "Date created".

johnml1135 · 2024-11-19T14:04:02Z

src/Serval/src/Serval.Translation/Controllers/TranslationEnginesController.cs line 1198 at r1 (raw file):

    }

    /// <summary>

This is actually not needed. The build summary should be present when just normally getting the build.

To do this, you will need to update BuildDto and the map function that maps the fields in the internal "Build" data structure to the BuildDto "data transfer object".

johnml1135

Reviewed 15 of 15 files at r1, all commit messages.
Reviewable status: all files reviewed, 9 unresolved discussions (waiting on @mudiagaobrikisil)

johnml1135

The comments still need to be resolved.

Reviewable status: all files reviewed, 9 unresolved discussions (waiting on @mudiagaobrikisil)

mudiagaobrikisil · 2024-11-26T05:36:25Z

Hi John, Will update the PR for this today and have a call with you when you are available. Thanks.

…

On Mon, 25 Nov 2024, 10:45 pm John Lambert, ***@***.***> wrote: ***@***.**** requested changes on this pull request. The comments still need to be resolved. *Reviewable <https://reviewable.io/reviews/sillsdev/serval/541#-:-OC_H6dE7kEQq8mN5l6n:bhzceys>* status: all files reviewed, 9 unresolved discussions (waiting on @mudiagaobrikisil <https://github.com/mudiagaobrikisil>) — Reply to this email directly, view it on GitHub <#541 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BLIZ45LJF5ST6G7JW4DYWVL2COK55AVCNFSM6AAAAABR67I2LGVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDINJZGY4TONBWGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

johnml1135 · 2024-11-26T14:01:04Z

src/Machine/src/Serval.Machine.Shared/Models/Build.cs line 32 at r1 (raw file):

Previously, johnml1135 (John Lambert) wrote…

@ddaspit - what do you think of the name "Statistics"? It is trying to capture any types of information created throughout the build and inference process, including (int the future) possibly warnings or other context.

runData?

mudiagaobrikisil

Reviewable status: 6 of 17 files reviewed, 9 unresolved discussions (waiting on @johnml1135)

src/Serval/src/Serval.Client/Client.g.cs line 4552 at r1 (raw file):

Previously, johnml1135 (John Lambert) wrote…

See comment in controller.

Done.

src/Serval/src/Serval.Translation/Controllers/TranslationEnginesController.cs line 1198 at r1 (raw file):

Previously, johnml1135 (John Lambert) wrote…

This is actually not needed. The build summary should be present when just normally getting the build.

To do this, you will need to update BuildDto and the map function that maps the fields in the internal "Build" data structure to the BuildDto "data transfer object".

Done.

src/Serval/src/Serval.Translation/Models/Build.cs line 21 at r1 (raw file):

Previously, johnml1135 (John Lambert) wrote…

I would initialize the statistics to an empty array.

Done.

src/Machine/src/Serval.Machine.Shared/Services/PostprocessBuildJob.cs line 36 at r1 (raw file):

Previously, johnml1135 (John Lambert) wrote…

This comment needs to be removed (or addressed).

Done.

src/Serval/src/Serval.Translation/Models/TranslationBuildSummary.cs line 7 at r1 (raw file):

Previously, johnml1135 (John Lambert) wrote…

This data structure is not needed.

Done.

johnml1135 · 2024-11-26T16:11:34Z

src/Serval/src/Serval.Translation/Services/TranslationPlatformServiceV1.cs line 279 at r3 (raw file):

        }

        var newStatistics = new Dictionary<string, string>();

Why was this change made? I am concerned that it will not accurately update parameters, but rather drop existing ones. Some thing you can do is make a unit test for this to ensure that the following happens:
Starting data:

A: Init
B: Init
Updated data:
B: Update
C: Update
Final data:
A: Init
B: Update
C: Update

johnml1135 · 2024-11-26T16:13:28Z

src/Serval/test/Serval.E2ETests/ServalApiTests.cs line 142 at r1 (raw file):

Previously, johnml1135 (John Lambert) wrote…

But an E2E test is needed - add to an existing E2E test such as NmtBatch and verify that the build statistics is properly created.

Correction: this is an E2E test (I read it wrong). All statistics fields and values should be confirmed (both the train and pretranslate count).

johnml1135 · 2024-11-26T16:15:36Z

Updates needed:

E2E test as per the conversation
The merging Statistics logic looks off. I also think that a unit test could be written. If you don't know how to approach it, we can talk.
@ddaspit - what do you think of the name "Statistics"? I am inclined to use RunData - or something that can also include things like warnings or other data gathered during the running of the build/job.

johnml1135

Reviewed 10 of 11 files at r3, all commit messages.
Reviewable status: 16 of 17 files reviewed, 3 unresolved discussions (waiting on @mudiagaobrikisil)

ddaspit

Reviewed 6 of 15 files at r1, 11 of 11 files at r3, all commit messages.
Reviewable status: all files reviewed, 6 unresolved discussions (waiting on @johnml1135 and @mudiagaobrikisil)

src/Machine/src/Serval.Machine.Shared/Models/Build.cs line 32 at r1 (raw file):

Previously, johnml1135 (John Lambert) wrote…

runData?

Maybe ExecutionData?

src/Machine/src/Serval.Machine.Shared/Services/IPlatformService.cs line 34 at r3 (raw file):

        string engineId,
        string buildId,
        IDictionary<string, string> statistics,

I would make this a IReadOnlyDictionary.

src/Serval/src/Serval.Translation/Models/Build.cs line 20 at r3 (raw file):

    public IReadOnlyDictionary<string, object>? Options { get; init; }
    public string? DeploymentVersion { get; init; }
    public Dictionary<string, string>[] Statistics { get; init; } = [];

I'm not sure I understand why this is an array of dictionaries. This should be a single dictionary that gets updated with the most recent values.

src/Serval/src/Serval.Translation/Services/TranslationPlatformServiceV1.cs line 287 at r3 (raw file):

        var updatedStatistics = build.Statistics.Concat(new[] { newStatistics }).ToArray();

        await _builds.UpdateAsync(

The update needs to be atomic or we will face race conditions. We should merge newStatistics into the existing statistics of the model. We will need to add a new "merge" operator to the IUpdateBuilder interface to support this.

mudiagaobrikisil

Reviewable status: all files reviewed, 6 unresolved discussions (waiting on @ddaspit and @johnml1135)

src/Serval/src/Serval.Translation/Models/Build.cs line 20 at r3 (raw file):

Previously, ddaspit (Damien Daspit) wrote…

I'm not sure I understand why this is an array of dictionaries. This should be a single dictionary that gets updated with the most recent values.

Just to be sure. Will it keep track of the previous statistics?

src/Serval/src/Serval.Translation/Services/TranslationPlatformServiceV1.cs line 287 at r3 (raw file):

Previously, ddaspit (Damien Daspit) wrote…

The update needs to be atomic or we will face race conditions. We should merge newStatistics into the existing statistics of the model. We will need to add a new "merge" operator to the IUpdateBuilder interface to support this.

Noted

johnml1135 · 2024-11-27T14:27:18Z

src/Machine/src/Serval.Machine.Shared/Models/Build.cs line 32 at r1 (raw file):

Previously, ddaspit (Damien Daspit) wrote…

Maybe ExecutionData?

I like it.

johnml1135 · 2024-11-27T14:50:51Z

src/Machine/src/Serval.Machine.Shared/Models/Build.cs line 32 at r1 (raw file):

Previously, johnml1135 (John Lambert) wrote…

I like it.

Please change all instances of "Statistics" to "ExecutionData".

johnml1135

Reviewed 1 of 11 files at r3.
Reviewable status: all files reviewed, 6 unresolved discussions (waiting on @ddaspit and @mudiagaobrikisil)

mudiagaobrikisil

Done

Reviewable status: all files reviewed, 6 unresolved discussions (waiting on @ddaspit and @johnml1135)

src/Machine/src/Serval.Machine.Shared/Models/Build.cs line 32 at r1 (raw file):

Previously, johnml1135 (John Lambert) wrote…

Please change all instances of "Statistics" to "ExecutionData".

Done

src/Machine/src/Serval.Machine.Shared/Services/IPlatformService.cs line 34 at r3 (raw file):

Previously, ddaspit (Damien Daspit) wrote…

I would make this a IReadOnlyDictionary.

Done

src/Serval/src/Serval.Translation/Controllers/TranslationEnginesController.cs line 1241 at r1 (raw file):

Previously, johnml1135 (John Lambert) wrote…

All commented out code should be removed.

Done

src/Serval/src/Serval.Translation/Models/Build.cs line 16 at r1 (raw file):

Previously, johnml1135 (John Lambert) wrote…

This can be fine - it allows the users to calculated the total time between creation and date finished. Still, I would name this "Date created".

Done

src/Serval/src/Serval.Translation/Models/Build.cs line 20 at r3 (raw file):

Previously, mudiagaobrikisil wrote…

Just to be sure. Will it keep track of the previous statistics?

Done

src/Serval/src/Serval.Translation/Services/TranslationPlatformServiceV1.cs line 279 at r3 (raw file):

Previously, johnml1135 (John Lambert) wrote…

Why was this change made? I am concerned that it will not accurately update parameters, but rather drop existing ones. Some thing you can do is make a unit test for this to ensure that the following happens:
Starting data:

A: Init

B: Init
Updated data:

B: Update

C: Update
Final data:

A: Init

B: Update

C: Update

E2E test done but need a little help with knowing how to track the init and final values in the unit tests, since the ExecutionData is initialized to an empty dictionary

src/Serval/test/Serval.E2ETests/ServalApiTests.cs line 142 at r1 (raw file):

Previously, johnml1135 (John Lambert) wrote…

Correction: this is an E2E test (I read it wrong). All statistics fields and values should be confirmed (both the train and pretranslate count).

Done

Enkidu93

Reviewed 4 of 11 files at r3, 13 of 13 files at r4, all commit messages.
Reviewable status: all files reviewed, 7 unresolved discussions (waiting on @ddaspit, @johnml1135, and @mudiagaobrikisil)

src/Serval/src/Serval.Grpc/Protos/serval/translation/v1/platform.proto line 67 at r4 (raw file):

    string engine_id = 1;
    string build_id = 2;
    map<string, string> ExecutionData = 3;

Why pascal case here?

…ummary_gprc

mudiagaobrikisil

Reviewable status: 13 of 17 files reviewed, 7 unresolved discussions (waiting on @ddaspit, @Enkidu93, and @johnml1135)

src/Serval/src/Serval.Grpc/Protos/serval/translation/v1/platform.proto line 67 at r4 (raw file):

Previously, Enkidu93 (Eli C. Lowry) wrote…

Why pascal case here?

Sorry. Was an oversight. Done

johnml1135 · 2024-12-03T01:43:02Z

-- commits line 107 at r5:
Please rebase these into a single commit on the latest main branch commit. I can show you how to do that.

johnml1135 · 2024-12-03T01:44:38Z

src/Machine/src/Serval.Machine.Shared/Services/PreprocessBuildJob.cs line 192 at r3 (raw file):

            int skipCount = 0;
            foreach (Row?[] rows in AlignTrainCorpus(sourceTrainingCorpora, targetTrainingCorpora))

These changes are from merging in from main. After you rebase, these changes should disappear.

johnml1135 · 2024-12-03T01:47:36Z

src/Serval/src/Serval.Translation/Services/TranslationPlatformServiceV1.cs line 287 at r3 (raw file):

Previously, mudiagaobrikisil wrote…

Noted

I told Mudi to get the plain implementation in now and I (or he) could work on updating the DataAccess layer after the other changes are in.

johnml1135 · 2024-12-03T01:48:17Z

src/Serval/test/Serval.E2ETests/ServalApiTests.cs line 142 at r1 (raw file):

Previously, mudiagaobrikisil wrote…

Done

Please verify the counts in the execution data as well.

johnml1135 · 2024-12-03T01:48:45Z

src/Machine/src/Serval.Machine.Shared/Models/Build.cs line 32 at r1 (raw file):

Previously, mudiagaobrikisil wrote…

Done

Where did the execution data go?

johnml1135 · 2024-12-03T01:49:12Z

src/Serval/src/Serval.Translation/Services/TranslationPlatformServiceV1.cs line 279 at r3 (raw file):

Previously, mudiagaobrikisil wrote…

E2E test done but need a little help with knowing how to track the init and final values in the unit tests, since the ExecutionData is initialized to an empty dictionary

Ok - we can talk later.

johnml1135

Reviewed 9 of 13 files at r4, 4 of 4 files at r5, all commit messages.
Reviewable status: all files reviewed, 8 unresolved discussions (waiting on @ddaspit and @mudiagaobrikisil)

mudiagaobrikisil

Reviewable status: all files reviewed, 8 unresolved discussions (waiting on @ddaspit and @johnml1135)

-- commits line 107 at r5:

Previously, johnml1135 (John Lambert) wrote…

Please rebase these into a single commit on the latest main branch commit. I can show you how to do that.

Okay. Will do that

src/Machine/src/Serval.Machine.Shared/Services/PreprocessBuildJob.cs line 192 at r3 (raw file):

Previously, johnml1135 (John Lambert) wrote…

These changes are from merging in from main. After you rebase, these changes should disappear.

Noted

src/Serval/src/Serval.Translation/Services/TranslationPlatformServiceV1.cs line 279 at r3 (raw file):

Previously, johnml1135 (John Lambert) wrote…

Ok - we can talk later.

Okay

src/Serval/test/Serval.E2ETests/ServalApiTests.cs line 142 at r1 (raw file):

Previously, johnml1135 (John Lambert) wrote…

Please verify the counts in the execution data as well.

Done

src/Machine/src/Serval.Machine.Shared/Models/Build.cs line 32 at r1 (raw file):

Previously, johnml1135 (John Lambert) wrote…

Where did the execution data go?

Done

mudiagaobrikisil added 13 commits October 18, 2024 16:28

Added serval release version to translation build

937c563

Merge branch 'main' into add_serval_release_version

7b1172a

Made requested changes

edf45ea

Merge remote-tracking branch 'origin/add_serval_release_version' into…

346e116

… add_serval_release_version

Fixed constructor issue

00a6d3f

Used deployment version

2cc125b

Checking if test will pass

f8485d3

Modification to use deployment version properly

3ca47ac

Made edits to the PR

678f03a

Removed commented code

56bd93f

refactored code to reflect suggestions

80682ba

Working on GRPC

de7f91e

Added summary

a6fff2f

mudiagaobrikisil closed this Nov 18, 2024

mudiagaobrikisil reopened this Nov 18, 2024

Merge branch 'main' into build_summary_gprc

2b91842

johnml1135 reviewed Nov 19, 2024

View reviewed changes

johnml1135 requested changes Nov 25, 2024

View reviewed changes

Initialized statistics to an empty array and made resulting changes

0812665

mudiagaobrikisil force-pushed the build_summary_gprc branch from 2b91842 to 0812665 Compare November 26, 2024 13:38

Merge branch 'main' into build_summary_gprc

40084eb

mudiagaobrikisil commented Nov 26, 2024

View reviewed changes

johnml1135 reviewed Nov 26, 2024

View reviewed changes

ddaspit requested changes Nov 26, 2024

View reviewed changes

mudiagaobrikisil commented Nov 27, 2024

View reviewed changes

johnml1135 reviewed Nov 27, 2024

View reviewed changes

mudiagaobrikisil commented Nov 28, 2024

View reviewed changes

Updated PR and changed Statistics to ExecutionData

3cb8e1f

mudiagaobrikisil force-pushed the build_summary_gprc branch from 40084eb to 3cb8e1f Compare November 28, 2024 23:08

Merge branch 'main' into build_summary_gprc

2d79791

Enkidu93 requested changes Dec 2, 2024

View reviewed changes

mudiagaobrikisil added 2 commits December 2, 2024 20:38

removed camel case in proto file

3dcedf4

Merge remote-tracking branch 'origin/build_summary_gprc' into build_s…

9e18ec5

…ummary_gprc

mudiagaobrikisil commented Dec 2, 2024

View reviewed changes

johnml1135 reviewed Dec 3, 2024

View reviewed changes

mudiagaobrikisil commented Dec 3, 2024

View reviewed changes

Build summary gprc #541

Are you sure you want to change the base?

Build summary gprc #541

Conversation

mudiagaobrikisil commented Nov 18, 2024 • edited by ddaspit Loading

codecov-commenter commented Nov 18, 2024 • edited Loading

Codecov Report

johnml1135 commented Nov 19, 2024

johnml1135 commented Nov 19, 2024

johnml1135 commented Nov 19, 2024

johnml1135 commented Nov 19, 2024

johnml1135 commented Nov 19, 2024

johnml1135 commented Nov 19, 2024

johnml1135 commented Nov 19, 2024

johnml1135 commented Nov 19, 2024

johnml1135 commented Nov 19, 2024

johnml1135 commented Nov 19, 2024

johnml1135 left a comment

Choose a reason for hiding this comment

johnml1135 left a comment

Choose a reason for hiding this comment

mudiagaobrikisil commented Nov 26, 2024 via email

johnml1135 commented Nov 26, 2024

mudiagaobrikisil left a comment

Choose a reason for hiding this comment

johnml1135 commented Nov 26, 2024

johnml1135 commented Nov 26, 2024

johnml1135 commented Nov 26, 2024

johnml1135 left a comment

Choose a reason for hiding this comment

ddaspit left a comment

Choose a reason for hiding this comment

mudiagaobrikisil left a comment

Choose a reason for hiding this comment

johnml1135 commented Nov 27, 2024

johnml1135 commented Nov 27, 2024

johnml1135 left a comment

Choose a reason for hiding this comment

mudiagaobrikisil left a comment

Choose a reason for hiding this comment

Enkidu93 left a comment

Choose a reason for hiding this comment

mudiagaobrikisil left a comment

Choose a reason for hiding this comment

johnml1135 commented Dec 3, 2024

johnml1135 commented Dec 3, 2024

johnml1135 commented Dec 3, 2024

johnml1135 commented Dec 3, 2024

johnml1135 commented Dec 3, 2024

johnml1135 commented Dec 3, 2024

johnml1135 left a comment

Choose a reason for hiding this comment

mudiagaobrikisil left a comment

Choose a reason for hiding this comment

mudiagaobrikisil commented Nov 18, 2024 •

edited by ddaspit

Loading

codecov-commenter commented Nov 18, 2024 •

edited

Loading