Feat/2965 db seed impl #3134

andrew-jameson · 2024-08-07T20:54:45Z

Summary of Changes

Pull request closes #2965

How to Test

task drop-db
task init-backend
task backend-exec-seed-db

Open http://localhost:3000/ and sign in.
Proceed to Django Admin Console and ensure hundreds of datafiles associated to various STTs exist.
Ensure a postegres dump file is in TANF-app/tdrs-backend/tdrs_db_seed.pg

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

We have a maintainable and extensible toolset

Seeding a database is performant ( sub hour )
Create a tool/mechanism for generating "random" and internally consistent dat
Tool will "fuzz" or generate out of range values to intentionally create issues [ stretch ]
Create django command such that tool can be pointed at deployed environments [ stretch ]

lfrohlich and/or adpennington confirmed that ACs are met.

Deliverable 2: Tested Code

Are all areas of code introduced in this PR meaningfully tested?
- If this PR introduces backend code changes, are they meaningfully tested?
- If this PR introduces frontend code changes, are they meaningfully tested?
Are code coverage minimums met?
- Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
- Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

Are backend code style checks passing on CircleCI?
Are frontend code style checks passing on CircleCI?
Are code maintainability principles being followed?

Deliverable 4: Accessible

Does this PR complete the epic?
Are links included to any other gov-approved PRs associated with epic?
Does PR include documentation for Raft's a11y review?
Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

Does this PR provide background for why coding decisions were made?
If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
If this PR introduces dependencies, are their licenses documented?
Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

Does the OWASP Scan pass on CircleCI?
Do manual code review and manual testing detect any new security issues?
If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

the purpose of the research
methods used to conduct the research
who participated in the research
what was tested and how
impact of research on TDP
(if applicable) final design mockups produced for TDP development

…hing with files

andrew-jameson · 2024-09-03T13:40:04Z

tdrs-backend/tdpservice/parsers/management/commands/seed_db.py

+    def handle(self, *args, **options):
+        """Populate datafiles, records, summaries, and errors for all STTs."""
+
+        for stt in STT.objects.all():  # filter(id__in=range(1,2)):


I think all might be excessive here, might take in an argument that utilizes the range in the comment to have a more limited subset of STTs.

tdrs-backend/tdpservice/parsers/management/commands/seed_db.py

andrew-jameson · 2024-09-03T13:42:44Z

tdrs-backend/tdpservice/parsers/management/commands/seed_db.py

+# https://faker.readthedocs.io/en/stable/providers/baseprovider.html#faker.providers.BaseProvider
+""" class FieldFaker(faker.providers.BaseProvider):..."""


Should probably delete this, but wanted to leave myself this breadcrumb.

elipe17 · 2024-09-09T13:02:48Z

tdrs-backend/tdpservice/parsers/management/commands/seed_db.py

+
+        # iterate over models and generate lines
+        for _, model in models_in_section.items():
+            if long_section in ['Active Case Data', 'Closed Case Data', 'Aggregate Data', 'Stratum Data']:


Is SSP covered with this too?

No, it definitely wouldn't. I'll get that fixed up

elipe17

This PR led me to find a very small but impactful error in another area of our parser code that causes this db seed implementation to generate zero valid records when the fix for the error is added. It's a bit verbose to explain here so I will bring it up today on OH and we can document it that way.

codecov · 2024-09-09T14:27:48Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.66%. Comparing base (c15cb7c) to head (0bece5e).
Report is 2 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #3134   +/-   ##
========================================
  Coverage    92.66%   92.66%           
========================================
  Files           47       47           
  Lines         1009     1009           
  Branches       169      169           
========================================
  Hits           935      935           
  Misses          42       42           
  Partials        32       32

Flag	Coverage Δ
dev-frontend	`92.66% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 30513b6...0bece5e. Read the comment docs.

elipe17 · 2024-09-09T18:19:09Z

2024-09-09 18:12:27,335 ERROR case_consistency_validator.py::validate:L159 :  Uncaught exception during category four validation.
Traceback (most recent call last):
  File "/tdpapp/tdpservice/parsers/case_consistency_validator.py", line 156, in validate
    num_errors = self.__validate()
  File "/tdpapp/tdpservice/parsers/case_consistency_validator.py", line 176, in __validate
    num_errors += self.__validate_section2(num_errors)
  File "/tdpapp/tdpservice/parsers/case_consistency_validator.py", line 192, in __validate_section2
    num_errors += self.__validate_t5_atd_and_ssi()
  File "/tdpapp/tdpservice/parsers/case_consistency_validator.py", line 484, in __validate_t5_atd_and_ssi
    dob_date = datetime.strptime(dob, '%Y%m%d')
  File "/usr/local/lib/python3.10/_strptime.py", line 568, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "/usr/local/lib/python3.10/_strptime.py", line 352, in _strptime
    raise ValueError("unconverted data remains: %s" %
ValueError: unconverted data remains: 08

andrew-jameson · 2024-09-23T19:49:22Z

2024-09-09 18:12:27,335 ERROR case_consistency_validator.py::validate:L159 :  Uncaught exception during category four validation.
Traceback (most recent call last):
  File "/tdpapp/tdpservice/parsers/case_consistency_validator.py", line 156, in validate
    num_errors = self.__validate()
  File "/tdpapp/tdpservice/parsers/case_consistency_validator.py", line 176, in __validate
    num_errors += self.__validate_section2(num_errors)
  File "/tdpapp/tdpservice/parsers/case_consistency_validator.py", line 192, in __validate_section2
    num_errors += self.__validate_t5_atd_and_ssi()
  File "/tdpapp/tdpservice/parsers/case_consistency_validator.py", line 484, in __validate_t5_atd_and_ssi
    dob_date = datetime.strptime(dob, '%Y%m%d')
  File "/usr/local/lib/python3.10/_strptime.py", line 568, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "/usr/local/lib/python3.10/_strptime.py", line 352, in _strptime
    raise ValueError("unconverted data remains: %s" %
ValueError: unconverted data remains: 08

@elipe17 Are you fine resolving this in follow on #3177 ?

elipe17 · 2024-09-30T14:00:24Z

2024-09-09 18:12:27,335 ERROR case_consistency_validator.py::validate:L159 :  Uncaught exception during category four validation.
Traceback (most recent call last):
  File "/tdpapp/tdpservice/parsers/case_consistency_validator.py", line 156, in validate
    num_errors = self.__validate()
  File "/tdpapp/tdpservice/parsers/case_consistency_validator.py", line 176, in __validate
    num_errors += self.__validate_section2(num_errors)
  File "/tdpapp/tdpservice/parsers/case_consistency_validator.py", line 192, in __validate_section2
    num_errors += self.__validate_t5_atd_and_ssi()
  File "/tdpapp/tdpservice/parsers/case_consistency_validator.py", line 484, in __validate_t5_atd_and_ssi
    dob_date = datetime.strptime(dob, '%Y%m%d')
  File "/usr/local/lib/python3.10/_strptime.py", line 568, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "/usr/local/lib/python3.10/_strptime.py", line 352, in _strptime
    raise ValueError("unconverted data remains: %s" %
ValueError: unconverted data remains: 08

@elipe17 Are you fine resolving this in follow on #3177 ?

@andrew-jameson Ya I think it makes sense to discuss/fix the cat4 bug in a separate ticket and leave the case consistent DB seed work as the ticket you linked.

andrew-jameson added 4 commits August 5, 2024 15:58

mgmt cmd structure

df9e009

Pushing after pair w/ Jan

c333266

Pushing latest prior to lunch

6d23208

ignore for research/grep results

fbdaece

andrew-jameson added the WIP label Aug 7, 2024

andrew-jameson self-assigned this Aug 7, 2024

andrew-jameson added 6 commits August 7, 2024 17:13

End of day commit, successful run w/o syntax issues. Need to do somet…

c3283c3

…hing with files

just the useful changes, sorry for the mess

03bab8e

latest work for RPY to get around preparser blocker

8636858

Latest changes, removing comments/prints mostly.

cb204ab

functional again after a header rework

e8db2f6

Cleaning up comments/prints for PR

d6cf1e1

andrew-jameson added backend devops DX Developer Experience and removed WIP labels Sep 3, 2024

andrew-jameson requested review from raftmsohani, jtimpe and elipe17 September 3, 2024 13:28

andrew-jameson added the raft review This issue is ready for raft review label Sep 3, 2024

andrew-jameson marked this pull request as ready for review September 3, 2024 13:29

andrew-jameson changed the title ~~wip - Feat/2965 db seed impl~~ Feat/2965 db seed impl Sep 3, 2024

Merge branch 'develop' into feat/2965-db-seed-impl

2cf08b9

andrew-jameson commented Sep 3, 2024

View reviewed changes

tdrs-backend/tdpservice/parsers/management/commands/seed_db.py Outdated Show resolved Hide resolved

andrew-jameson commented Sep 3, 2024

View reviewed changes

andrew-jameson added 2 commits September 3, 2024 11:01

Linter clean-up

c10f796

linter pt2

7b44e63

elipe17 reviewed Sep 9, 2024

View reviewed changes

elipe17 requested changes Sep 9, 2024

View reviewed changes

Improvements to header, linting, and Eric's PR feedback

5fcf423

andrew-jameson mentioned this pull request Sep 9, 2024

Update db_seed with consistent case records #3177

Open

7 tasks

andrew-jameson added 2 commits September 23, 2024 12:11

Fixed quarter preparing issue

8a6f875

Comment clean up, linting fix

014fddb

raftmsohani approved these changes Sep 25, 2024

View reviewed changes

elipe17 self-requested a review September 27, 2024 18:38

elipe17 approved these changes Sep 30, 2024

View reviewed changes

Merge branch 'develop' into feat/2965-db-seed-impl

bae59cb

andrew-jameson added QASP Review and removed raft review This issue is ready for raft review labels Oct 2, 2024

andrew-jameson assigned ADPennington and unassigned ADPennington Oct 2, 2024

andrew-jameson requested a review from ADPennington October 2, 2024 14:19

ADPennington removed the QASP Review label Oct 2, 2024

Merge branch 'develop' into feat/2965-db-seed-impl

0bece5e

andrew-jameson merged commit 542823f into develop Oct 8, 2024
18 checks passed

andrew-jameson deleted the feat/2965-db-seed-impl branch October 8, 2024 13:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/2965 db seed impl #3134

Feat/2965 db seed impl #3134

andrew-jameson commented Aug 7, 2024 •

edited

Loading

andrew-jameson Sep 3, 2024

andrew-jameson Sep 3, 2024

elipe17 Sep 9, 2024

andrew-jameson Sep 9, 2024

elipe17 left a comment

codecov bot commented Sep 9, 2024 •

edited

Loading

elipe17 commented Sep 9, 2024

andrew-jameson commented Sep 23, 2024 •

edited

Loading

elipe17 commented Sep 30, 2024

		# https://faker.readthedocs.io/en/stable/providers/baseprovider.html#faker.providers.BaseProvider
		""" class FieldFaker(faker.providers.BaseProvider):..."""

Feat/2965 db seed impl #3134

Feat/2965 db seed impl #3134

Conversation

andrew-jameson commented Aug 7, 2024 • edited Loading

Summary of Changes

How to Test

Deliverables

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elipe17 left a comment

Choose a reason for hiding this comment

codecov bot commented Sep 9, 2024 • edited Loading

Codecov Report

elipe17 commented Sep 9, 2024

andrew-jameson commented Sep 23, 2024 • edited Loading

elipe17 commented Sep 30, 2024

andrew-jameson commented Aug 7, 2024 •

edited

Loading

codecov bot commented Sep 9, 2024 •

edited

Loading

andrew-jameson commented Sep 23, 2024 •

edited

Loading