Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Avoid stalling LOD syncs during immediate enqueues and discovery hooks #11489

Merged
merged 4 commits into from
Nov 3, 2023
Merged

Avoid stalling LOD syncs during immediate enqueues and discovery hooks #11489

merged 4 commits into from
Nov 3, 2023

Conversation

bjester
Copy link
Member

@bjester bjester commented Nov 2, 2023 •
edited
Loading

Summary

  • A python datetime.timedelta can be falsy if the time delta is 0, so this ensures we reschedule the LOD syncing tasks when they should be scheduled immediately (in 0 seconds)
  • Adds logic to prevent network discovery hooks from interfering with ongoing sync queue scheduling
  • Adds tests

References

Resolves #11487

Reviewer guidance

  1. Have a LOD sync with a server
  2. Disconnect the LOD from the network or stop its Kolibri for at least 1 minute
  3. Reconnect the LOD and verify it syncs upon connecting

Testing checklist

  • Contributor has fully tested the PR manually
  • If there are any front-end changes, before/after screenshots are included
  • Critical user journeys are covered by Gherkin stories
  • Critical and brittle code paths are covered by unit tests

PR process

  • PR has the correct target branch and milestone
  • PR has 'needs review' or 'work-in-progress' label
  • If PR is ready for review, a reviewer has been added. (Don't use 'Assignees')
  • If this is an important user-facing change, PR or related issue has a 'changelog' label
  • If this includes an internal dependency change, a link to the diff is provided

Reviewer checklist

  • Automated test coverage is satisfactory
  • PR is fully functional
  • PR has been tested for accessibility regressions
  • External dependency files were updated if necessary (yarn and pip)
  • Documentation is updated
  • Contributor is in AUTHORS.md

@github-actions github-actions bot added DEV: backend Python, databases, networking, filesystem... SIZE: very small labels Nov 2, 2023
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 2, 2023 •
edited
Loading

Build Artifacts

Asset type Download link
PEX file kolibri-0.16.0b7.dev0_git.5.gf9d630c0.pex
Windows Installer (EXE) kolibri-0.16.0b7.dev0+git.5.gf9d630c0-unsigned.exe
Debian Package kolibri_0.16.0b7.dev0+git.5.gf9d630c0-0ubuntu1_all.deb
Mac Installer (DMG) kolibri-0.16.0b7.dev0+git.5.gf9d630c0-0.3.0.dmg
Android Package (APK) kolibri-0.16.0b7.dev0+git.5.gf9d630c0-0.1.0-debug.apk
TAR file kolibri-0.16.0b7.dev0+git.5.gf9d630c0.tar.gz
WHL file kolibri-0.16.0b7.dev0+git.5.gf9d630c0-py2.py3-none-any.whl

@bjester bjester changed the title [WIP] Avoid boolean comparisons with timedelta Avoid boolean comparisons with timedelta Nov 3, 2023
@bjester bjester added the TODO: needs review Waiting for review label Nov 3, 2023
@bjester bjester marked this pull request as ready for review November 3, 2023 14:59
jredrejo
jredrejo approved these changes Nov 3, 2023
View reviewed changes
Copy link
Member

@jredrejo jredrejo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

simple and clear, thanks!

@pcenov
Copy link
Member

pcenov commented Nov 3, 2023

Hi @bjester,
Unfortunately twice the build installed on my Android phone got into a state of not syncing which cannot be restarted after both restarting the server and relaunching the app - not sure if these are caused by changes made here or other issues with the app so here are the logs for you to investigate:

AndroidLogs2.zip
AndroidLogs1.zip

Server logs:

ServerLogs.zip

Logs of a LOD installed on Ubuntu VM that was syncing correctly throughout the entire time:

UbuntuLearnerLogs.zip

@pcenov pcenov mentioned this pull request Nov 3, 2023
Closed
@rtibbles
Copy link
Member

rtibbles commented Nov 3, 2023 •
edited
Loading

The last successful run of the sync queue processing job in Android 1 seemed to coincide with the startup of the app:

INFO 2023-11-03 09:38:00,313 __main__ Starting Kolibri task worker, for job 50
INFO 2023-11-03 09:38:00,413 kolibri.core.device.soud [user=ff9e4fd916a64b73788dd2e3d782894f] [server=6a40d55608b125e4a9e45fb854476922] SoUD sync ready
INFO 2023-11-03 09:38:00,476 kolibri.core.discovery.utils.network.client Attempting connections to variations of the URL: http://192.168.100.254:8080/
INFO 2023-11-03 09:38:00,478 kolibri.core.discovery.utils.network.client Attempting connection to: http://192.168.100.254:8080/
INFO 2023-11-03 09:38:00,498 kolibri.core.discovery.utils.network.client Success! We connected to: http://192.168.100.254:8080/api/public/info/?v=3
INFO 2023-11-03 09:38:00,982 kolibri.core.auth.management.utils Remotely preparing data
INFO 2023-11-03 09:38:02,779 kolibri.utils.options Option CHERRYPY_THREAD_POOL in section [Server] being overridden by environment variable KOLIBRI_CHERRYPY_THREAD_POOL
INFO 2023-11-03 09:38:02,779 kolibri.utils.options Option DEBUG in section [Server] being overridden by environment variable KOLIBRI_DEBUG
INFO 2023-11-03 09:38:02,780 kolibri.utils.options Option DEBUG_LOG_DATABASE in section [Server] being overridden by environment variable KOLIBRI_DEBUG_LOG_DATABASE
INFO 2023-11-03 09:38:02,780 kolibri.utils.options Option RUN_MODE in section [Deployment] being overridden by environment variable KOLIBRI_RUN_MODE
INFO 2023-11-03 09:38:02,780 kolibri.utils.options Option RESTART_HOOKS in section [Deployment] being overridden by environment variable KOLIBRI_RESTART_HOOKS
INFO 2023-11-03 09:38:02,862 kolibri.core.auth.management.utils No records transferred
INFO 2023-11-03 09:38:03,557 kolibri.core.apps Running Kolibri with the following settings: kolibri_app_settings
INFO 2023-11-03 09:38:04,353 kolibri.core.content.kolibri_plugin Processing content requests for synced dataset: dadee434baef8f7ee819e715c5b9cb9f
INFO 2023-11-03 09:38:04,355 kolibri.core.content.utils.content_request Processing new content assignment requests
INFO 2023-11-03 09:38:04,363 kolibri.core.content.utils.content_request Processing new content removal requests
INFO 2023-11-03 09:38:04,371 kolibri.core.content.kolibri_plugin Completed content requests for synced dataset: dadee434baef8f7ee819e715c5b9cb9f
INFO 2023-11-03 09:38:05,033 kolibri.core.discovery.utils.network.client Unable to connect: http://10.0.2.16/api/public/info/
INFO 2023-11-03 09:38:05,037 kolibri.core.discovery.utils.network.client Attempting connection to: http://10.0.2.16:8008/
INFO 2023-11-03 09:38:05,185 __main__ Starting Kolibri task worker, for job 512a7fb299de43c6bfb4036f150a3ba1
INFO 2023-11-03 09:38:05,278 __main__ Ending Kolibri task worker, for job 512a7fb299de43c6bfb4036f150a3ba1
INFO 2023-11-03 09:38:05,364 kolibri.core.auth.management.utils Locally preparing data to send
INFO 2023-11-03 09:38:07,340 kolibri.core.auth.management.utils No records transferred
INFO 2023-11-03 09:38:07,570 kolibri.utils.server Starting Kolibri 0.16.0b7.dev0+git.2.gfd5b9531
INFO 2023-11-03 09:38:07,684 kolibri.utils.server Bus state: ENTER
INFO 2023-11-03 09:38:07,685 kolibri.utils.server Listening for SIGTERM.
INFO 2023-11-03 09:38:07,686 kolibri.utils.server Listening for SIGHUP.
INFO 2023-11-03 09:38:07,686 kolibri.utils.server Listening for SIGUSR1.
INFO 2023-11-03 09:38:07,686 kolibri.utils.server Listening for SIGINT.
INFO 2023-11-03 09:38:07,687 kolibri.utils.server Bus state: IDLE
INFO 2023-11-03 09:38:07,687 kolibri.utils.server Bus state: START
INFO 2023-11-03 09:38:07,775 kolibri.utils.server Started monitor thread 'ZeroConfPlugin'.
INFO 2023-11-03 09:38:07,775 kolibri.utils.server Started monitor thread 'ProcessControlPlugin'.
INFO 2023-11-03 09:38:07,777 kolibri.utils.server Starting on 0.0.0.0:0
INFO 2023-11-03 09:38:07,878 kolibri.utils.server Waiting for 0.0.0.0:46821
INFO 2023-11-03 09:38:07,880 kolibri.utils.server Serving on 0.0.0.0:46821
INFO 2023-11-03 09:38:07,883 kolibri.utils.server Starting on 0.0.0.0:0
INFO 2023-11-03 09:38:07,984 kolibri.utils.server Waiting for 0.0.0.0:60757
INFO 2023-11-03 09:38:07,994 kolibri.utils.server Serving on 0.0.0.0:60757
INFO 2023-11-03 09:38:08,567 kolibri.utils.server Kolibri running on: http://192.168.100.3:60757/
INFO 2023-11-03 09:38:08,568 kolibri.utils.server Kolibri running on: http://127.0.0.1:60757/
INFO 2023-11-03 09:38:08,568 kolibri.utils.server Kolibri running on: http://10.14.223.72:60757/
INFO 2023-11-03 09:38:08,568 kolibri.utils.server Bus state: RUN
INFO 2023-11-03 09:38:08,825 cherrypy.access 127.0.0.1 - - "GET /app/api/initialize/bac2169d487e448297572e602f79715b" 302 0 "" "Mozilla/5.0 (Linux; Android 13; SM-N770F Build/TP1A.220624.014; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/119.0.6045.66 Mobile Safari/537.36"
INFO 2023-11-03 09:38:08,837 kolibri.core.discovery.utils.network.broadcast Registering ourselves to zeroconf network with id 'f3d82845104e0b02e36212b0c9e7f7ce' and port '60757'
INFO 2023-11-03 09:38:09,058 cherrypy.access 127.0.0.1 - - "GET /" 302 0 "" "Mozilla/5.0 (Linux; Android 13; SM-N770F Build/TP1A.220624.014; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/119.0.6045.66 Mobile Safari/537.36"
INFO 2023-11-03 09:38:09,306 __main__ Ending Kolibri task worker, for job 50

@bjester bjester requested a review from jredrejo November 3, 2023 19:33
@bjester
Copy link
Member Author

bjester commented Nov 3, 2023

Thanks @pcenov. I've pushed more changes based off what we saw in your Android logs. Could you test again?

@bjester bjester changed the title Avoid boolean comparisons with timedelta Avoid stalling LOD syncs during immediate enqueues and discovery hooks Nov 3, 2023
@radinamatic
Copy link
Member

Jumping in for Petar here, it's looking better! 🙂

  1. Lesson1 synced to LoD (Android 13 phone), and progress about completion recorded back on the server side.
  2. Lesson2 prepared on server and synced on LoD, but without opening the resources.
  3. LoD disconnected from the shared WiFi, Lesson2 completed offline and after 5 minutes LoD reconnected to WiFi. Sync was successful!!! 🎉
LoD disconnected LoD reconnected
2023-11-03_22-07-46 2023-11-03_22-09-05
  1. Tried again, (por si las moscas, as we say in Spanish), Lesson3 prepared on server and synced on LoD, but without opening the resources.
  2. LoD disconnected from the shared WiFi, Lesson3 completed offline and after 5 minutes LoD reconnected to WiFi. Sync was successful again!!! 🎉
LoD disconnected LoD reconnected
2023-11-03_22-15-26 2023-11-03_22-16-22

@radinamatic
Copy link
Member

Attaching the logs, just in case you're curious, but approving asap anyways! Android13-user-a-logs.zip

radinamatic
radinamatic approved these changes Nov 3, 2023
View reviewed changes
Copy link
Member

@radinamatic radinamatic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LoD restarts the syncing after the disconnecting and connecting again! 🎉
Merge away! 💯 👏🏽 :shipit:

@bjester bjester merged commit 974a15e into learningequality:release-v0.16.x Nov 3, 2023
35 checks passed
@radinamatic radinamatic mentioned this pull request Nov 3, 2023
Closed
This was referenced Nov 6, 2023
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@radinamatic radinamatic radinamatic approved these changes

@rtibbles rtibbles Awaiting requested review from rtibbles

@pcenov pcenov Awaiting requested review from pcenov

@jredrejo jredrejo Awaiting requested review from jredrejo

Assignees
No one assigned
Labels
DEV: backend Python, databases, networking, filesystem... SIZE: medium SIZE: very small TODO: needs review Waiting for review
Projects
None yet
Milestone
Kolibri 0.16.0: Bugs and Regressions
Development

Successfully merging this pull request may close these issues.

Syncing Android LoDs on 0.16b7
5 participants
@bjester @pcenov @rtibbles @radinamatic @jredrejo