Optimize Labdoo's cron job

×

Statusmeldung

You are not a member of this team. If you want to be part of this team, click on 'Subscribe to this team'.
Typ: 
General task
Status: 
Open
Priorität: 
Normal
Beschreibung: 

This task is about looking into the Labdoo cron job and seeing if we can optimize both its execution time and the time it takes to load a first page after the cron job has run.

Kommentare

Bild des Benutzers jordi
Gespeichert von jordi am So, 05/19/2019 - 11:35

Time that it takes to load the main page before and after cron:
ubuntu@ip-172-31-154-115:/tmp$ time wget www.labdoo.org
--2019-05-19 10:58:39-- http://www.labdoo.org/
Resolving www.labdoo.org (www.labdoo.org)... 18.185.13.101
Connecting to www.labdoo.org (www.labdoo.org)|18.185.13.101|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.labdoo.org/ [following]
--2019-05-19 10:58:39-- https://www.labdoo.org/
Connecting to www.labdoo.org (www.labdoo.org)|18.185.13.101|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html.6’
real 0m1.160s
user 0m0.000s
sys 0m0.008s
ubuntu@ip-172-31-154-115:/tmp$ time wget www.labdoo.org
--2019-05-19 10:58:51-- http://www.labdoo.org/
Resolving www.labdoo.org (www.labdoo.org)... 18.185.13.101
Connecting to www.labdoo.org (www.labdoo.org)|18.185.13.101|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.labdoo.org/ [following]
--2019-05-19 10:58:51-- https://www.labdoo.org/
Connecting to www.labdoo.org (www.labdoo.org)|18.185.13.101|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html.7’
real 0m6.963s
user 0m0.000s
sys 0m0.008s

Time that it takes to load the dootronics dashboard before and after cron:
ubuntu@ip-172-31-154-115:/tmp$ time wget https://www.labdoo.org/content/dootronics-dashboard
--2019-05-19 10:52:37-- https://www.labdoo.org/content/dootronics-dashboard
Resolving www.labdoo.org (www.labdoo.org)... 18.185.13.101
Connecting to www.labdoo.org (www.labdoo.org)|18.185.13.101|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘dootronics-dashboard.1’
2019-05-19 10:52:43 (9.42 MB/s) - ‘dootronics-dashboard.1’ saved [16673289]
real 0m5.226s
user 0m0.000s
sys 0m0.148s
ubuntu@ip-172-31-154-115:/tmp$ time wget https://www.labdoo.org/content/dootronics-dashboard
--2019-05-19 10:52:52-- https://www.labdoo.org/content/dootronics-dashboard
Resolving www.labdoo.org (www.labdoo.org)... 18.185.13.101
Connecting to www.labdoo.org (www.labdoo.org)|18.185.13.101|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘dootronics-dashboard.2’
2019-05-19 10:53:03 (9.41 MB/s) - ‘dootronics-dashboard.2’ saved [16673289]
real 0m10.532s
user 0m0.119s
sys 0m0.000s

Time that it takes to run the cron job at around XX:45 minutes past the hour (this is almost close to the top of the hour when the actual cron job is run):
== # time drush @lbd cron
Cron run successful. success
real 0m16.779s
user 0m4.336s
sys 0m0.298s
(lbd)root@www:/var/www/lbd/profiles/labdoo/modules/custom(feature/fixEdoovillageID)
== # time drush @lbd cron
Cron run successful. success
real 0m2.799s
user 0m1.235s
sys 0m0.098s

So the above shows that after running cron, the wait time to load the first page is actually fairly small. In the past it used to be 60 seconds, but now it is just 6.9 seconds for the main page and 10 seconds for the most heavy dashboard. Please notice also that because Labdoo is continuously being hit by bots, it is most likely that the first loaded page is triggered by a bot, rather than a human. Thus the first page loaded by a human is likely to be at the normal fast speed. Further, notice that running cron itself is just 16.7 seconds (and an immediate follow up run just takes 2.7s). So st this point i am not sure Cron poses a performance issue to Labdoo, although i would recommend to continue monitoring the system.
Additional information
Here is the list of modules that execute tasks within Labdoo's cron job:
==> # find /var/www/lbd/ -type f -regex ".*\.\(module\|inc\)" -print | xargs grep "function.*_cron"
/var/www/lbd/modules/poll/poll.module:function poll_cron() {
/var/www/lbd/modules/field/field.module:function field_cron() {
/var/www/lbd/modules/node/node.module:function node_cron() {
/var/www/lbd/modules/simpletest/tests/common_test_cron_helper.module:function common_test_cron_helper_cron() {
/var/www/lbd/modules/simpletest/tests/common_test.module:function common_test_cron() {
/var/www/lbd/modules/system/tests/cron_queue_test.module:function cron_queue_test_cron_queue_info() {
/var/www/lbd/modules/system/tests/system_cron_test.module:function system_cron_test_flush_caches() {
/var/www/lbd/modules/system/system.admin.inc:function system_cron_settings() {
/var/www/lbd/modules/system/system.admin.inc:function system_run_cron_submit($form, &$form_state) {
/var/www/lbd/modules/system/system.admin.inc:function system_run_cron() {
/var/www/lbd/modules/system/system.module:function system_cron() {
/var/www/lbd/modules/system/system.module:function system_run_automated_cron() {
/var/www/lbd/modules/dblog/dblog.module:function dblog_cron() {
/var/www/lbd/modules/search/search.module:function search_cron() {
/var/www/lbd/modules/update/update.module:function update_cron() {
/var/www/lbd/modules/update/update.fetch.inc:function _update_cron_notify() {
/var/www/lbd/modules/openid/openid.module:function openid_cron() {
/var/www/lbd/modules/trigger/trigger.module:function trigger_cron() {
/var/www/lbd/modules/trigger/tests/trigger_test.module:function trigger_test_system_cron_action() {
/var/www/lbd/modules/trigger/tests/trigger_test.module:function trigger_test_system_cron_conf_action($object, $context) {
/var/www/lbd/modules/trigger/tests/trigger_test.module:function trigger_test_system_cron_conf_action_form($context) {
/var/www/lbd/modules/trigger/tests/trigger_test.module:function trigger_test_system_cron_conf_action_submit($form, $form_state) {
/var/www/lbd/modules/statistics/statistics.module:function statistics_cron() {
/var/www/lbd/modules/aggregator/aggregator.module:function aggregator_cron() {
/var/www/lbd/modules/aggregator/aggregator.module:function aggregator_cron_queue_info() {
/var/www/lbd/modules/tracker/tracker.module:function tracker_cron() {
/var/www/lbd/includes/common.inc:function drupal_cron_run() {
/var/www/lbd/includes/common.inc:function drupal_cron_cleanup() {
/var/www/lbd/profiles/labdoo/modules/custom/labdoo_lib/labdoo_lib.module:function labdoo_lib_cron() {
/var/www/lbd/profiles/labdoo/modules/custom/lbd_communicate/lbd_communicate.module:function lbd_communicate_cron() {
/var/www/lbd/profiles/labdoo/modules/contrib/honeypot/honeypot.module:function honeypot_cron() {
/var/www/lbd/profiles/labdoo/modules/contrib/location/contrib/location_autofill/la.module:function la_cron() {
/var/www/lbd/profiles/labdoo/modules/contrib/captcha/captcha.module:function captcha_cron() {
/var/www/lbd/profiles/labdoo/modules/contrib/boost/boost.module:function boost_cron() {
/var/www/lbd/profiles/labdoo/modules/contrib/boost/boost_crawler/boost_crawler.module:function boost_crawler_cron_queue_info() {
/var/www/lbd/profiles/labdoo/modules/contrib/smtp/smtp.module:function smtp_cron_queue_info() {
/var/www/lbd/profiles/labdoo/modules/contrib/l10n_update/l10n_update.module:function l10n_update_cron() {
/var/www/lbd/profiles/labdoo/modules/contrib/logintoboggan/logintoboggan.module:function logintoboggan_cron() {
/var/www/lbd/profiles/labdoo/modules/contrib/file_entity/file_entity.module:function file_entity_cron_queue_info() {
/var/www/lbd/profiles/labdoo/modules/contrib/ctools/includes/object-cache.cron.inc:function ctools_object_cache_cron() {
/var/www/lbd/profiles/labdoo/modules/contrib/ctools/ctools.module:function ctools_cron() {
/var/www/lbd/profiles/labdoo/modules/contrib/drupalchat/drupalchat.module:function drupalchat_cron() {
/var/www/lbd/profiles/labdoo/modules/contrib/rules/modules/events.inc:function rules_cron() {
/var/www/lbd/profiles/labdoo/modules/contrib/rules/rules_scheduler/rules_scheduler.module:function rules_scheduler_cron() {
/var/www/lbd/profiles/labdoo/modules/contrib/rules/rules_scheduler/rules_scheduler.module:function rules_scheduler_cron_queue_info() {
/var/www/lbd/profiles/labdoo/modules/contrib/mass_contact/mass_contact.module:function mass_contact_cron() {
/var/www/lbd/profiles/labdoo/modules/contrib/oauth2_server/oauth2_server.module:function oauth2_server_cron() {
/var/www/lbd/profiles/labdoo/modules/contrib/og/og.module:function og_cron_queue_info() {
/var/www/lbd/profiles/labdoo/modules/contrib/simplenews/simplenews.module:function simplenews_cron() {
/var/www/lbd/profiles/labdoo/modules/contrib/simplenews/simplenews.module:function simplenews_cronapi($op) {
/var/www/lbd/profiles/labdoo/modules/contrib/node_export/modules/node_export_dependency/node_export_dependency.module:function node_export_dependency_cron() {
/var/www/lbd/profiles/labdoo/modules/contrib/user_restrictions/user_restrictions.module:function user_restrictions_cron() {
/var/www/lbd/profiles/labdoo/modules/contrib/google_analytics/googleanalytics.module:function googleanalytics_cron() {
/var/www/lbd/sites/all/modules/spambot/spambot.module:function spambot_cron() {
/var/www/lbd/sites/all/modules/backup_migrate/backup_migrate.module:function backup_migrate_cron() {
/var/www/lbd/sites/all/modules/backup_migrate/backup_migrate.module:function backup_migrate_cronapi($op, $job = NULL) {
/var/www/lbd/sites/all/modules/backup_migrate/includes/schedules.inc:function backup_migrate_schedules_cron() {
/var/www/lbd/sites/all/modules/linkchecker/linkchecker.module:function linkchecker_cron() {
/var/www/lbd/sites/all/modules/registration/registration.module:function registration_cron() {
/var/www/lbd/sites/all/modules/views_bulk_operations/views_bulk_operations.module:function views_bulk_operations_cron() {
/var/www/lbd/sites/all/modules/views_bulk_operations/views_bulk_operations.module:function views_bulk_operations_cron_queue_info() {