Using periodic tasks (cronjobs)

in Usage Tags: SSH

Cronjobs are periodic tasks that run in the background of your Hypernode. They can be used for all sorts of maintenance, such as cleaning up logs or importing new products. In fact, Magento makes extensive use of cronjobs.

If you need some help to get the timing settings right, check corntab and cronjob time predictor, these little helpers can assist you configuring cron jobs.

Setting up cron jobs

Enabling the Magento cron

This is recommended for every installation! Log in using SSH and run crontab -e and add the following lines:

# Run the Magento cron every 5 minutes, skip if previous run is still
# busy, and mail any output to my@email.com
# MAILTO=your@email.com # Uncomment to receive the error output on your own email address
*/5 * * * * flock -n ~/.cron.lock php /data/web/public/cron.php

After adding the cronjob, press CTRL+X, Y and then ENTER to save the cronjob into your crontab.

Enabling the Magento 2 cron

To configure the cron for Magento2 add the following snippet to your crontab:

# MAILTO=your@email.com # Uncomment to receive the error output on your own email address
* * * * * flock -n ~/.cron.lock php /data/web/magento2/bin/magento cron:run
* * * * * flock -n ~/.update-cron.lock php /data/web/magento2/update/cron.php
* * * * * flock -n ~/.setup-cron.lock php /data/web/magento2/bin/magento setup:cron:run

Magento 2 uses another mechanism for scheduling tasks. That’s why the Magento 2 cron needs to run every minute instead of every 5 minutes. More information can be found in the Magento 2 documentation.

Import cronjobs from a Byte account

To copy your cronjobs from your Byte Magento plan to your Hypernode account, use the following command:

ssh mydomain.nl@ssh.mydomain.nl "crontab -l" | crontab -

Make sure to adjust any absolute paths, and replace lockrun with flock, otherwise your cron will not work.

Adding custom cronjobs

For most cases, cronjob syntax is actually quite easy. See the full documentation or use these examples.

# Run an hourly job at 10 minutes past the hour
10 * * * * flock -n ~/.myjob.lock php /data/web/mycron.php
# Run a daily job at 3:20 in the night
20 3 * * * flock -n ~/.myjob.lock php /data/web/mycron.php
# Run a job on the first day of the month at 3:20
20 3 1 * * flock -n ~/.myjob.lock php /data/web/mycron.php

Stopping cronjobs after a timeout period

To avoid cronjobs running for hours, blocking all other crontasks to be executed by magento, you can make use of the timeout utility.
This tool can be given a “max execution time” and will stop the running task when this time period has exeeded.

For example:

# Run a daily job at 3:20 in the night and kill after 120 seconds
20 3 * * * timeout --kill-after=120 php /data/web/mycron.php

Cron Deadlock Protection

Our system will automatically add flock to your cron commands. This will prevent many problems, for example when multiple concurrent imports cause a database deadlock. If you do not want our auto flock (not recommended!) you can add # noflock to your command, like this:

# Run a command that does not care whether it is already running
* * * * * php /data/web/mycron.php # noflock

Changing your crontab editor

When you use crontab -e for the first time, you’ll be asked which editor you want to use. If you’d like to change your editor after your initial choice, please use the select-editor command. If you want to temporary change your editor, you can do this by exporting the EDITOR variable: export EDITOR=vim; crontab -e

More info can be found in our article about configuring your editor

Recommendations

  • Idem-potency: When you write your own cron script, make sure the script is idempotent. When a script should only be run after a certain time or when a new import file is present, always make sure to exit the script when the conditions are not met. This avoids things breaking down when people run cron scripts manually to debug cron issues.

  • Time is always based on UTC (= GMT). This might differ from your local time: Dutch time is UTC+1 in winter and UTC+2 in summer. Or use this handy converter)

  • Do not run the Magento crons using wget or curl, as this is slow, might give partial results, and occupies webslots for real visitors). See above how to invoke the non-HTTP Magento cron.

  • Do not run the same cron simultaneously on multiple hosts, as results are unpredictable. When migrating, do not forget to disable the old crons.

  • Be carefull when using day of month and day of week together. Using Dom and Dow without wildcards makes this an OR condition and not an AND, running the job more often then you’d expect.

  • Cron does not deal with seconds so you can’t have cron job’s going off in any time period dealing with seconds. Like a cronjob going off every 30 seconds.

  • Monitor your cronjobs! You can do this by checking the exitcode and or the mail that is sent, or using a third party monitor. A free third party cron monitoring tool is healthchecks.io

Debugging cron issues

For new users, cron can be difficult to work with, as it’s hard finding out wat is causing cron not to run or to end before finishing the job. This section gives you some tips and tricks to debug cron issues.

Test if the cron’s timing is actually what you are expecting

If the cron is configured, but appears not to be running, check if the time configuration is at what you’d expect.
A great helper utility to validate the cron’s timing, is crontab.guru.

Test if the cron is currently running

If it appears your cron is not running anymore, check whether it is currently running using flock.
Sometimes a cronjob takes more time than expected or ends up running in a recursive loop.
If this happens, the lock never gets removed causing the cron daemon to delay the consecutive jobs.
To resolve this issue, find out why it’s still running and kill the current running cronjob. Then start a new one.

You can use the ps faux command to check the running processes. If you see a cronjob still running, you can kill it by using its PID number:

ps faux

You can kill the process (with PID 2) by using this command:
kill -9 2.

Check if the cron daemon is running

We monitor cron extensively, but to ensure yourself, grep in the process list to verify whether the daemon is really running:

ps -ef | grep -i cron

Verify the cron command from your logs

When a cronjob ran, a log entry is written to /var/log/syslog. Retrieve the cron command from the logs and check whether the command can be run on the commandline to avoid typo’s or badly copied example cronjobs.

Check if all cronjobs use their own lockfile for flock or share one

If multiple cronjobs share the same lockfile and one job is running, the other can’t start.
To avoid this, ensure that all crons use a unique lockfile.

Test if the lock files are cleaned up

Sometimes when a cronjob crashes, the locks are not cleaned up successfully. This will obstruct the cronjob from running.
To fix, cleanup the old lockfiles by removing them:

rm ~/.a6fe0d58.lock

This will remove the file, effectively removing the lock set on the file as well.

Test if the cron has ever run since you configured it

To get a history of all cron actions, like editing, updating and listing the cron files, all jobs of all users etc, you can easily grep for cron in syslog:

grep -i cron /var/log/syslog

This will generate a huge list of all jobs cron has run including editting, listing and removing the cronjob.
To verify whether it has run, grep for all cronjobs that are run by user app and check if your cronjob has ever run:

grep -i cron /var/log/syslog | grep '(app)'

Verify whether your PATH variable and other environment variables are set correctly.

Cron does not automatically source environment and or shell settings from .bashrc, .profile etc.
You can’t use predefined shell environment settings like $PATH, $HOME, $USER or homedir expansion like ~/sbin.
If you want to use these settings you’d have to source the corresponding settings file accordingly by adding it to your cronjob:

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

Or alternatively if you want all the settings you use in the shell, source your bashrc file prior to executing:

*/5 * * * * source /data/web/.bashrc && cp $HOME/somefile $HOME/public/exports

You can set things like MAILTO=, SHELL= or PATH= and other environment variables the shell uses.

For example:

IMPORTFILE="/data/web/public/some-very-long-import-file-in-cvs-format.products-from-some-website.cvs"
DESTINATION="/data/web/public/some/very/long/path/in/the/magento/directory`
*/5 * * * * cp $IMPORTFILE $DESTINATION/

Verify whether your locations are setup correctly

Verify the correct path in cron arguments.

Cronjobs often contain full paths to scripts inside your homedirectory. Be sure to replace them for your situation at Hypernode. For example, if your site is currently hosted at Byte, you need to replace /home/users/domaiftp with /data/web/, and /home/users/domaiftp/domain.com with /data/web/public.

Verify whether your cron is working when you run it on the commandline

If your cronjob appears to be started by cron, but does not finish the job it is supposed to handle, check what output is generated by running the cron on the commandline.

Ensure yourself you are not using special characters like % and @ in your cron command

Cron uses some special characters in it’s syntax. Therefor you cannot use chars: *, ,, -, @, # and %
If used with command substitution (IE: the date command) you can use shell command substitution ($()).
To use special chars, escape them with a backslash: \% instead of %

Verify the cron output

Many users will add 2> /dev/null or 2>&1 > /dev/null to their cronjobs, redirecting all error output to /dev/null.
If you use this option, you will never see the error output the job creates.

To check whether a job generates error output, redirect output to standard out and let cron mail it’s output by setting a MAILTO variable containing your email address. This will send all output of the cronjob to your email address.
If you only want to receive a mail when the jobs fails, use chronic. This lovely tool only returns output when the exit code is not 0, therefor only sending the output to cron when the job did not succeed.

Catching error output in a log file

If you want to debug cron timing issues, use a wrapper that logs all issues to log files:

Save it as ~/bin/debug-cron and make it executable with chmod +x ~/bin/debug-cron
Now you can debug your cron issues using the following syntax:

*/5 * * * * flock -n ~/.a6fe0d58.lock -c '/data/web/bin/debug-cron php -f /data/web/public/cron.php'

To check the output and run/stop time of the cronjob, check the logfiles in /data/web/public/var/log/crons

2