diff options
Diffstat (limited to '')
| -rw-r--r-- | health/guides/beanstalk/beanstalk_number_of_tubes.md | 40 | ||||
| -rw-r--r-- | health/guides/beanstalk/beanstalk_server_buried_jobs.md | 32 | ||||
| -rw-r--r-- | health/guides/beanstalk/beanstalk_tube_buried_jobs.md | 30 |
3 files changed, 102 insertions, 0 deletions
diff --git a/health/guides/beanstalk/beanstalk_number_of_tubes.md b/health/guides/beanstalk/beanstalk_number_of_tubes.md new file mode 100644 index 00000000..8f14f07f --- /dev/null +++ b/health/guides/beanstalk/beanstalk_number_of_tubes.md @@ -0,0 +1,40 @@ +### Understand the alert + +This alert monitors the current number of tubes on a Beanstalk server. If the number of tubes drops below 5, you will receive a warning. Tubes are used as queues for jobs in Beanstalk, and having a low number of tubes may indicate an issue with service configuration or job processing. + +### What are tubes in Beanstalk? + +Beanstalk is a simple, fast work queue service that allows you to distribute tasks among different workers. In Beanstalk, *tubes* are essentially queues for jobs. Each tube stores jobs with specific priorities, Time-to-run (TTR) values, and other relevant data. Workers can reserve jobs from specific tubes, process the jobs, and delete them when finished. + +### Troubleshoot the alert + +1. Check Beanstalk server status. + + Use the following command to display the current Beanstalk server status: + + ``` + beanstalkctl stats + ``` + + Look for the current number of tubes (`current-tubes`). If it is too low (below 5), proceed to the next step. + +2. Identify recently deleted tubes. + + Determine if any tubes have been deleted recently. Check your application logs, Beanstalk daemon logs, or discuss with your development team to find out if any tube deletion is intentional. + +3. Check for misconfigurations or code issues. + + Inspect your Beanstalk server configuration and verify that the expected tubes are correctly defined. Additionally, review the application code and deployment scripts to ensure that tubes are being created and used as intended. + +4. Check worker status and processing. + + Verify that your worker processes are running and processing jobs from the tubes correctly. If there are issues with worker processes, it may lead to unused or unprocessed tubes. + +5. Create missing tubes if necessary. + + If you've identified that some tubes are missing and need to be created, add the required tubes using your application code or Beanstalk configuration. + +### Useful resources + +1. [Beanstalk Introduction](https://beanstalkd.github.io/) +2. [Beanstalk Protocol Documentation](https://raw.githubusercontent.com/beanstalkd/beanstalkd/master/doc/protocol.txt) diff --git a/health/guides/beanstalk/beanstalk_server_buried_jobs.md b/health/guides/beanstalk/beanstalk_server_buried_jobs.md new file mode 100644 index 00000000..99d4f507 --- /dev/null +++ b/health/guides/beanstalk/beanstalk_server_buried_jobs.md @@ -0,0 +1,32 @@ +### Understand the alert + +This alert is related to the `Beanstalk` message queue system and is triggered when there are buried jobs in the queue across all tubes. A buried job is one that has encountered an issue during processing by the consumer, so it remains in the queue waiting for manual action. This alert is raised in a warning state if there are more than 0 buried jobs and in a critical state if there are more than 10. + +### What are buried jobs? + +Buried jobs are tasks that have faced an error or issue during processing by the consumer, and as a result, have been `buried`. This means these jobs remain in the queue, awaiting manual intervention for them to be processed again. The presence of buried jobs does not affect the processing of new jobs in the queue. + +### Troubleshoot the alert + +1. Identify the buried jobs: Use the `beanstalk-tool` to inspect the Beanstalk server and list the buried jobs in the tubes. If you don't have `beanstalk-tool`, install it using pip: + + ``` + pip install beanstalkc + beanstalk-tool <beanstalk server host>:<beanstalk server port> stats_tube <tube_name> + ``` + +2. Examine the buried jobs: To investigate the cause of the buried jobs, find related logs, either from the Beanstalk server or from the consumer application. Analyzing the logs can lead to the root cause of the problem. + +3. Fix the issue: Once you identify the cause, resolve the issue in either the consumer application, or if necessary, in the Beanstalk server configuration. + +4. Kick the buried jobs: After resolving the issue, you need to manually kick the buried jobs back into the queue for processing. Use the following command with `beanstalk-tool`: + + ``` + beanstalk-tool <beanstalk server host>:<beanstalk server port> kick <number_of_jobs_to_kick> --tube=<tube_name> + ``` + +5. Monitor the queue: After kicking the buried jobs, monitor the queue and ensure that the jobs are processed without encountering more errors. + +### Useful resources + +1. [Beanstalk Documentation](https://beanstalkd.github.io/) diff --git a/health/guides/beanstalk/beanstalk_tube_buried_jobs.md b/health/guides/beanstalk/beanstalk_tube_buried_jobs.md new file mode 100644 index 00000000..76a43cc6 --- /dev/null +++ b/health/guides/beanstalk/beanstalk_tube_buried_jobs.md @@ -0,0 +1,30 @@ +### Understand the alert + +This alert monitors the number of buried jobs in each beanstalkd tube. If you receive this alert, it means that there are jobs that have been buried, and you need to investigate the cause. The warning threshold is set at more than zero buried jobs, and the critical threshold is set at more than ten buried jobs. + +### What are buried jobs? + +In Beanstalkd, buried jobs are jobs that were moved to the buried state deliberately or jobs that have failed repeatedly. They're kept in a separate queue and will not be processed by workers until they're explicitly handled or deleted. + +### Troubleshoot the alert + +1. Check the Beanstalkd logs for any errors or pertinent information related to the buried jobs. You can find the logs in the `/var/log/beanstalkd.log` file (the default log file location) or any other custom location defined for Beanstalkd. + +2. Use the `beanstalk-console` or a similar tool to inspect the buried jobs to determine their causes. You can download `beanstalk-console` [here](https://github.com/ptrofimov/beanstalk_console). + +3. Review the applications or workers that are interacting with the affected tubes to find any possible issues or bugs. + +4. If the buried jobs are blocking the processing of other jobs, consider moving them to another tube with higher priority or increase the number of workers processing the tube. + +5. If the buried jobs are safe to delete or requeue, do so to clear the count and alleviate the alert. You can use the following commands to kick or delete jobs using the `beanstalk-cli`: + ``` + beanstalk-cli kick-job [<job-id>] + beanstalk-cli delete-job [<job-id>] + ``` + +6. If none of the steps above help mitigate the issue, consider contacting the sysadmin or developers of the application using Beanstalkd. + +### Useful resources + +1. [Beanstalkd Protocol](https://raw.githubusercontent.com/beanstalkd/beanstalkd/master/doc/protocol.txt) +2. [Beanstalk_console - a web-based beanstalk queue server console](https://github.com/ptrofimov/beanstalk_console) |