[CIP-Security] [CR2.10] Response to audit processing failure ============================================================ .. contents:: .. list-table:: Revision History :header-rows: 1 * - Revision No - Date - Change description - Author - Reviewed by * - 001 - 2022-02-21 - Draft IEC-62443-4-2 CR-2.10 guideline - George Hsiao
Jared Wu - To be reviewed by CIP Security WG members 1. Objective ------------ The primary objective of this document is to create a guideline for protection of critical system functions in case of audit processing failiure. Without appropriate response to audit processing failure, an attacker’s activities can go unnoticed, and evidence of whether or not the attack led to a breach can be inconclusive. 2. Common Approach for Response to audit processing failure ----------------------------------------------------------- The audit logs are stored with-in a pre-allocated storage capability. The CIP user should consider the types of audit logging to be performed and the audit log processing requirements when allocating audit log storage capacity. Allocating sufficient audit log storage capacity reduces the likelihood of capacity exceeded and resulting in the potential loss or reduction of audit logging capability. The CIP platform should consider to response for the audit processing failure 2.1. Alert the allocated audit log storage volume is nearly full ---------------------------------------------------------------- The CIP user shall consider to monitor the audit log storage volume usage. Alert to the organization-defined personnel or roles in organization-defined time period. 2.2. Take the actions to response to audit log processing failure ----------------------------------------------------------------- Audit logging process failures include software and hardware errors, failures in audit log capturing mechanisms, and reaching or exceeding audit log storage capacity. Organization-defined actions include overwriting oldest audit records, shutting down the system, and stopping the generation of audit records. 3. CIP Features for Response to Audit Processing Failure -------------------------------------------------------- This requirement is supported by the CIP platform. However, this requirement cannot be met by the CIP platform alone. This will be done by the application developers using the CIP platform. For example, building a GUI in application to configure **auditd** configuration. 3.1. auditd ----------- **3.1.1. Space Left Action for auditd** **auditd** provided in CIP can be used to detect and set the appropriate action when the free space in the filesystem containing log files drops drop to a specified threshold CIP user can configure `space_left `__ and `space_left_action `__ parameters in /etc/audit/auditd.conf to spcify the remaining space (in megabytes) for low disk alert and the what action (ignore, syslog, rotate, email, exec, suspend, single, halt\ *)* to take. In example below, warning email will be sent to email account specified in `action_mail_acct `__ parameter when the free space in the filesystem containing log files drop below 75 megabytes :: space_left = 75 space_left_action = email **3.1.2. Disk Error and Disk Full Detection for auditd** Configure `disk_full_action `__ and `disk_error_action `__ in /etc/audit/auditd.conf to suspend when disk got error or full. The actions are ignore, syslog, rotate (for disk full only), exec, suspend, single, and halt. For example, action “syslog” means that it will issue a warning to syslog. :: disk_full_action = syslog disk_error_action = syslog **3.1.3. The max log file action for auditd** Configure the `max_log_file `__, `max_log_file_action `__ and `num_logs `__ in auditd.conf. The actions are\ *ignore*, *syslog*, *suspend*, *rotate* and \*keep_logs**.\* In example below, the max_log_file is 30 M and number of log files to keep is 6 as “rotate” action is given. Their multiplied value is equal to 180 MB. :: max_log_file = 30 num_logs = 6 max_log_file_action = ROTATE 3.2. The log daemon not support the space left, error detection or max log file features ---------------------------------------------------------------------------------------- **3.2.1. The space left action for the log daemon** **that does not** **support space left detection** Use the cron daemon on CIP platform to monitor the disk usage periodic and send a mail to alert the organization-defined personnel or roles. 3.2.1.1. Use ``df`` to report the storage usage :: $ df -H | grep -vE '^Filesystem|tmpfs|udev' | awk '{ print $5 " " $1 }' 28% /dev/sda1` Save this result in a variable :: $ disk_usage = `df -H | grep -vE '^Filesystem|tmpfs|udev' | awk '{ print $5 " " $1 }'` 3.2.1.2. Use awk and cut to get the storage usage number :: $ echo $disk_usage | awk '{print $1}'|cut -d '%' -f1 28 Keep this result in a variable :: $ usage = `df -H | grep -vE '^Filesystem|tmpfs|udev' | awk '{ print $5 " " $1 }'` 3.2.1.3. Compare the disk usage and send a mail to alert :: if [ "$usage" -ge "80" ]; then logger -t "system.storage" -p syslog.warn "Running out of space \"$partition ($usage%)\"" mail -s "Alert: Almost out of disk space $usage%" your@email.tld fi Use the above disk usage code in a script for checking disk usage and alert by e-mail if over the threashold. Setup cron job to monitor the disk usage hourly. :: $ cp diskAlert.sh /etc/cron.hourly/ $ chmod +x /etc/cron.hourly/diskAlert.sh **3.2.2. For the log daemon** **that does** **not support disk error detection** Alert when within the predefined time period to the predefined person when the audit processing failure occurs 3.2.2.1. Use smart tool to report the disk health :: $ smartctl -H /dev/sda smartctl 6.6 2016-05-31 r4324 [i686-linux-4.9.0-16-686] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org === START OF READ SMART DATA SECTION === SMART Health Status: OK Keep the SMART Health Status in a variable. :: $ SMART_status=`smartctl -H /dev/sda|grep 'SMART Health Status'| awk '{ print $4 }'` 3.2.2.2. Check for disk bad blocks or bad sector using *badblocks*. :: $ badblocks -v /dev/sda1 Checking blocks 0 to 16777215 Checking for bad blocks (read-only test): done Pass completed, 0 bad blocks found. (0/0/0 errors) Keep the badblocks scan result in a variable. :: $ BADBLOCK_status=$? 3.2.2.3. Alert if one of the above disk health status is not OK. :: if [ "$SMART_status" != "OK" -o "$BADBLOCK_status" != "0" ]; then logger -t "system.storage" -p syslog.warn "Disk error detected" mail -s "Alert: Disk error detected" your@email.tld fi Use the above disk error code in a diskError.sh script and setup cron job to monitor the disk error periodically. :: $ cp diskError.sh /etc/cron.hourly/ $ chmod +x /etc/cron.hourly/diskError.sh **3.2.3. For the log daemon** **that does** **not support log rotation** Use logrotate to limit the disk space usage. In /etc/logrotate.config and all the files in /etc/logrotate.d/\* to rotate the log file. This example we configure /etc/logrotate.d/rsyslog to rotate /var/log/syslog while it overs the size **2M** with only **3 rotation**. :: /var/log/syslog { { rotate 3 maxsize 2M ... } } Reference --------- | https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-53r5.pdf#page=95 | https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-53r5.pdf#page=96 | https://manpages.debian.org/testing/auditd/auditd.conf.5.en.html | https://linux.die.net/man/8/logrotate | https://linux.die.net/man/8/syslog-ng | https://linux.die.net/man/1/df | https://man7.org/linux/man-pages/man5/crontab.5.html | https://www.simplified.guide/linux/disk-error-check