Rsyslog Notes For Future Me

Table of Contents

• Installation and Setup
• Configuration Structure
• Core Modules
• Performance
• Monitoring and Troubleshooting
• Best Practices

Installation and Setup

Rsyslog is pre-installed on most Ubuntu distributions, but ensure you're on the latest:

sudo apt-get update && sudo apt-get install rsyslog

Configuration Structure

Directory Organization

This configuration pattern is the least cool thing about rsyslog but, store configuration files in /etc/rsyslog.d/ using a numbered prefix system for predictable loading order:

Note: 10- is the default prefix to enable you to assign lesser indexed configs (09, 08, 07, ..)

/etc/rsyslog.d/
├── 10-system.conf     # System-wide defaults and global settings
├── 20-supervisor.conf # Supervisor-specific configuration
├── 30-django.conf     # Django application logs
├── 40-rabbitmq.conf   # RabbitMQ server logs
└── 50-redis.conf      # Redis server logs

Example Configurations

10-system.conf (Base Configuration)

# Enable UDP and TCP inputs
module(load="imudp")
module(load="imtcp")

input(type="imudp" port="514")
input(type="imtcp" port="514")

# Main queue configuration
main_queue(
    type="LinkedList"
    queue.workerThreads="4"
    queue.dequeueBatchSize="500"
    queue.size="100000"
)

Service-Specific Configuration (e.g., 30-django.conf)

module(load="imfile")

input(type="imfile"
      File="/var/log/django/*.log"
      Tag="django"
      Severity="info"
      Facility="local6")

# Remote forwarding configuration
action(type="omfwd"
      target="<target_ip>"
      protocol="tcp"
      port="514"
      action.resumeRetryCount="-1"
      queue.type="LinkedList"
      queue.filename="django-forward-queue")

Core Modules

Essential modules for high-throughput environments:

| Module | Purpose | Use Case | |--------|---------|----------| | imudp | UDP input | Fast ingestion, suitable for non-critical logs | | imtcp | TCP input | Reliable ingestion for critical logs | | imfile | File monitoring | Watching log files from applications | | omfile | File output | Local log storage | | omfwd | Log forwarding | Remote syslog or aggregator integration | | omkafka | Kafka integration | Real-time log streaming |

Performance Optimization

Threading Configuration

main_queue(
    type="LinkedList"
    queue.workerThreads="4"     # Increase for higher concurrency
    queue.dequeueBatchSize="500" # Larger batches for better throughput
    queue.size="100000"         # Adjust based on memory availability
)

Disk-Based Queues

For guaranteed delivery when remote endpoints are unavailable:

action(
    type="omfwd"
    target="10.0.0.51"
    protocol="tcp"
    action.resumeRetryCount="-1"
    queue.type="LinkedList"
    queue.filename="fwdQueue"
    queue.maxdiskspace="5g"
    queue.saveonshutdown="on"
)

Rate Limiting

Protect against traffic spikes:

module(
  load="imudp"
  rateLimit.Interval="1"
  rateLimit.Burst="20000"
  )

Monitoring and Troubleshooting

Health Checks

1. Enable impstats module for internal metrics:

module(
  load="impstats"
  interval="60"
  severity="7"
  log.file="/var/log/rsyslog-stats.log"
)

2. Key metrics to monitor:

• Queue depth
• Message throughput
• Drop counts
• Processing delays

Testing Tools

# Test log generation
logger "Test message"

# Validate configuration
sudo rsyslogd -N1

# Debug mode (development only)
sudo rsyslogd -dn

Best Practices

Performance

• Check queue depths regularly
• Use disk queues for critical logs
• Rate limiting
• Scale worker threads by CPU

Maintenance

• Log rotation
• DU monitoring
• Use reload instead of restart for config changes:

sudo systemctl reload rsyslog

JSON Output

For better log parsing, maybe use JSON templates:

template(name="json-template"
         type="list") {
    constant(value="{")
      constant(value="\"timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"message\":\"")    property(name="msg")
      constant(value="\",\"host\":\"")       property(name="hostname")
    constant(value="\"}")
}

Checklist

• [ ] Check file permissions
• [ ] Configure rate limiting
• [ ] Test (rsyslogd -N1)