Cyber/ravenscott-blog
1
0
Fork 0
forked from snxraven/ravenscott-blog
Code Pull Requests Activity

Add new article

Browse Source
This commit is contained in:
Raven Scott 2024-09-16 09:55:10 -04:00
parent b097c5c604
commit cea6e560fb
1 changed files with 470 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

470
markdown/Using AI to Monitor Access Logs.md Normal file
View File

@ -0,0 +1,470 @@
# AI Log Monitoring System
# Source
## https://git.ssh.surf/snxraven/ai-nginx-log-security
## Introduction
The AI Log Monitoring System is a powerful and extensible solution designed to enhance security monitoring by integrating AI-based analysis into traditional NGINX log management.
By continuously tailing log files and utilizing an AI model to analyze log entries in real-time, this system automates the detection of security threats, sends alerts to designated channels (like Discord), and manages conversation history for ongoing log analysis.
The solution is composed of two primary scripts:
- `ai_log.js`: Handles log monitoring, buffering, and sending logs to a backend for AI processing.
- `ai_log_backend.js`: Manages AI-based analysis, conversation history, and exposes an API for interacting with the AI service.
## Screenshots
Here are some screenshots of the application in action:
![Screenshot 1](https://git.ssh.surf/snxraven/ai-nginx-log-security/media/branch/main/screenshots/s1.png)
![Screenshot 2](https://git.ssh.surf/snxraven/ai-nginx-log-security/media/branch/main/screenshots/s2.png)
## Prerequisites
Before setting up and running the AI Log Monitoring System, ensure you have the following components installed and configured on your machine:
- **Node.js**: Version 14.x or higher is required to run the JavaScript code.
- **npm**: Version 6.x or higher is needed to manage project dependencies.
- **Docker**: Required for running the AI model, particularly if using a containerized llama 3.1 model for processing.
- **NGINX**: The web server generating logs that the system will monitor.
- **Discord**: A Discord webhook URL is necessary for sending security alerts.
- **llama-cpp-python[web]**: A docker container preferably although not required: [Read the Docs](https://llama-cpp-python.readthedocs.io/en/latest/server/)
Ensure you have administrative privileges on the machine to install and configure these dependencies.
## Installation
Clone the repository to your local machine:
```bash
git clone git@git.ssh.surf:snxraven/ai-nginx-log-security.git
cd ai-nginx-log-security
```
Install the required Node.js dependencies for both applications:
```bash
npm install
```
## Configuration
### Environment Variables
Environment variables play a critical role in configuring the behavior of the AI Log Monitoring System.
They allow you to customize log directories, adjust token limits, and set up necessary credentials.
Create a `.env` file in the root of your project and populate it with the following variables:
```bash
# General Configuration
DEBUG=true # Enable detailed logging output for debugging
WEBHOOKURL=https://discord.com/api/webhooks/your-webhook-id # Discord webhook URL for sending alerts
MAX_CONTENT_LENGTH=2000 # Maximum length for content scraped from web pages
# AI Service Configuration
TIMEOUT_DURATION=100000 # Maximum duration (in ms) for API requests to complete
MAX_TOKENS=8000 # Maximum tokens allowed in a conversation before trimming
TOLERANCE=100 # Extra tokens allowed before forcing a trim of conversation history
```
**Explanation of Environment Variables**:
- **DEBUG**: Enables verbose logging, including debug-level messages. Useful during development and troubleshooting.
- **WEBHOOKURL**: The Discord webhook URL where alerts will be sent. This should be a valid and secure URL.
- **MAX_CONTENT_LENGTH**: Limits the length of content extracted from web pages to avoid overloading the system or sending excessively long messages.
- **TIMEOUT_DURATION**: Sets a timeout for requests to the AI service, preventing the system from hanging indefinitely.
- **MAX_TOKENS**: Controls the total number of tokens (words + punctuation) allowed in a conversation with the AI. This prevents the AI from consuming too much memory or processing power.
- **TOLERANCE**: A buffer to avoid hitting the MAX_TOKENS limit exactly, ensuring smoother operation by providing a cushion before trimming conversation history.
### Directory Structure
The system monitors log files stored in a specific directory. Ensure this directory exists and is correctly set in the script:
```bash
mkdir -p /dockerData/logs
```
If your NGINX logs are stored in a different directory, update the `LOG_DIRECTORY` constant in `ai_log.js` accordingly:
```javascript
const LOG_DIRECTORY = '/your/custom/path/to/logs';
```
### Ignored IPs and Subnets
The system allows you to specify IP addresses and subnets that should be ignored during log monitoring.
This is particularly useful for filtering out trusted sources or known harmless traffic (e.g., public DNS servers).
- **Ignored IPs**: Directly listed IP addresses that should be skipped during processing.
- **Ignored Subnets**: Subnets specified in CIDR notation that represent ranges of IP addresses to be ignored.
To customize the ignored IPs and subnets, modify the `ignoredIPs` and `ignoredSubnets` arrays in `ai_log.js`:
```javascript
const ignoredIPs = ['1.1.1.1', '1.0.0.1', '8.8.8.8', '8.8.4.4'];
const ignoredSubnets = [
'173.245.48.0/20', '103.21.244.0/22', '103.22.200.0/22',
// Add more subnets as needed
];
```
This ensures that traffic from these sources is not flagged or acted upon, reducing false positives and focusing the AI on more critical threats.
## Usage
### Running `ai_log.js`
The `ai_log.js` script is responsible for continuously monitoring NGINX logs, buffering log entries, and sending them to the backend for analysis.
It also handles real-time actions, such as banning IP addresses and sending alerts.
To start the log monitoring process, execute the following command:
```bash
node ai_log.js
```
The script will immediately begin reading logs from the specified directory and processing them according to the rules defined in the script.
The logs will be buffered and periodically sent to the backend for AI-based analysis.
### Running `ai_log_backend.js`
The `ai_log_backend.js` script sets up an Express server that interfaces with the AI model to analyze log data.
It also manages conversation history and provides endpoints for interacting with the system.
To start the backend server:
```bash
node ai_log_backend.js
```
By default, the server will be running on `http://localhost:3001`.
This server handles incoming log data, processes it with the AI model, and returns actionable insights, including potential security alerts.
## How It Works
### Log Monitoring and Buffering
The `ai_log.js` script uses the `Tail` module to monitor NGINX log files in real-time.
As new lines are added to the logs, the script reads and buffers them.
The buffer size is configurable, allowing the system to batch-process logs before sending them to the backend.
**Key Features**:
- **Real-Time Monitoring**: Continuously monitors specified log files for new entries.
- **Buffering**: Collects log entries in a buffer to reduce the frequency of network requests to the backend.
- **Ignored Entries**: Filters out log entries from specified IPs and subnets, as well as entries matching certain patterns.
### Sending Logs to Backend
When the log buffer reaches a predefined size or a set time interval elapses, the buffered logs are sent to the backend for AI processing.
The backend analyzes the logs to detect potential security threats, generate alerts, and manage conversation history.
**Process Overview**:
1. **Buffer Accumulation**: Logs are collected in a buffer until a threshold is met.
2. **Buffer Flush**: The buffer is sent to the backend in one request, optimizing network usage.
3. **Backend Analysis**: The AI model analyzes the logs for suspicious activity or patterns.
### AI-Based Log Analysis
The backend server (`ai_log_backend.js`) leverages an AI model (e.g., llama 3.1) to analyze the logs and detect potential security threats.
The AI operates based on a custom prompt that instructs it on how to interpret the logs, which IPs to ignore, and what actions to take.
**AI Model Usage**:
- **Custom Prompt**: The AI is guided by a detailed prompt that defines its behavior and decision-making process.
- **Log Parsing**: The AI processes log lines to identify malicious patterns, potential attacks, or other security concerns.
- **Actionable Insights**: Based on the analysis, the AI generates alerts, suggests actions (like banning IPs), or provides general observations.
### Token Management and Conversation History
To ensure efficient operation and prevent resource exhaustion, the system carefully manages the number of tokens used in AI conversations.
Token management involves trimming older parts of the conversation history to stay within predefined limits.
**Token Management Strategies**:
- **Counting Tokens**: The system counts tokens for each message in the conversation history.
- **Trimming History**: If the token count exceeds the maximum allowed, the oldest messages are removed.
- **Tolerance Buffer**: A small buffer is maintained to avoid hitting the exact token limit, ensuring smoother performance.
### Security Alert Handling
When the AI detects a potential security threat, it generates an alert.
These alerts are processed by the backend and can trigger actions like banning an IP address or sending a notification to a Discord channel.
**Alert Workflow**:
1. **Detection**: The AI identifies a suspicious activity or pattern in the logs.
2. **Alert Generation**: The AI creates an alert message, formatted for clarity and readability.
3. **IP Banning**: If an IP is identified as malicious, the system can execute a ban command to block it.
4. **Discord Notification**: Alerts are sent to a designated Discord channel for real-time monitoring and action.
### Discord Integration
The system integrates with Discord to send alerts and notifications.
This is particularly useful for real-time monitoring, allowing administrators to receive and act on security alerts instantly.
**Integration Details**:
- **Webhook-Based Alerts**: The system uses a Discord webhook to send alerts as embedded messages.
- **Formatted Messages**: Alerts are formatted with titles, descriptions, and timestamps to ensure they are easy to read and understand.
- **IP Banning Alerts**: When an IP is banned, the system includes the IP address in the Discord alert.
## API Endpoints
The backend server (`ai_log_backend.js`) exposes several API endpoints for interacting with the AI service, managing conversation history, and controlling the system.
### POST /api/v1/chat
This endpoint processes incoming NGINX logs by sending them to the AI model for analysis.
- **Request Body**:
- `message`: A string containing one or more NGINX log lines.
- **Response**:
- A JSON object with the AI's analysis and any detected alerts.
**Example Request**:
```json
{
"message": "127.0.0.1 - - [12/Mar/2024:10:12:33 +0000] \"GET /index.html HTTP/1.1\" 200 3050"
}
```
**Example Response**:
```json
{
"role": "assistant",
"content": "GENERAL: No suspicious activity detected. Routine request logged."
}
```
### GET /api/v1/conversation-history
This endpoint retrieves the conversation history for the IP address making the request.
It is useful for reviewing the AI's past analyses and actions.
- **Response**:
- A JSON array containing the conversation history.
**Example Response**:
```json
[
{ "role": "system", "content": "You are a security AI..." },
{ "role": "user", "content": "127.0.0.1 - - [12/Mar/2024:10:12:33 +0000] ..." },
{ "role": "assistant", "content": "GENERAL: No suspicious activity detected..." }
]
```
### POST /api/v1/restart-core
This endpoint restarts the core AI service running in a Docker container.
This is useful if the AI service becomes unresponsive or needs to refresh its state.
- **Response**:
- A JSON object with the output of the Docker restart command.
**Example Request**:
```bash
curl -X POST http://localhost:3001/api/v1/restart-core
```
**Example Response**:
```json
{
"stdout": "llama-gpu-server\n"
}
```
### POST /api/v1/reset-conversation
This endpoint resets the conversation history for the requesting IP address, effectively starting a new session with the AI.
This can be useful for clearing outdated context and beginning a fresh analysis.
- **Response**:
- A JSON object confirming the reset action.
**Example Request**:
```bash
curl -X POST http://localhost:3001/api/v1/reset-conversation
```
**Example Response**:
```json
{
"message": "Conversation history reset for IP: 127.0.0.1"
}
```
## Logging
Logging is a critical component of the AI Log Monitoring System, providing insights into system operations, debugging information, and records of security actions.
### Log Levels
The system categorizes logs into different levels to help you quickly identify the nature and severity of messages:
- **INFO**: General information about the system's operations.
- **WARN**: Indications of potential issues that may require attention.
- **ERROR**: Logs generated when an error occurs during processing.
- **SUCCESS**: Messages that indicate successful operations or actions.
- **DEBUG**: Detailed messages intended for debugging purposes, enabled when `DEBUG=true`.
### Log Structure
Each log message includes a timestamp, log level, and message content. For example:
```plaintext
2024-03-12 10:12:33 [INFO] Starting to read log from: /dockerData/logs/access.log
2024-03-12 10:12:35 [DEBUG] Read line: 127.0.0.1 - - [12/Mar/2024:10:12:33 +0000] "GET /index.html HTTP/1.1" 200 3050
2024-03-12 10:12:35 [SUCCESS] Log buffer sent to backend successfully.
```
### Debugging
When `DEBUG=true`, the system provides detailed logs that include every step of the processing workflow.
This includes reading log lines, checking for ignored IPs, sending data to the backend, and receiving responses from the AI.
These logs are invaluable during development and troubleshooting, as they offer full visibility into the system's inner workings.
## Security Considerations
Security is a paramount concern when monitoring logs and responding to potential threats.
The AI Log Monitoring System includes several mechanisms to enhance security and minimize false positives.
### IP Whitelisting
The system allows you to specify IP addresses and subnets that should be ignored during analysis.
This is particularly useful for avoiding alerts from known and trusted sources, such as public DNS servers or internal IP ranges.
### Rate Limiting and Banning
To protect your infrastructure from repeated attacks, the system can automatically ban IP addresses identified as malicious.
The banning process is executed via shell commands, and the system includes a delay mechanism to prevent overloading the network with too many ban requests in a short period.
### Data Privacy
All sensitive data, such as IP addresses and conversation history, is handled securely.
The system ensures that only necessary data is stored and processed, with an emphasis on minimizing exposure to potential vulnerabilities.
## Performance Optimization
The AI Log Monitoring System is designed to be efficient and scalable, handling large volumes of log data with minimal overhead.
However, some optimizations can further enhance performance, especially in high-traffic environments.
### Managing Token Limits
By carefully managing the number of tokens in the AI's conversation history, the system prevents memory overuse and ensures faster response times. The `MAX_TOKENS` and `TOLERANCE` variables allow you to fine-tune this behavior.
### Efficient Log Parsing
The system uses regular expressions and filtering techniques to parse and analyze log files efficiently. By focusing only on relevant log entries and ignoring unnecessary ones, the system reduces processing time and improves accuracy.
## Troubleshooting
If you encounter issues while using the AI Log Monitoring System, this section provides guidance on common problems and how to resolve them.
### Common Issues
- **No Logs Detected**: Ensure that the log directory is correctly specified and that log files are present.
- **AI Service Unresponsive**: Restart the AI service using the `/api/v1/restart-core` endpoint.
- **Excessive False Positives**: Review the ignored IPs and subnets to ensure that known safe traffic is excluded.
### Restarting Services
If the AI service becomes unresponsive or you need to apply changes, use the `/api/v1/restart-core` endpoint to restart the core AI Docker container.
This refreshes the model and clears any stale states.
### Log Analysis
Review the logs generated by the system to identify potential issues. Focus on `ERROR` and `WARN` level messages to spot critical problems quickly. Use the `DEBUG` logs for deeper investigation during development or when troubleshooting specific issues.
## Customization
The AI Log Monitoring System is highly customizable, allowing you to tailor its behavior to your specific needs.
### Modifying the AI Prompt
The AI's behavior is guided by a custom prompt that defines how it should interpret log data and what actions it should take.
You can modify this prompt in `ai_log_backend.js` to adjust the AI's focus or add new rules.
**Example Customization**:
```javascript
const prompt = `
You are a security AI responsible for analyzing web traffic...
- Ignore any IPs within the range 10.0.0.0/8.
- Flag any requests to /admin as potential threats.
- Use emojis to convey the severity of alerts.
`;
```
### Adjusting Buffer Limits
The log buffer size determines how many log lines are collected before they are sent to the backend.
Adjust this size to balance network usage and processing frequency.
**Example Customization**:
```javascript
const LOG_BUFFER_LIMIT = 30; // Increase the buffer size to 30 lines
```
### Extending Log Parsing Capabilities
You can extend the log parsing functionality by adding new regular expressions or parsing logic to handle different log formats or detect new types of threats.
**Example Customization**:
```javascript
tail.on('line', async (line) => {
// Add custom logic to detect SQL injection attempts
if (/SELECT.*FROM/i.test(line)) {
log.warn(`Potential SQL injection detected: ${line}`);
}
});
```
# My Findings
The AI Log Monitoring System is an advanced and highly customizable solution for real-time security monitoring of NGINX logs. By leveraging AI-based analysis, it significantly improves the detection of potential security threats, streamlines alert notifications, and provides administrators with actionable insights through Discord integration. With features like log buffering, IP whitelisting, and token management, the system efficiently handles large volumes of log data while minimizing false positives. Its modular design, supported by two core scripts, ensures scalability and adaptability, making it a robust tool for securing web infrastructure.