| screenshots | ||
| .gitignore | ||
| ai_log_backend.js | ||
| ai_log.js | ||
| package.json | ||
| README.md | ||
AI Log Monitoring System
This repository contains two interdependent Node.js applications, ai_log.js and ai_log_backend.js.
These applications collaborate to monitor NGINX logs, detect potential security threats, and manage conversation history for AI-based log analysis.
Screenshots
Here are some screenshots of the application in action:
Table of Contents
- Introduction
- Prerequisites
- Installation
- Configuration
- Usage
- How It Works
- API Endpoints
- Logging
- Security Considerations
- Performance Optimization
- Troubleshooting
- Customization
- Contributing
- License
Introduction
The AI Log Monitoring System is a powerful and extensible solution designed to enhance security monitoring by integrating AI-based analysis into traditional NGINX log management.
By continuously tailing log files and utilizing an AI model to analyze log entries in real-time, this system automates the detection of security threats, sends alerts to designated channels (like Discord), and manages conversation history for ongoing log analysis.
The solution is composed of two primary scripts:
ai_log.js: Handles log monitoring, buffering, and sending logs to a backend for AI processing.ai_log_backend.js: Manages AI-based analysis, conversation history, and exposes an API for interacting with the AI service.
Prerequisites
Before setting up and running the AI Log Monitoring System, ensure you have the following components installed and configured on your machine:
- Node.js: Version 14.x or higher is required to run the JavaScript code.
- npm: Version 6.x or higher is needed to manage project dependencies.
- Docker: Required for running the AI model, particularly if using a containerized llama 3.1 model for processing.
- NGINX: The web server generating logs that the system will monitor.
- Discord: A Discord webhook URL is necessary for sending security alerts.
- llama-cpp-python[web]: A docker container preferably although not required: Read the Docs
Ensure you have administrative privileges on the machine to install and configure these dependencies.
Installation
Clone the repository to your local machine:
git clone git@git.ssh.surf:snxraven/ai-nginx-log-security.git
cd ai-nginx-log-security
Install the required Node.js dependencies for both applications:
npm install
Configuration
Environment Variables
Environment variables play a critical role in configuring the behavior of the AI Log Monitoring System.
They allow you to customize log directories, adjust token limits, and set up necessary credentials.
Create a .env file in the root of your project and populate it with the following variables:
# General Configuration
DEBUG=true # Enable detailed logging output for debugging
WEBHOOKURL=https://discord.com/api/webhooks/your-webhook-id # Discord webhook URL for sending alerts
MAX_CONTENT_LENGTH=2000 # Maximum length for content scraped from web pages
# AI Service Configuration
TIMEOUT_DURATION=100000 # Maximum duration (in ms) for API requests to complete
MAX_TOKENS=8000 # Maximum tokens allowed in a conversation before trimming
TOLERANCE=100 # Extra tokens allowed before forcing a trim of conversation history
Explanation of Environment Variables:
- DEBUG: Enables verbose logging, including debug-level messages. Useful during development and troubleshooting.
- WEBHOOKURL: The Discord webhook URL where alerts will be sent. This should be a valid and secure URL.
- MAX_CONTENT_LENGTH: Limits the length of content extracted from web pages to avoid overloading the system or sending excessively long messages.
- TIMEOUT_DURATION: Sets a timeout for requests to the AI service, preventing the system from hanging indefinitely.
- MAX_TOKENS: Controls the total number of tokens (words + punctuation) allowed in a conversation with the AI. This prevents the AI from consuming too much memory or processing power.
- TOLERANCE: A buffer to avoid hitting the MAX_TOKENS limit exactly, ensuring smoother operation by providing a cushion before trimming conversation history.
Directory Structure
The system monitors log files stored in a specific directory. Ensure this directory exists and is correctly set in the script:
mkdir -p /dockerData/logs
If your NGINX logs are stored in a different directory, update the LOG_DIRECTORY constant in ai_log.js accordingly:
const LOG_DIRECTORY = '/your/custom/path/to/logs';
Ignored IPs and Subnets
The system allows you to specify IP addresses and subnets that should be ignored during log monitoring.
This is particularly useful for filtering out trusted sources or known harmless traffic (e.g., public DNS servers).
- Ignored IPs: Directly listed IP addresses that should be skipped during processing.
- Ignored Subnets: Subnets specified in CIDR notation that represent ranges of IP addresses to be ignored.
To customize the ignored IPs and subnets, modify the ignoredIPs and ignoredSubnets arrays in ai_log.js:
const ignoredIPs = ['1.1.1.1', '1.0.0.1', '8.8.8.8', '8.8.4.4'];
const ignoredSubnets = [
'173.245.48.0/20', '103.21.244.0/22', '103.22.200.0/22',
// Add more subnets as needed
];
This ensures that traffic from these sources is not flagged or acted upon, reducing false positives and focusing the AI on more critical threats.
Usage
Running ai_log.js
The ai_log.js script is responsible for continuously monitoring NGINX logs, buffering log entries, and sending them to the backend for analysis.
It also handles real-time actions, such as banning IP addresses and sending alerts.
To start the log monitoring process, execute the following command:
node ai_log.js
The script will immediately begin reading logs from the specified directory and processing them according to the rules defined in the script.
The logs will be buffered and periodically sent to the backend for AI-based analysis.
Running ai_log_backend.js
The ai_log_backend.js script sets up an Express server that interfaces with the AI model to analyze log data.
It also manages conversation history and provides endpoints for interacting with the system.
To start the backend server:
node ai_log_backend.js
By default, the server will be running on http://localhost:3001.
This server handles incoming log data, processes it with the AI model, and returns actionable insights, including potential security alerts.
How It Works
Log Monitoring and Buffering
The ai_log.js script uses the Tail module to monitor NGINX log files in real-time.
As new lines are added to the logs, the script reads and buffers them.
The buffer size is configurable, allowing the system to batch-process logs before sending them to the backend.
Key Features:
- Real-Time Monitoring: Continuously monitors specified log files for new entries.
- Buffering: Collects log entries in a buffer to reduce the frequency of network requests to the backend.
- Ignored Entries: Filters out log entries from specified IPs and subnets, as well as entries matching certain patterns.
Sending Logs to Backend
When the log buffer reaches a predefined size or a set time interval elapses, the buffered logs are sent to the backend for AI processing.
The backend analyzes the logs to detect potential security threats, generate alerts, and manage conversation history.
Process Overview:
- Buffer Accumulation: Logs are collected in a buffer until a threshold is met.
- Buffer Flush: The buffer is sent to the backend in one request, optimizing network usage.
- Backend Analysis: The AI model analyzes the logs for suspicious activity or patterns.
AI-Based Log Analysis
The backend server (ai_log_backend.js) leverages an AI model (e.g., llama 3.1) to analyze the logs and detect potential security threats.
The AI operates based on a custom prompt that instructs it on how to interpret the logs, which IPs to ignore, and what actions to take.
AI Model Usage:
- Custom Prompt: The AI is guided by a detailed prompt that defines its behavior and decision-making process.
- Log Parsing: The AI processes log lines to identify malicious patterns, potential attacks, or other security concerns.
- Actionable Insights: Based on the analysis, the AI generates alerts, suggests actions (like banning IPs), or provides general observations.
Token Management and Conversation History
To ensure efficient operation and prevent resource exhaustion, the system carefully manages the number of tokens used in AI conversations.
Token management involves trimming older parts of the conversation history to stay within predefined limits.
Token Management Strategies:
- Counting Tokens: The system counts tokens for each message in the conversation history.
- Trimming History: If the token count exceeds the maximum allowed, the oldest messages are removed.
- Tolerance Buffer: A small buffer is maintained to avoid hitting the exact token limit, ensuring smoother performance.
Security Alert Handling
When the AI detects a potential security threat, it generates an alert.
These alerts are processed by the backend and can trigger actions like banning an IP address or sending a notification to a Discord channel.
Alert Workflow:
- Detection: The AI identifies a suspicious activity or pattern in the logs.
- Alert Generation: The AI creates an alert message, formatted for clarity and readability.
- IP Banning: If an IP is identified as malicious, the system can execute a ban command to block it.
- Discord Notification: Alerts are sent to a designated Discord channel for real-time monitoring and action.
Discord Integration
The system integrates with Discord to send alerts and notifications.
This is particularly useful for real-time monitoring, allowing administrators to receive and act on security alerts instantly.
Integration Details:
- Webhook-Based Alerts: The system uses a Discord webhook to send alerts as embedded messages.
- Formatted Messages: Alerts are formatted with titles, descriptions, and timestamps to ensure they are easy to read and understand.
- IP Banning Alerts: When an IP is banned, the system includes the IP address in the Discord alert.
API Endpoints
The backend server (ai_log_backend.js) exposes several API endpoints for interacting with the AI service, managing conversation history, and controlling the system.
POST /api/v1/chat
This endpoint processes incoming NGINX logs by sending them to the AI model for analysis.
-
Request Body:
message: A string containing one or more NGINX log lines.
-
Response:
- A JSON object with the AI's analysis and any detected alerts.
Example Request:
{
"message": "127.0.0.1 - - [12/Mar/2024:10:12:33 +0000] \"GET /index.html HTTP/1.1\" 200 3050"
}
Example Response:
{
"role": "assistant",
"content": "GENERAL: No suspicious activity detected. Routine request logged."
}
GET /api/v1/conversation-history
This endpoint retrieves the conversation history for the IP address making the request.
It is useful for reviewing the AI's past analyses and actions.
- Response:
- A JSON array containing the conversation history.
Example Response:
[
{ "role": "system", "content": "You are a security AI..." },
{ "role": "user", "content": "127.0.0.1 - - [12/Mar/2024:10:12:33 +0000] ..." },
{ "role": "assistant", "content": "GENERAL: No suspicious activity detected..." }
]
POST /api/v1/restart-core
This endpoint restarts the core AI service running in a Docker container.
This is useful if the AI service becomes unresponsive or needs to refresh its state.
- Response:
- A JSON object with the output of the Docker restart command.
Example Request:
curl -X POST http://localhost:3001/api/v1/restart-core
Example Response:
{
"stdout": "llama-gpu-server\n"
}
POST /api/v1/reset-conversation
This endpoint resets the conversation history for the requesting IP address, effectively starting a new session with the AI.
This can be useful for clearing outdated context and beginning a fresh analysis.
- Response:
- A JSON object confirming the reset action.
Example Request:
curl -X POST http://localhost:3001/api/v1/reset-conversation
Example Response:
{
"message": "Conversation history reset for IP: 127.0.0.1"
}
Logging
Logging is a critical component of the AI Log Monitoring System, providing insights into system operations, debugging information, and records of security actions.
Log Levels
The system categorizes logs into different levels to help you quickly identify the nature and severity of messages:
- INFO: General information about the system's operations.
- WARN: Indications of potential issues that may require attention.
- ERROR: Logs generated when an error occurs during processing.
- SUCCESS: Messages that indicate successful operations or actions.
- DEBUG: Detailed messages intended for debugging purposes, enabled when
DEBUG=true.
Log Structure
Each log message includes a timestamp, log level, and message content. For example:
2024-03-12 10:12:33 [INFO] Starting to read log from: /dockerData/logs/access.log
2024-03-12 10:12:35 [DEBUG] Read line: 127.0.0.1 - - [12/Mar/2024:10:12:33 +0000] "GET /index.html HTTP/1.1" 200 3050
2024-03-12 10:12:35 [SUCCESS] Log buffer sent to backend successfully.
Debugging
When DEBUG=true, the system provides detailed logs that include every step of the processing workflow.
This includes reading log lines, checking for ignored IPs, sending data to the backend, and receiving responses from the AI.
These logs are invaluable during development and troubleshooting, as they offer full visibility into the system's inner workings.
Security Considerations
Security is a paramount concern when monitoring logs and responding to potential threats.
The AI Log Monitoring System includes several mechanisms to enhance security and minimize false positives.
IP Whitelisting
The system allows you to specify IP addresses and subnets that should be ignored during analysis.
This is particularly useful for avoiding alerts from known and trusted sources, such as public DNS servers or internal IP ranges.
Rate Limiting and Banning
To protect your infrastructure from repeated attacks, the system can automatically ban IP addresses identified as malicious.
The banning process is executed via shell commands, and the system includes a delay mechanism to prevent overloading the network with too many ban requests in a short period.
Data Privacy
All sensitive data, such as IP addresses and conversation history, is handled securely.
The system ensures that only necessary data is stored and processed, with an emphasis on minimizing exposure to potential vulnerabilities.
Performance Optimization
The AI Log Monitoring System is designed to be efficient and scalable, handling large volumes of log data with minimal overhead.
However, some optimizations can further enhance performance, especially in high-traffic environments.
Managing Token Limits
By carefully managing the number of tokens in the AI's conversation history, the system prevents memory overuse and ensures faster response times. The MAX_TOKENS and TOLERANCE variables allow you to fine-tune this behavior.
Efficient Log Parsing
The system uses regular expressions and filtering techniques to parse and analyze log files efficiently. By focusing only on relevant log entries and ignoring unnecessary ones, the system reduces processing time and improves accuracy.
Troubleshooting
If you encounter issues while using the AI Log Monitoring System, this section provides guidance on common problems and how to resolve them.
Common Issues
- No Logs Detected: Ensure that the log directory is correctly specified and that log files are present.
- AI Service Unresponsive: Restart the AI service using the
/api/v1/restart-coreendpoint. - Excessive False Positives: Review the ignored IPs and subnets to ensure that known safe traffic is excluded.
Restarting Services
If the AI service becomes unresponsive or you need to apply changes, use the /api/v1/restart-core endpoint to restart the core AI Docker container.
This refreshes the model and clears any stale states.
Log Analysis
Review the logs generated by the system to identify potential issues. Focus on ERROR and WARN level messages to spot critical problems quickly. Use the DEBUG logs for deeper investigation during development or when troubleshooting specific issues.
Customization
The AI Log Monitoring System is highly customizable, allowing you to tailor its behavior to your specific needs.
Modifying the AI Prompt
The AI's behavior is guided by a custom prompt that defines how it should interpret log data and what actions it should take.
You can modify this prompt in ai_log_backend.js to adjust the AI's focus or add new rules.
Example Customization:
const prompt = `
You are a security AI responsible for analyzing web traffic...
- Ignore any IPs within the range 10.0.0.0/8.
- Flag any requests to /admin as potential threats.
- Use emojis to convey the severity of alerts.
`;
Adjusting Buffer Limits
The log buffer size determines how many log lines are collected before they are sent to the backend.
Adjust this size to balance network usage and processing frequency.
Example Customization:
const LOG_BUFFER_LIMIT = 30; // Increase the buffer size to 30 lines
Extending Log Parsing Capabilities
You can extend the log parsing functionality by adding new regular expressions or parsing logic to handle different log formats or detect new types of threats.
Example Customization:
tail.on('line', async (line) => {
// Add custom logic to detect SQL injection attempts
if (/SELECT.*FROM/i.test(line)) {
log.warn(`Potential SQL injection detected: ${line}`);
}
});