snxraven/ai-nginx-log-security
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

format update

Browse Source
This commit is contained in:
Raven 2024-08-09 04:38:42 -04:00
parent 17a99d9ecf
commit bafa863c33
1 changed files with 63 additions and 25 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

88
README.md
View File

@ -160,7 +160,9 @@ This ensures that traffic from these sources is not flagged or acted upon, reduc
### Running `ai_log.js`
The `ai_log.js` script is responsible for continuously monitoring NGINX logs, buffering log entries, and sending them to the backend for analysis. It also handles real-time actions, such as banning IP addresses and sending alerts.
The `ai_log.js` script is responsible for continuously monitoring NGINX logs, buffering log entries, and sending them to the backend for analysis.
It also handles real-time actions, such as banning IP addresses and sending alerts.
To start the log monitoring process, execute the following command:
@ -168,11 +170,15 @@ To start the log monitoring process, execute the following command:
node ai_log.js
```
The script will immediately begin reading logs from the specified directory and processing them according to the rules defined in the script. The logs will be buffered and periodically sent to the backend for AI-based analysis.
The script will immediately begin reading logs from the specified directory and processing them according to the rules defined in the script.
The logs will be buffered and periodically sent to the backend for AI-based analysis.
### Running `ai_log_backend.js`
The `ai_log_backend.js` script sets up an Express server that interfaces with the AI model to analyze log data. It also manages conversation history and provides endpoints for interacting with the system.
The `ai_log_backend.js` script sets up an Express server that interfaces with the AI model to analyze log data.
It also manages conversation history and provides endpoints for interacting with the system.
To start the backend server:
@ -180,13 +186,17 @@ To start the backend server:
node ai_log_backend.js
```
By default, the server will be running on `http://localhost:3001`. This server handles incoming log data, processes it with the AI model, and returns actionable insights, including potential security alerts.
By default, the server will be running on `http://localhost:3001`.
This server handles incoming log data, processes it with the AI model, and returns actionable insights, including potential security alerts.
## How It Works
### Log Monitoring and Buffering
The `ai_log.js` script uses the `Tail` module to monitor NGINX log files in real-time. As new lines are added to the logs, the script reads and buffers them. The buffer size is configurable, allowing the system to batch-process logs before sending them to the backend.
The `ai_log.js` script uses the `Tail` module to monitor NGINX log files in real-time.
As new lines are added to the logs, the script reads and buffers them. The buffer size is configurable, allowing the system to batch-process logs before sending them to the backend.
**Key Features**:
@ -196,9 +206,9 @@ The `ai_log.js` script uses the `Tail` module to monitor NGINX log files in real
### Sending Logs to Backend
When the log buffer reaches a predefined size or a set time interval elapses, the
buffered logs are sent to the backend for AI processing. The backend analyzes the logs to detect potential security threats, generate alerts, and manage conversation history.
When the log buffer reaches a predefined size or a set time interval elapses, the buffered logs are sent to the backend for AI processing.
The backend analyzes the logs to detect potential security threats, generate alerts, and manage conversation history.
**Process Overview**:
@ -208,7 +218,9 @@ When the log buffer reaches a predefined size or a set time interval elapses, th
### AI-Based Log Analysis
The backend server (`ai_log_backend.js`) leverages an AI model (e.g., GPT) to analyze the logs and detect potential security threats. The AI operates based on a custom prompt that instructs it on how to interpret the logs, which IPs to ignore, and what actions to take.
The backend server (`ai_log_backend.js`) leverages an AI model (e.g., GPT) to analyze the logs and detect potential security threats.
The AI operates based on a custom prompt that instructs it on how to interpret the logs, which IPs to ignore, and what actions to take.
**AI Model Usage**:
@ -218,7 +230,9 @@ The backend server (`ai_log_backend.js`) leverages an AI model (e.g., GPT) to an
### Token Management and Conversation History
To ensure efficient operation and prevent resource exhaustion, the system carefully manages the number of tokens used in AI conversations. Token management involves trimming older parts of the conversation history to stay within predefined limits.
To ensure efficient operation and prevent resource exhaustion, the system carefully manages the number of tokens used in AI conversations.
Token management involves trimming older parts of the conversation history to stay within predefined limits.
**Token Management Strategies**:
@ -228,7 +242,9 @@ To ensure efficient operation and prevent resource exhaustion, the system carefu
### Security Alert Handling
When the AI detects a potential security threat, it generates an alert. These alerts are processed by the backend and can trigger actions like banning an IP address or sending a notification to a Discord channel.
When the AI detects a potential security threat, it generates an alert.
These alerts are processed by the backend and can trigger actions like banning an IP address or sending a notification to a Discord channel.
**Alert Workflow**:
@ -239,7 +255,9 @@ When the AI detects a potential security threat, it generates an alert. These al
### Discord Integration
The system integrates with Discord to send alerts and notifications. This is particularly useful for real-time monitoring, allowing administrators to receive and act on security alerts instantly.
The system integrates with Discord to send alerts and notifications.
This is particularly useful for real-time monitoring, allowing administrators to receive and act on security alerts instantly.
**Integration Details**:
@ -280,7 +298,9 @@ This endpoint processes incoming NGINX logs by sending them to the AI model for
### GET /api/v1/conversation-history
This endpoint retrieves the conversation history for the IP address making the request. It is useful for reviewing the AI's past analyses and actions.
This endpoint retrieves the conversation history for the IP address making the request.
It is useful for reviewing the AI's past analyses and actions.
- **Response**:
- A JSON array containing the conversation history.
@ -297,7 +317,9 @@ This endpoint retrieves the conversation history for the IP address making the r
### POST /api/v1/restart-core
This endpoint restarts the core AI service running in a Docker container. This is useful if the AI service becomes unresponsive or needs to refresh its state.
This endpoint restarts the core AI service running in a Docker container.
This is useful if the AI service becomes unresponsive or needs to refresh its state.
- **Response**:
- A JSON object with the output of the Docker restart command.
@ -318,7 +340,9 @@ curl -X POST http://localhost:3001/api/v1/restart-core
### POST /api/v1/reset-conversation
This endpoint resets the conversation history for the requesting IP address, effectively starting a new session with the AI. This can be useful for clearing outdated context and beginning a fresh analysis.
This endpoint resets the conversation history for the requesting IP address, effectively starting a new session with the AI.
This can be useful for clearing outdated context and beginning a fresh analysis.
- **Response**:
- A JSON object confirming the reset action.
@ -363,25 +387,35 @@ Each log message includes a timestamp, log level, and message content. For examp
### Debugging
When `DEBUG=true`, the system provides detailed logs that include every step of the processing workflow. This includes reading log lines, checking for ignored IPs, sending data to the backend, and receiving responses from the AI.
When `DEBUG=true`, the system provides detailed logs that include every step of the processing workflow.
This includes reading log lines, checking for ignored IPs, sending data to the backend, and receiving responses from the AI.
These logs are invaluable during development and troubleshooting, as they offer full visibility into the system's inner workings.
## Security Considerations
Security is a paramount concern when monitoring logs and responding to potential threats. The AI Log Monitoring System includes several mechanisms to enhance security and minimize false positives.
Security is a paramount concern when monitoring logs and responding to potential threats.
The AI Log Monitoring System includes several mechanisms to enhance security and minimize false positives.
### IP Whitelisting
The system allows you to specify IP addresses and subnets that should be ignored during analysis. This is particularly useful for avoiding alerts from known and trusted sources, such as public DNS servers or internal IP ranges.
The system allows you to specify IP addresses and subnets that should be ignored during analysis.
This is particularly useful for avoiding alerts from known and trusted sources, such as public DNS servers or internal IP ranges.
### Rate Limiting and Banning
To protect your infrastructure from repeated attacks, the system can automatically ban IP addresses identified as malicious. The banning process is executed via shell commands, and the system includes a delay mechanism to prevent overloading the network with too many ban requests in a short period.
To protect your infrastructure from repeated attacks, the system can automatically ban IP addresses identified as malicious.
The banning process is executed via shell commands, and the system includes a delay mechanism to prevent overloading the network with too many ban requests in a short period.
### Data Privacy
All sensitive data, such as IP addresses and conversation history, is handled securely. The system ensures that only necessary data is stored and processed, with an emphasis on minimizing exposure to potential vulnerabilities.
All sensitive data, such as IP addresses and conversation history, is handled securely.
The system ensures that only necessary data is stored and processed, with an emphasis on minimizing exposure to potential vulnerabilities.
## Performance Optimization
@ -415,13 +449,13 @@ Review the logs generated by the system to identify potential issues. Focus on `
## Customization
The AI Log Monitoring System is highly customizable, allowing you to tailor its behavior to your specific needs.
### Modifying the AI Prompt
The AI's behavior is guided by a custom prompt that defines how it should interpret log data and what actions it should take. You can modify this prompt in `ai_log_backend.js` to adjust the AI's focus or add new rules.
The AI's behavior is guided by a custom prompt that defines how it should interpret log data and what actions it should take.
You can modify this prompt in `ai_log_backend.js` to adjust the AI's focus or add new rules.
**Example Customization**:
@ -436,7 +470,9 @@ You are a security AI responsible for analyzing web traffic...
### Adjusting Buffer Limits
The log buffer size determines how many log lines are collected before they are sent to the backend. Adjust this size to balance network usage and processing frequency.
The log buffer size determines how many log lines are collected before they are sent to the backend.
Adjust this size to balance network usage and processing frequency.
**Example Customization**:
@ -461,4 +497,6 @@ tail.on('line', async (line) => {
## Contributing
Contributions are welcome! To contribute, fork the repository, create a new branch for your changes, and submit a pull request. Please ensure that your code adheres to the existing style and that you include tests for any new features.
Contributions are welcome! To contribute, fork the repository, create a new branch for your changes, and submit a pull request.
Please ensure that your code adheres to the existing style and that you include tests for any new features.