format update
This commit is contained in:
parent
17a99d9ecf
commit
bafa863c33
88
README.md
88
README.md
@ -160,7 +160,9 @@ This ensures that traffic from these sources is not flagged or acted upon, reduc
|
|||||||
|
|
||||||
### Running `ai_log.js`
|
### Running `ai_log.js`
|
||||||
|
|
||||||
The `ai_log.js` script is responsible for continuously monitoring NGINX logs, buffering log entries, and sending them to the backend for analysis. It also handles real-time actions, such as banning IP addresses and sending alerts.
|
The `ai_log.js` script is responsible for continuously monitoring NGINX logs, buffering log entries, and sending them to the backend for analysis.
|
||||||
|
|
||||||
|
It also handles real-time actions, such as banning IP addresses and sending alerts.
|
||||||
|
|
||||||
To start the log monitoring process, execute the following command:
|
To start the log monitoring process, execute the following command:
|
||||||
|
|
||||||
@ -168,11 +170,15 @@ To start the log monitoring process, execute the following command:
|
|||||||
node ai_log.js
|
node ai_log.js
|
||||||
```
|
```
|
||||||
|
|
||||||
The script will immediately begin reading logs from the specified directory and processing them according to the rules defined in the script. The logs will be buffered and periodically sent to the backend for AI-based analysis.
|
The script will immediately begin reading logs from the specified directory and processing them according to the rules defined in the script.
|
||||||
|
|
||||||
|
The logs will be buffered and periodically sent to the backend for AI-based analysis.
|
||||||
|
|
||||||
### Running `ai_log_backend.js`
|
### Running `ai_log_backend.js`
|
||||||
|
|
||||||
The `ai_log_backend.js` script sets up an Express server that interfaces with the AI model to analyze log data. It also manages conversation history and provides endpoints for interacting with the system.
|
The `ai_log_backend.js` script sets up an Express server that interfaces with the AI model to analyze log data.
|
||||||
|
|
||||||
|
It also manages conversation history and provides endpoints for interacting with the system.
|
||||||
|
|
||||||
To start the backend server:
|
To start the backend server:
|
||||||
|
|
||||||
@ -180,13 +186,17 @@ To start the backend server:
|
|||||||
node ai_log_backend.js
|
node ai_log_backend.js
|
||||||
```
|
```
|
||||||
|
|
||||||
By default, the server will be running on `http://localhost:3001`. This server handles incoming log data, processes it with the AI model, and returns actionable insights, including potential security alerts.
|
By default, the server will be running on `http://localhost:3001`.
|
||||||
|
|
||||||
|
This server handles incoming log data, processes it with the AI model, and returns actionable insights, including potential security alerts.
|
||||||
|
|
||||||
## How It Works
|
## How It Works
|
||||||
|
|
||||||
### Log Monitoring and Buffering
|
### Log Monitoring and Buffering
|
||||||
|
|
||||||
The `ai_log.js` script uses the `Tail` module to monitor NGINX log files in real-time. As new lines are added to the logs, the script reads and buffers them. The buffer size is configurable, allowing the system to batch-process logs before sending them to the backend.
|
The `ai_log.js` script uses the `Tail` module to monitor NGINX log files in real-time.
|
||||||
|
|
||||||
|
As new lines are added to the logs, the script reads and buffers them. The buffer size is configurable, allowing the system to batch-process logs before sending them to the backend.
|
||||||
|
|
||||||
**Key Features**:
|
**Key Features**:
|
||||||
|
|
||||||
@ -196,9 +206,9 @@ The `ai_log.js` script uses the `Tail` module to monitor NGINX log files in real
|
|||||||
|
|
||||||
### Sending Logs to Backend
|
### Sending Logs to Backend
|
||||||
|
|
||||||
When the log buffer reaches a predefined size or a set time interval elapses, the
|
When the log buffer reaches a predefined size or a set time interval elapses, the buffered logs are sent to the backend for AI processing.
|
||||||
|
|
||||||
buffered logs are sent to the backend for AI processing. The backend analyzes the logs to detect potential security threats, generate alerts, and manage conversation history.
|
The backend analyzes the logs to detect potential security threats, generate alerts, and manage conversation history.
|
||||||
|
|
||||||
**Process Overview**:
|
**Process Overview**:
|
||||||
|
|
||||||
@ -208,7 +218,9 @@ When the log buffer reaches a predefined size or a set time interval elapses, th
|
|||||||
|
|
||||||
### AI-Based Log Analysis
|
### AI-Based Log Analysis
|
||||||
|
|
||||||
The backend server (`ai_log_backend.js`) leverages an AI model (e.g., GPT) to analyze the logs and detect potential security threats. The AI operates based on a custom prompt that instructs it on how to interpret the logs, which IPs to ignore, and what actions to take.
|
The backend server (`ai_log_backend.js`) leverages an AI model (e.g., GPT) to analyze the logs and detect potential security threats.
|
||||||
|
|
||||||
|
The AI operates based on a custom prompt that instructs it on how to interpret the logs, which IPs to ignore, and what actions to take.
|
||||||
|
|
||||||
**AI Model Usage**:
|
**AI Model Usage**:
|
||||||
|
|
||||||
@ -218,7 +230,9 @@ The backend server (`ai_log_backend.js`) leverages an AI model (e.g., GPT) to an
|
|||||||
|
|
||||||
### Token Management and Conversation History
|
### Token Management and Conversation History
|
||||||
|
|
||||||
To ensure efficient operation and prevent resource exhaustion, the system carefully manages the number of tokens used in AI conversations. Token management involves trimming older parts of the conversation history to stay within predefined limits.
|
To ensure efficient operation and prevent resource exhaustion, the system carefully manages the number of tokens used in AI conversations.
|
||||||
|
|
||||||
|
Token management involves trimming older parts of the conversation history to stay within predefined limits.
|
||||||
|
|
||||||
**Token Management Strategies**:
|
**Token Management Strategies**:
|
||||||
|
|
||||||
@ -228,7 +242,9 @@ To ensure efficient operation and prevent resource exhaustion, the system carefu
|
|||||||
|
|
||||||
### Security Alert Handling
|
### Security Alert Handling
|
||||||
|
|
||||||
When the AI detects a potential security threat, it generates an alert. These alerts are processed by the backend and can trigger actions like banning an IP address or sending a notification to a Discord channel.
|
When the AI detects a potential security threat, it generates an alert.
|
||||||
|
|
||||||
|
These alerts are processed by the backend and can trigger actions like banning an IP address or sending a notification to a Discord channel.
|
||||||
|
|
||||||
**Alert Workflow**:
|
**Alert Workflow**:
|
||||||
|
|
||||||
@ -239,7 +255,9 @@ When the AI detects a potential security threat, it generates an alert. These al
|
|||||||
|
|
||||||
### Discord Integration
|
### Discord Integration
|
||||||
|
|
||||||
The system integrates with Discord to send alerts and notifications. This is particularly useful for real-time monitoring, allowing administrators to receive and act on security alerts instantly.
|
The system integrates with Discord to send alerts and notifications.
|
||||||
|
|
||||||
|
This is particularly useful for real-time monitoring, allowing administrators to receive and act on security alerts instantly.
|
||||||
|
|
||||||
**Integration Details**:
|
**Integration Details**:
|
||||||
|
|
||||||
@ -280,7 +298,9 @@ This endpoint processes incoming NGINX logs by sending them to the AI model for
|
|||||||
|
|
||||||
### GET /api/v1/conversation-history
|
### GET /api/v1/conversation-history
|
||||||
|
|
||||||
This endpoint retrieves the conversation history for the IP address making the request. It is useful for reviewing the AI's past analyses and actions.
|
This endpoint retrieves the conversation history for the IP address making the request.
|
||||||
|
|
||||||
|
It is useful for reviewing the AI's past analyses and actions.
|
||||||
|
|
||||||
- **Response**:
|
- **Response**:
|
||||||
- A JSON array containing the conversation history.
|
- A JSON array containing the conversation history.
|
||||||
@ -297,7 +317,9 @@ This endpoint retrieves the conversation history for the IP address making the r
|
|||||||
|
|
||||||
### POST /api/v1/restart-core
|
### POST /api/v1/restart-core
|
||||||
|
|
||||||
This endpoint restarts the core AI service running in a Docker container. This is useful if the AI service becomes unresponsive or needs to refresh its state.
|
This endpoint restarts the core AI service running in a Docker container.
|
||||||
|
|
||||||
|
This is useful if the AI service becomes unresponsive or needs to refresh its state.
|
||||||
|
|
||||||
- **Response**:
|
- **Response**:
|
||||||
- A JSON object with the output of the Docker restart command.
|
- A JSON object with the output of the Docker restart command.
|
||||||
@ -318,7 +340,9 @@ curl -X POST http://localhost:3001/api/v1/restart-core
|
|||||||
|
|
||||||
### POST /api/v1/reset-conversation
|
### POST /api/v1/reset-conversation
|
||||||
|
|
||||||
This endpoint resets the conversation history for the requesting IP address, effectively starting a new session with the AI. This can be useful for clearing outdated context and beginning a fresh analysis.
|
This endpoint resets the conversation history for the requesting IP address, effectively starting a new session with the AI.
|
||||||
|
|
||||||
|
This can be useful for clearing outdated context and beginning a fresh analysis.
|
||||||
|
|
||||||
- **Response**:
|
- **Response**:
|
||||||
- A JSON object confirming the reset action.
|
- A JSON object confirming the reset action.
|
||||||
@ -363,25 +387,35 @@ Each log message includes a timestamp, log level, and message content. For examp
|
|||||||
|
|
||||||
### Debugging
|
### Debugging
|
||||||
|
|
||||||
When `DEBUG=true`, the system provides detailed logs that include every step of the processing workflow. This includes reading log lines, checking for ignored IPs, sending data to the backend, and receiving responses from the AI.
|
When `DEBUG=true`, the system provides detailed logs that include every step of the processing workflow.
|
||||||
|
|
||||||
|
This includes reading log lines, checking for ignored IPs, sending data to the backend, and receiving responses from the AI.
|
||||||
|
|
||||||
These logs are invaluable during development and troubleshooting, as they offer full visibility into the system's inner workings.
|
These logs are invaluable during development and troubleshooting, as they offer full visibility into the system's inner workings.
|
||||||
|
|
||||||
## Security Considerations
|
## Security Considerations
|
||||||
|
|
||||||
Security is a paramount concern when monitoring logs and responding to potential threats. The AI Log Monitoring System includes several mechanisms to enhance security and minimize false positives.
|
Security is a paramount concern when monitoring logs and responding to potential threats.
|
||||||
|
|
||||||
|
The AI Log Monitoring System includes several mechanisms to enhance security and minimize false positives.
|
||||||
|
|
||||||
### IP Whitelisting
|
### IP Whitelisting
|
||||||
|
|
||||||
The system allows you to specify IP addresses and subnets that should be ignored during analysis. This is particularly useful for avoiding alerts from known and trusted sources, such as public DNS servers or internal IP ranges.
|
The system allows you to specify IP addresses and subnets that should be ignored during analysis.
|
||||||
|
|
||||||
|
This is particularly useful for avoiding alerts from known and trusted sources, such as public DNS servers or internal IP ranges.
|
||||||
|
|
||||||
### Rate Limiting and Banning
|
### Rate Limiting and Banning
|
||||||
|
|
||||||
To protect your infrastructure from repeated attacks, the system can automatically ban IP addresses identified as malicious. The banning process is executed via shell commands, and the system includes a delay mechanism to prevent overloading the network with too many ban requests in a short period.
|
To protect your infrastructure from repeated attacks, the system can automatically ban IP addresses identified as malicious.
|
||||||
|
|
||||||
|
The banning process is executed via shell commands, and the system includes a delay mechanism to prevent overloading the network with too many ban requests in a short period.
|
||||||
|
|
||||||
### Data Privacy
|
### Data Privacy
|
||||||
|
|
||||||
All sensitive data, such as IP addresses and conversation history, is handled securely. The system ensures that only necessary data is stored and processed, with an emphasis on minimizing exposure to potential vulnerabilities.
|
All sensitive data, such as IP addresses and conversation history, is handled securely.
|
||||||
|
|
||||||
|
The system ensures that only necessary data is stored and processed, with an emphasis on minimizing exposure to potential vulnerabilities.
|
||||||
|
|
||||||
## Performance Optimization
|
## Performance Optimization
|
||||||
|
|
||||||
@ -415,13 +449,13 @@ Review the logs generated by the system to identify potential issues. Focus on `
|
|||||||
|
|
||||||
## Customization
|
## Customization
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
The AI Log Monitoring System is highly customizable, allowing you to tailor its behavior to your specific needs.
|
The AI Log Monitoring System is highly customizable, allowing you to tailor its behavior to your specific needs.
|
||||||
|
|
||||||
### Modifying the AI Prompt
|
### Modifying the AI Prompt
|
||||||
|
|
||||||
The AI's behavior is guided by a custom prompt that defines how it should interpret log data and what actions it should take. You can modify this prompt in `ai_log_backend.js` to adjust the AI's focus or add new rules.
|
The AI's behavior is guided by a custom prompt that defines how it should interpret log data and what actions it should take.
|
||||||
|
|
||||||
|
You can modify this prompt in `ai_log_backend.js` to adjust the AI's focus or add new rules.
|
||||||
|
|
||||||
**Example Customization**:
|
**Example Customization**:
|
||||||
|
|
||||||
@ -436,7 +470,9 @@ You are a security AI responsible for analyzing web traffic...
|
|||||||
|
|
||||||
### Adjusting Buffer Limits
|
### Adjusting Buffer Limits
|
||||||
|
|
||||||
The log buffer size determines how many log lines are collected before they are sent to the backend. Adjust this size to balance network usage and processing frequency.
|
The log buffer size determines how many log lines are collected before they are sent to the backend.
|
||||||
|
|
||||||
|
Adjust this size to balance network usage and processing frequency.
|
||||||
|
|
||||||
**Example Customization**:
|
**Example Customization**:
|
||||||
|
|
||||||
@ -461,4 +497,6 @@ tail.on('line', async (line) => {
|
|||||||
|
|
||||||
## Contributing
|
## Contributing
|
||||||
|
|
||||||
Contributions are welcome! To contribute, fork the repository, create a new branch for your changes, and submit a pull request. Please ensure that your code adheres to the existing style and that you include tests for any new features.
|
Contributions are welcome! To contribute, fork the repository, create a new branch for your changes, and submit a pull request.
|
||||||
|
|
||||||
|
Please ensure that your code adheres to the existing style and that you include tests for any new features.
|
||||||
|
Loading…
Reference in New Issue
Block a user