Adding client side + new readme
This commit is contained in:
parent
7e75ea80a6
commit
e35919c004
488
README.md
488
README.md
@ -1,16 +1,6 @@
|
|||||||
# AI NGINX Log Analysis
|
# AI Log Monitoring System
|
||||||
## Overview
|
|
||||||
|
|
||||||
This project is an Express.js-based server designed to interact with models via llama-cpp-python[web] emulation server to analyze NGINX logs for potential security threats. The server processes incoming requests, analyzes the content, and provides responses that include alerts or general insights. The server can also scrape web pages and manage conversation histories per IP address.
|
This repository contains two interdependent Node.js applications, `ai_log.js` and `ai_log_backend.js`. These applications collaborate to monitor NGINX logs, detect potential security threats, and manage conversation history for AI-based log analysis. This README provides a comprehensive guide for setting up, configuring, and operating these applications, along with detailed explanations of the underlying mechanisms and customizable features.
|
||||||
|
|
||||||
## Features
|
|
||||||
|
|
||||||
- **NGINX Log Analysis**: Analyzes web traffic logs to identify potential security threats and generate appropriate responses.
|
|
||||||
- **Conversation History**: Maintains a history of interactions for each client IP, allowing for context-aware responses.
|
|
||||||
- **Web Scraping**: Scrapes web pages to extract and format relevant information.
|
|
||||||
- **Token Management**: Limits the number of tokens used in the conversation to ensure responses are within the model's limits.
|
|
||||||
- **Core Service Management**: Provides endpoints to restart the core GPT service and reset conversation histories.
|
|
||||||
- **Cross-Origin Resource Sharing (CORS)**: Enabled for all routes, allowing for flexible API usage.
|
|
||||||
|
|
||||||
## Screenshots
|
## Screenshots
|
||||||
|
|
||||||
@ -24,81 +14,451 @@ Here are some screenshots of the application in action:
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## Table of Contents
|
||||||
|
|
||||||
|
- [Introduction](#introduction)
|
||||||
|
- [Prerequisites](#prerequisites)
|
||||||
|
- [Installation](#installation)
|
||||||
|
- [Configuration](#configuration)
|
||||||
|
- [Environment Variables](#environment-variables)
|
||||||
|
- [Directory Structure](#directory-structure)
|
||||||
|
- [Ignored IPs and Subnets](#ignored-ips-and-subnets)
|
||||||
|
- [Usage](#usage)
|
||||||
|
- [Running `ai_log.js`](#running-ai_logjs)
|
||||||
|
- [Running `ai_log_backend.js`](#running-ai_log_backendjs)
|
||||||
|
- [How It Works](#how-it-works)
|
||||||
|
- [Log Monitoring and Buffering](#log-monitoring-and-buffering)
|
||||||
|
- [Sending Logs to Backend](#sending-logs-to-backend)
|
||||||
|
- [AI-Based Log Analysis](#ai-based-log-analysis)
|
||||||
|
- [Token Management and Conversation History](#token-management-and-conversation-history)
|
||||||
|
- [Security Alert Handling](#security-alert-handling)
|
||||||
|
- [Discord Integration](#discord-integration)
|
||||||
|
- [API Endpoints](#api-endpoints)
|
||||||
|
- [POST /api/v1/chat](#post-apiv1chat)
|
||||||
|
- [GET /api/v1/conversation-history](#get-apiv1conversation-history)
|
||||||
|
- [POST /api/v1/restart-core](#post-apiv1restart-core)
|
||||||
|
- [POST /api/v1/reset-conversation](#post-apiv1reset-conversation)
|
||||||
|
- [Logging](#logging)
|
||||||
|
- [Log Levels](#log-levels)
|
||||||
|
- [Log Structure](#log-structure)
|
||||||
|
- [Debugging](#debugging)
|
||||||
|
- [Security Considerations](#security-considerations)
|
||||||
|
- [IP Whitelisting](#ip-whitelisting)
|
||||||
|
- [Rate Limiting and Banning](#rate-limiting-and-banning)
|
||||||
|
- [Data Privacy](#data-privacy)
|
||||||
|
- [Performance Optimization](#performance-optimization)
|
||||||
|
- [Managing Token Limits](#managing-token-limits)
|
||||||
|
- [Efficient Log Parsing](#efficient-log-parsing)
|
||||||
|
- [Troubleshooting](#troubleshooting)
|
||||||
|
- [Common Issues](#common-issues)
|
||||||
|
- [Restarting Services](#restarting-services)
|
||||||
|
- [Log Analysis](#log-analysis)
|
||||||
|
- [Customization](#customization)
|
||||||
|
- [Modifying the AI Prompt](#modifying-the-ai-prompt)
|
||||||
|
- [Adjusting Buffer Limits](#adjusting-buffer-limits)
|
||||||
|
- [Extending Log Parsing Capabilities](#extending-log-parsing-capabilities)
|
||||||
|
- [Contributing](#contributing)
|
||||||
|
- [License](#license)
|
||||||
|
|
||||||
|
## Introduction
|
||||||
|
|
||||||
|
The AI Log Monitoring System is a powerful and extensible solution designed to enhance security monitoring by integrating AI-based analysis into traditional NGINX log management. By continuously tailing log files and utilizing an AI model to analyze log entries in real-time, this system automates the detection of security threats, sends alerts to designated channels (like Discord), and manages conversation history for ongoing log analysis.
|
||||||
|
|
||||||
|
The solution is composed of two primary scripts:
|
||||||
|
|
||||||
|
- `ai_log.js`: Handles log monitoring, buffering, and sending logs to a backend for AI processing.
|
||||||
|
- `ai_log_backend.js`: Manages AI-based analysis, conversation history, and exposes an API for interacting with the AI service.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
Before setting up and running the AI Log Monitoring System, ensure you have the following components installed and configured on your machine:
|
||||||
|
|
||||||
|
- **Node.js**: Version 14.x or higher is required to run the JavaScript code.
|
||||||
|
- **npm**: Version 6.x or higher is needed to manage project dependencies.
|
||||||
|
- **Docker**: Required for running the AI model, particularly if using a containerized GPT model for processing.
|
||||||
|
- **NGINX**: The web server generating logs that the system will monitor.
|
||||||
|
- **Discord**: A Discord webhook URL is necessary for sending security alerts.
|
||||||
|
|
||||||
|
Ensure you have administrative privileges on the machine to install and configure these dependencies.
|
||||||
|
|
||||||
## Installation
|
## Installation
|
||||||
|
|
||||||
**Note**: A `llama-cpp-python` OpenAI Emulation Server is required alongside the backend server. The server in this code is configured to run on `127.0.0.1:8003`.
|
Clone the repository to your local machine:
|
||||||
|
|
||||||
1. Clone the repository:
|
```bash
|
||||||
```bash
|
git clone git@git.ssh.surf:snxraven/ai-nginx-log-security.git
|
||||||
git clone git@git.ssh.surf:snxraven/ai-nginx-log-security.git
|
cd ai-log-monitoring
|
||||||
cd ai-nginx-log-security
|
```
|
||||||
```
|
|
||||||
|
|
||||||
2. Install dependencies:
|
Install the required Node.js dependencies for both applications:
|
||||||
```bash
|
|
||||||
npm install
|
|
||||||
```
|
|
||||||
|
|
||||||
3. Create a `.env` file in the project root with the following content:
|
```bash
|
||||||
```bash
|
npm install
|
||||||
MAX_CONTENT_LENGTH=2000
|
```
|
||||||
```
|
|
||||||
|
|
||||||
4. Start the server:
|
## Configuration
|
||||||
```bash
|
|
||||||
ai_log.js
|
|
||||||
```
|
|
||||||
|
|
||||||
## Endpoints
|
### Environment Variables
|
||||||
|
|
||||||
### 1. `/api/v1/chat`
|
Environment variables play a critical role in configuring the behavior of the AI Log Monitoring System. They allow you to customize log directories, adjust token limits, and set up necessary credentials. Create a `.env` file in the root of your project and populate it with the following variables:
|
||||||
- **Method**: `POST`
|
|
||||||
- **Description**: Processes a user message, analyzes it for security threats, and returns a response.
|
|
||||||
- **Request Body**:
|
|
||||||
- `message`: The message to be analyzed.
|
|
||||||
- **Response**: JSON object containing the response from the GPT model.
|
|
||||||
|
|
||||||
### 2. `/api/v1/conversation-history`
|
```bash
|
||||||
- **Method**: `GET`
|
# General Configuration
|
||||||
- **Description**: Retrieves the conversation history for the requesting client's IP.
|
DEBUG=true # Enable detailed logging output for debugging
|
||||||
- **Response**: JSON object containing the conversation history.
|
WEBHOOKURL=https://discord.com/api/webhooks/your-webhook-id # Discord webhook URL for sending alerts
|
||||||
|
MAX_CONTENT_LENGTH=2000 # Maximum length for content scraped from web pages
|
||||||
|
|
||||||
### 3. `/api/v1/restart-core`
|
# AI Service Configuration
|
||||||
- **Method**: `POST`
|
TIMEOUT_DURATION=100000 # Maximum duration (in ms) for API requests to complete
|
||||||
- **Description**: Restarts the core GPT service running in a Docker container.
|
MAX_TOKENS=8000 # Maximum tokens allowed in a conversation before trimming
|
||||||
- **Response**: JSON object confirming the restart or detailing any errors.
|
TOLERANCE=100 # Extra tokens allowed before forcing a trim of conversation history
|
||||||
|
```
|
||||||
|
|
||||||
### 4. `/api/v1/reset-conversation`
|
**Explanation of Environment Variables**:
|
||||||
- **Method**: `POST`
|
|
||||||
- **Description**: Resets the conversation history for the requesting client's IP.
|
|
||||||
- **Response**: JSON object confirming the reset.
|
|
||||||
|
|
||||||
## Environment Variables
|
- **DEBUG**: Enables verbose logging, including debug-level messages. Useful during development and troubleshooting.
|
||||||
|
- **WEBHOOKURL**: The Discord webhook URL where alerts will be sent. This should be a valid and secure URL.
|
||||||
|
- **MAX_CONTENT_LENGTH**: Limits the length of content extracted from web pages to avoid overloading the system or sending excessively long messages.
|
||||||
|
- **TIMEOUT_DURATION**: Sets a timeout for requests to the AI service, preventing the system from hanging indefinitely.
|
||||||
|
- **MAX_TOKENS**: Controls the total number of tokens (words + punctuation) allowed in a conversation with the AI. This prevents the AI from consuming too much memory or processing power.
|
||||||
|
- **TOLERANCE**: A buffer to avoid hitting the MAX_TOKENS limit exactly, ensuring smoother operation by providing a cushion before trimming conversation history.
|
||||||
|
|
||||||
- `MAX_CONTENT_LENGTH`: The maximum length for the content extracted from web pages during scraping (default: 2000 characters).
|
### Directory Structure
|
||||||
|
|
||||||
|
The system monitors log files stored in a specific directory. Ensure this directory exists and is correctly set in the script:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mkdir -p /dockerData/logs
|
||||||
|
```
|
||||||
|
|
||||||
|
If your NGINX logs are stored in a different directory, update the `LOG_DIRECTORY` constant in `ai_log.js` accordingly:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
const LOG_DIRECTORY = '/your/custom/path/to/logs';
|
||||||
|
```
|
||||||
|
|
||||||
|
### Ignored IPs and Subnets
|
||||||
|
|
||||||
|
The system allows you to specify IP addresses and subnets that should be ignored during log monitoring. This is particularly useful for filtering out trusted sources or known harmless traffic (e.g., public DNS servers).
|
||||||
|
|
||||||
|
- **Ignored IPs**: Directly listed IP addresses that should be skipped during processing.
|
||||||
|
- **Ignored Subnets**: Subnets specified in CIDR notation that represent ranges of IP addresses to be ignored.
|
||||||
|
|
||||||
|
To customize the ignored IPs and subnets, modify the `ignoredIPs` and `ignoredSubnets` arrays in `ai_log.js`:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
const ignoredIPs = ['1.1.1.1', '1.0.0.1', '8.8.8.8', '8.8.4.4'];
|
||||||
|
const ignoredSubnets = [
|
||||||
|
'173.245.48.0/20', '103.21.244.0/22', '103.22.200.0/22',
|
||||||
|
// Add more subnets as needed
|
||||||
|
];
|
||||||
|
```
|
||||||
|
|
||||||
|
This ensures that traffic from these sources is not flagged or acted upon, reducing false positives and focusing the AI on more critical threats.
|
||||||
|
|
||||||
## Usage
|
## Usage
|
||||||
|
|
||||||
1. Send a POST request to `/api/v1/chat` with a message to analyze web traffic logs.
|
### Running `ai_log.js`
|
||||||
2. Use the `/api/v1/conversation-history` endpoint to fetch the chat history.
|
|
||||||
3. Restart the core service using the `/api/v1/restart-core` endpoint if needed.
|
The `ai_log.js` script is responsible for continuously monitoring NGINX logs, buffering log entries, and sending them to the backend for analysis. It also handles real-time actions, such as banning IP addresses and sending alerts.
|
||||||
4. Reset conversation history using the `/api/v1/reset-conversation` endpoint.
|
|
||||||
|
To start the log monitoring process, execute the following command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
node ai_log.js
|
||||||
|
```
|
||||||
|
|
||||||
|
The script will immediately begin reading logs from the specified directory and processing them according to the rules defined in the script. The logs will be buffered and periodically sent to the backend for AI-based analysis.
|
||||||
|
|
||||||
|
### Running `ai_log_backend.js`
|
||||||
|
|
||||||
|
The `ai_log_backend.js` script sets up an Express server that interfaces with the AI model to analyze log data. It also manages conversation history and provides endpoints for interacting with the system.
|
||||||
|
|
||||||
|
To start the backend server:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
node ai_log_backend.js
|
||||||
|
```
|
||||||
|
|
||||||
|
By default, the server will be running on `http://localhost:3001`. This server handles incoming log data, processes it with the AI model, and returns actionable insights, including potential security alerts.
|
||||||
|
|
||||||
|
## How It Works
|
||||||
|
|
||||||
|
### Log Monitoring and Buffering
|
||||||
|
|
||||||
|
The `ai_log.js` script uses the `Tail` module to monitor NGINX log files in real-time. As new lines are added to the logs, the script reads and buffers them. The buffer size is configurable, allowing the system to batch-process logs before sending them to the backend.
|
||||||
|
|
||||||
|
**Key Features**:
|
||||||
|
|
||||||
|
- **Real-Time Monitoring**: Continuously monitors specified log files for new entries.
|
||||||
|
- **Buffering**: Collects log entries in a buffer to reduce the frequency of network requests to the backend.
|
||||||
|
- **Ignored Entries**: Filters out log entries from specified IPs and subnets, as well as entries matching certain patterns.
|
||||||
|
|
||||||
|
### Sending Logs to Backend
|
||||||
|
|
||||||
|
When the log buffer reaches a predefined size or a set time interval elapses, the
|
||||||
|
|
||||||
|
buffered logs are sent to the backend for AI processing. The backend analyzes the logs to detect potential security threats, generate alerts, and manage conversation history.
|
||||||
|
|
||||||
|
**Process Overview**:
|
||||||
|
|
||||||
|
1. **Buffer Accumulation**: Logs are collected in a buffer until a threshold is met.
|
||||||
|
2. **Buffer Flush**: The buffer is sent to the backend in one request, optimizing network usage.
|
||||||
|
3. **Backend Analysis**: The AI model analyzes the logs for suspicious activity or patterns.
|
||||||
|
|
||||||
|
### AI-Based Log Analysis
|
||||||
|
|
||||||
|
The backend server (`ai_log_backend.js`) leverages an AI model (e.g., GPT) to analyze the logs and detect potential security threats. The AI operates based on a custom prompt that instructs it on how to interpret the logs, which IPs to ignore, and what actions to take.
|
||||||
|
|
||||||
|
**AI Model Usage**:
|
||||||
|
|
||||||
|
- **Custom Prompt**: The AI is guided by a detailed prompt that defines its behavior and decision-making process.
|
||||||
|
- **Log Parsing**: The AI processes log lines to identify malicious patterns, potential attacks, or other security concerns.
|
||||||
|
- **Actionable Insights**: Based on the analysis, the AI generates alerts, suggests actions (like banning IPs), or provides general observations.
|
||||||
|
|
||||||
|
### Token Management and Conversation History
|
||||||
|
|
||||||
|
To ensure efficient operation and prevent resource exhaustion, the system carefully manages the number of tokens used in AI conversations. Token management involves trimming older parts of the conversation history to stay within predefined limits.
|
||||||
|
|
||||||
|
**Token Management Strategies**:
|
||||||
|
|
||||||
|
- **Counting Tokens**: The system counts tokens for each message in the conversation history.
|
||||||
|
- **Trimming History**: If the token count exceeds the maximum allowed, the oldest messages are removed.
|
||||||
|
- **Tolerance Buffer**: A small buffer is maintained to avoid hitting the exact token limit, ensuring smoother performance.
|
||||||
|
|
||||||
|
### Security Alert Handling
|
||||||
|
|
||||||
|
When the AI detects a potential security threat, it generates an alert. These alerts are processed by the backend and can trigger actions like banning an IP address or sending a notification to a Discord channel.
|
||||||
|
|
||||||
|
**Alert Workflow**:
|
||||||
|
|
||||||
|
1. **Detection**: The AI identifies a suspicious activity or pattern in the logs.
|
||||||
|
2. **Alert Generation**: The AI creates an alert message, formatted for clarity and readability.
|
||||||
|
3. **IP Banning**: If an IP is identified as malicious, the system can execute a ban command to block it.
|
||||||
|
4. **Discord Notification**: Alerts are sent to a designated Discord channel for real-time monitoring and action.
|
||||||
|
|
||||||
|
### Discord Integration
|
||||||
|
|
||||||
|
The system integrates with Discord to send alerts and notifications. This is particularly useful for real-time monitoring, allowing administrators to receive and act on security alerts instantly.
|
||||||
|
|
||||||
|
**Integration Details**:
|
||||||
|
|
||||||
|
- **Webhook-Based Alerts**: The system uses a Discord webhook to send alerts as embedded messages.
|
||||||
|
- **Formatted Messages**: Alerts are formatted with titles, descriptions, and timestamps to ensure they are easy to read and understand.
|
||||||
|
- **IP Banning Alerts**: When an IP is banned, the system includes the IP address in the Discord alert.
|
||||||
|
|
||||||
|
## API Endpoints
|
||||||
|
|
||||||
|
The backend server (`ai_log_backend.js`) exposes several API endpoints for interacting with the AI service, managing conversation history, and controlling the system.
|
||||||
|
|
||||||
|
### POST /api/v1/chat
|
||||||
|
|
||||||
|
This endpoint processes incoming NGINX logs by sending them to the AI model for analysis.
|
||||||
|
|
||||||
|
- **Request Body**:
|
||||||
|
- `message`: A string containing one or more NGINX log lines.
|
||||||
|
|
||||||
|
- **Response**:
|
||||||
|
- A JSON object with the AI's analysis and any detected alerts.
|
||||||
|
|
||||||
|
**Example Request**:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"message": "127.0.0.1 - - [12/Mar/2024:10:12:33 +0000] \"GET /index.html HTTP/1.1\" 200 3050"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Example Response**:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"role": "assistant",
|
||||||
|
"content": "GENERAL: No suspicious activity detected. Routine request logged."
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### GET /api/v1/conversation-history
|
||||||
|
|
||||||
|
This endpoint retrieves the conversation history for the IP address making the request. It is useful for reviewing the AI's past analyses and actions.
|
||||||
|
|
||||||
|
- **Response**:
|
||||||
|
- A JSON array containing the conversation history.
|
||||||
|
|
||||||
|
**Example Response**:
|
||||||
|
|
||||||
|
```json
|
||||||
|
[
|
||||||
|
{ "role": "system", "content": "You are a security AI..." },
|
||||||
|
{ "role": "user", "content": "127.0.0.1 - - [12/Mar/2024:10:12:33 +0000] ..." },
|
||||||
|
{ "role": "assistant", "content": "GENERAL: No suspicious activity detected..." }
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
### POST /api/v1/restart-core
|
||||||
|
|
||||||
|
This endpoint restarts the core AI service running in a Docker container. This is useful if the AI service becomes unresponsive or needs to refresh its state.
|
||||||
|
|
||||||
|
- **Response**:
|
||||||
|
- A JSON object with the output of the Docker restart command.
|
||||||
|
|
||||||
|
**Example Request**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:3001/api/v1/restart-core
|
||||||
|
```
|
||||||
|
|
||||||
|
**Example Response**:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"stdout": "llama-gpu-server\n"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### POST /api/v1/reset-conversation
|
||||||
|
|
||||||
|
This endpoint resets the conversation history for the requesting IP address, effectively starting a new session with the AI. This can be useful for clearing outdated context and beginning a fresh analysis.
|
||||||
|
|
||||||
|
- **Response**:
|
||||||
|
- A JSON object confirming the reset action.
|
||||||
|
|
||||||
|
**Example Request**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:3001/api/v1/reset-conversation
|
||||||
|
```
|
||||||
|
|
||||||
|
**Example Response**:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"message": "Conversation history reset for IP: 127.0.0.1"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
## Logging
|
## Logging
|
||||||
|
|
||||||
The server logs key actions, including incoming requests, conversation history management, and errors. Logs are timestamped and include IP addresses for traceability.
|
Logging is a critical component of the AI Log Monitoring System, providing insights into system operations, debugging information, and records of security actions.
|
||||||
|
|
||||||
## Notes
|
### Log Levels
|
||||||
|
|
||||||
- Ensure the `llama-gpu-server` Docker container is running before starting the Express.js server.
|
The system categorizes logs into different levels to help you quickly identify the nature and severity of messages:
|
||||||
- Conversation history is stored in memory and will be lost when the server restarts. Consider implementing persistent storage if long-term history is required.
|
|
||||||
|
|
||||||
## Contributions
|
- **INFO**: General information about the system's operations.
|
||||||
|
- **WARN**: Indications of potential issues that may require attention.
|
||||||
|
- **ERROR**: Logs generated when an error occurs during processing.
|
||||||
|
- **SUCCESS**: Messages that indicate successful operations or actions.
|
||||||
|
- **DEBUG**: Detailed messages intended for debugging purposes, enabled when `DEBUG=true`.
|
||||||
|
|
||||||
Contributions are welcome! Please fork the repository and submit a pull request with your changes.
|
### Log Structure
|
||||||
|
|
||||||
This project leverages cutting-edge AI to enhance web security analysis, making it easier to identify and respond to threats in real-time.
|
Each log message includes a timestamp, log level, and message content. For example:
|
||||||
|
|
||||||
---
|
```plaintext
|
||||||
|
2024-03-12 10:12:33 [INFO] Starting to read log from: /dockerData/logs/access.log
|
||||||
|
2024-03-12 10:12:35 [DEBUG] Read line: 127.0.0.1 - - [12/Mar/2024:10:12:33 +0000] "GET /index.html HTTP/1.1" 200 3050
|
||||||
|
2024-03-12 10:12:35 [SUCCESS] Log buffer sent to backend successfully.
|
||||||
|
```
|
||||||
|
|
||||||
Let me know if you need any further changes!
|
### Debugging
|
||||||
|
|
||||||
|
When `DEBUG=true`, the system provides detailed logs that include every step of the processing workflow. This includes reading log lines, checking for ignored IPs, sending data to the backend, and receiving responses from the AI.
|
||||||
|
|
||||||
|
These logs are invaluable during development and troubleshooting, as they offer full visibility into the system's inner workings.
|
||||||
|
|
||||||
|
## Security Considerations
|
||||||
|
|
||||||
|
Security is a paramount concern when monitoring logs and responding to potential threats. The AI Log Monitoring System includes several mechanisms to enhance security and minimize false positives.
|
||||||
|
|
||||||
|
### IP Whitelisting
|
||||||
|
|
||||||
|
The system allows you to specify IP addresses and subnets that should be ignored during analysis. This is particularly useful for avoiding alerts from known and trusted sources, such as public DNS servers or internal IP ranges.
|
||||||
|
|
||||||
|
### Rate Limiting and Banning
|
||||||
|
|
||||||
|
To protect your infrastructure from repeated attacks, the system can automatically ban IP addresses identified as malicious. The banning process is executed via shell commands, and the system includes a delay mechanism to prevent overloading the network with too many ban requests in a short period.
|
||||||
|
|
||||||
|
### Data Privacy
|
||||||
|
|
||||||
|
All sensitive data, such as IP addresses and conversation history, is handled securely. The system ensures that only necessary data is stored and processed, with an emphasis on minimizing exposure to potential vulnerabilities.
|
||||||
|
|
||||||
|
## Performance Optimization
|
||||||
|
|
||||||
|
The AI Log Monitoring System is designed to be efficient and scalable, handling large volumes of log data with minimal overhead. However, some optimizations can further enhance performance, especially in high-traffic environments.
|
||||||
|
|
||||||
|
### Managing Token Limits
|
||||||
|
|
||||||
|
By carefully managing the number of tokens in the AI's conversation history, the system prevents memory overuse and ensures faster response times. The `MAX_TOKENS` and `TOLERANCE` variables allow you to fine-tune this behavior.
|
||||||
|
|
||||||
|
### Efficient Log Parsing
|
||||||
|
|
||||||
|
The system uses regular expressions and filtering techniques to parse and analyze log files efficiently. By focusing only on relevant log entries and ignoring unnecessary ones, the system reduces processing time and improves accuracy.
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
If you encounter issues while using the AI Log Monitoring System, this section provides guidance on common problems and how to resolve them.
|
||||||
|
|
||||||
|
### Common Issues
|
||||||
|
|
||||||
|
- **No Logs Detected**: Ensure that the log directory is correctly specified and that log files are present.
|
||||||
|
- **AI Service Unresponsive**: Restart the AI service using the `/api/v1/restart-core` endpoint.
|
||||||
|
- **Excessive False Positives**: Review the ignored IPs and subnets to ensure that known safe traffic is excluded.
|
||||||
|
|
||||||
|
### Restarting Services
|
||||||
|
|
||||||
|
If the AI service becomes unresponsive or you need to apply changes, use the `/api/v1/restart-core` endpoint to restart the core AI Docker container. This refreshes the model and clears any stale states.
|
||||||
|
|
||||||
|
### Log Analysis
|
||||||
|
|
||||||
|
Review the logs generated by the system to identify potential issues. Focus on `ERROR` and `WARN` level messages to spot critical problems quickly. Use the `DEBUG` logs for deeper investigation during development or when troubleshooting specific issues.
|
||||||
|
|
||||||
|
## Customization
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
The AI Log Monitoring System is highly customizable, allowing you to tailor its behavior to your specific needs.
|
||||||
|
|
||||||
|
### Modifying the AI Prompt
|
||||||
|
|
||||||
|
The AI's behavior is guided by a custom prompt that defines how it should interpret log data and what actions it should take. You can modify this prompt in `ai_log_backend.js` to adjust the AI's focus or add new rules.
|
||||||
|
|
||||||
|
**Example Customization**:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
const prompt = `
|
||||||
|
You are a security AI responsible for analyzing web traffic...
|
||||||
|
- Ignore any IPs within the range 10.0.0.0/8.
|
||||||
|
- Flag any requests to /admin as potential threats.
|
||||||
|
- Use emojis to convey the severity of alerts.
|
||||||
|
`;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Adjusting Buffer Limits
|
||||||
|
|
||||||
|
The log buffer size determines how many log lines are collected before they are sent to the backend. Adjust this size to balance network usage and processing frequency.
|
||||||
|
|
||||||
|
**Example Customization**:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
const LOG_BUFFER_LIMIT = 30; // Increase the buffer size to 30 lines
|
||||||
|
```
|
||||||
|
|
||||||
|
### Extending Log Parsing Capabilities
|
||||||
|
|
||||||
|
You can extend the log parsing functionality by adding new regular expressions or parsing logic to handle different log formats or detect new types of threats.
|
||||||
|
|
||||||
|
**Example Customization**:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
tail.on('line', async (line) => {
|
||||||
|
// Add custom logic to detect SQL injection attempts
|
||||||
|
if (/SELECT.*FROM/i.test(line)) {
|
||||||
|
log.warn(`Potential SQL injection detected: ${line}`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
## Contributing
|
||||||
|
|
||||||
|
Contributions are welcome! To contribute, fork the repository, create a new branch for your changes, and submit a pull request. Please ensure that your code adheres to the existing style and that you include tests for any new features.
|
||||||
|
|||||||
547
ai_log.js
547
ai_log.js
@ -1,279 +1,332 @@
|
|||||||
|
|
||||||
// Import necessary modules for the application
|
// Import necessary modules for the application
|
||||||
import express from 'express'; // Express framework for building web server applications and handling HTTP requests and responses
|
const axios = require('axios'); // Axios is used to make HTTP requests to external APIs or services
|
||||||
import axios from 'axios'; // Axios is used to make HTTP requests to external APIs or services
|
const { exec } = require('child_process'); // exec is used to execute shell commands in a child process
|
||||||
import bodyParser from 'body-parser'; // Middleware for parsing incoming request bodies, specifically for handling JSON data
|
const moment = require('moment'); // Moment.js is a library for parsing, validating, manipulating, and formatting dates
|
||||||
import cmd from 'cmd-promise'; // A module that allows execution of shell commands in a promise-based manner, making it easier to manage async operations
|
const fs = require('fs'); // File system module for interacting with the file system
|
||||||
import cors from 'cors'; // Middleware to enable Cross-Origin Resource Sharing, allowing resources to be requested from another domain
|
const path = require('path'); // Path module for handling and transforming file paths
|
||||||
import cheerio from 'cheerio'; // Cheerio is a server-side jQuery-like library for parsing and manipulating HTML content
|
const Tail = require('tail').Tail; // Tail module is used for monitoring log files and reacting to new lines as they are added
|
||||||
import 'dotenv/config'; // Loads environment variables from a .env file into process.env, allowing secure storage of sensitive information
|
|
||||||
import llamaTokenizer from 'llama-tokenizer-js'; // A library for tokenizing text, which is crucial for managing the length of text inputs to the AI model
|
|
||||||
|
|
||||||
// Define a prompt that will guide the AI's behavior when analyzing NGINX logs for potential security issues
|
// Configuration constants
|
||||||
const prompt = `
|
const LOG_DIRECTORY = '/dockerData/logs'; // Directory where NGINX logs are stored
|
||||||
You are a security AI responsible for analyzing web traffic from NGINX logs and blocking malicious IPs. Your task is to review the logs for potential attacks and issues. If you identify a verified problem, include [ALERT] followed by a detailed description of the issue in your response. Ensure your message is formatted in Markdown compatible with Discord.
|
const BACKEND_URL = 'http://127.0.0.1:3001'; // URL for the backend process that handles log processing
|
||||||
|
const DISCORD_WEBHOOK_URL = 'WEBHOOKURL'; // URL of the Discord webhook for sending alerts and notifications
|
||||||
|
|
||||||
**Guidelines:**
|
// Environment-dependent configuration
|
||||||
- Domains on the server: shells.lol, hehe.rest, dcord.rest, nodejs.lol, dht.rest, my-mc.link, ssh.surf, tcp.quest
|
const DEBUG = process.env.DEBUG === 'true'; // Enable or disable debug logging based on environment variable
|
||||||
- Use emojis to enhance communication.
|
const LOG_BUFFER_LIMIT = 15; // Number of log lines to accumulate before sending them to the backend
|
||||||
- Do not report IP addresses for scraping or crawling.
|
const TIME_LIMIT = 10 * 60 * 1000; // Time interval (in milliseconds) to send logs even if buffer is not full (10 minutes)
|
||||||
- Ignore IPs: x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x. Do not mention these in reports.
|
|
||||||
- Ignore IP addresses with BOGONs such as 192.168.0.1 or 127.0.0.2, etc.
|
|
||||||
- Avoid alerting for false positives or irregular activity.
|
|
||||||
- If there are no alerts but you have interesting findings, write: GENERAL followed by your insights in Markdown.
|
|
||||||
- Only send GENERAL messages for noteworthy events, not for routine traffic reports.
|
|
||||||
- In a GENERAL message, feel free to provide a long explainer on your deductions.
|
|
||||||
- Be decisive. If an IP is being malicious, block it. Do not monitor IPs for further traffic.
|
|
||||||
- Do not ban an IP address without a corresponding log entry, provide this in your response.
|
|
||||||
- Block all bot IPs and information scanners except Google.
|
|
||||||
- Provide proof and reasoning for each ban.
|
|
||||||
- DO NOT BAN AN IP THAT IS NOT IN A LOG EVER! YOU MUST SEE THE IP ADDRESS!
|
|
||||||
- To ban an IP or flag it as a security risk, wrap it in a Markdown spoiler: ||IPHERE||
|
|
||||||
`;
|
|
||||||
|
|
||||||
// Initialize the Express application and define the port on which the server will run
|
let logBuffer = []; // Array to store log lines temporarily before sending to backend
|
||||||
const app = express(); // Create an instance of an Express application
|
let logTails = []; // Array to store active Tail instances for each log file being monitored
|
||||||
const port = 3001; // Define the port number for the server, 3001 is commonly used for development
|
let isSendingLogs = false; // Flag to prevent multiple simultaneous log sending operations
|
||||||
|
|
||||||
// Middleware to enable CORS for all routes
|
// List of IP addresses to ignore in logs (e.g., trusted IPs, public DNS servers)
|
||||||
app.use(cors()); // This allows the server to accept requests from any origin, useful for APIs that may be accessed by web applications from different domains
|
const ignoredIPs = ['1.1.1.1', '1.0.0.1', '8.8.8.8', '8.8.4.4'];
|
||||||
|
|
||||||
// Set a larger limit for the request body to handle large data payloads
|
// List of IP subnets to ignore, commonly used to filter out traffic from known sources like Cloudflare
|
||||||
app.use(bodyParser.json({ limit: '50mb' })); // The JSON body parser is configured with a 50MB limit, suitable for handling large JSON payloads
|
const ignoredSubnets = [
|
||||||
|
'173.245.48.0/20', '103.21.244.0/22', '103.22.200.0/22', '103.31.4.0/22',
|
||||||
|
'141.101.64.0/18', '108.162.192.0/18', '190.93.240.0/20', '188.114.96.0/20',
|
||||||
|
'197.234.240.0/22', '198.41.128.0/17', '162.158.0.0/15', '104.16.0.0/13',
|
||||||
|
'104.24.0.0/14', '172.64.0.0/13', '131.0.72.0/22'
|
||||||
|
];
|
||||||
|
|
||||||
// Define constants for the application, used to control various aspects of the server's behavior
|
// List of specific log files to ignore (e.g., specific proxy logs)
|
||||||
const TIMEOUT_DURATION = 100000; // The maximum time (in milliseconds) the server will wait before timing out a request, set to 100 seconds
|
const ignoredFiles = ['proxy-host-149_access.log', 'proxy-host-2_access.log', 'proxy-host-99_access.log'];
|
||||||
const MAX_TOKENS = 8000; // The maximum number of tokens (words and punctuation) allowed in a conversation, this limit helps manage API usage
|
|
||||||
const TOLERANCE = 100; // A buffer value used to prevent exceeding the MAX_TOKENS limit, ensuring the conversation stays within safe bounds
|
|
||||||
let conversationHistory = {}; // An object to store conversation history for each IP address, allowing the server to maintain context for each user
|
|
||||||
|
|
||||||
// Helper function to get the current timestamp in a formatted string
|
// Function to get current timestamp in a formatted string (e.g., 'YYYY-MM-DD HH:mm:ss')
|
||||||
const getTimestamp = () => {
|
const getTimestamp = () => moment().format('YYYY-MM-DD HH:mm:ss');
|
||||||
const now = new Date(); // Get the current date and time
|
|
||||||
const date = now.toLocaleDateString('en-US'); // Format the date in the US locale
|
// Logging functions for different log levels (INFO, WARN, ERROR, SUCCESS, DEBUG)
|
||||||
const time = now.toLocaleTimeString('en-US'); // Format the time in the US locale
|
const log = {
|
||||||
return `${date} [${time}]`; // Return the formatted date and time as a string
|
info: (message) => console.log(`[${getTimestamp()}] [INFO] ${message}`), // Log informational messages
|
||||||
|
warn: (message) => console.log(`[${getTimestamp()}] [WARN] ${message}`), // Log warning messages
|
||||||
|
error: (message) => console.log(`[${getTimestamp()}] [ERROR] ${message}`), // Log error messages
|
||||||
|
success: (message) => console.log(`[${getTimestamp()}] [SUCCESS] ${message}`), // Log success messages
|
||||||
|
debug: (message) => {
|
||||||
|
if (DEBUG) { // Log debug messages only if DEBUG mode is enabled
|
||||||
|
console.log(`[${getTimestamp()}] [DEBUG] ${message}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
// Middleware to track conversation history based on the client's IP address
|
// Function to check if an IP address is in the ignored list or subnets
|
||||||
app.use((req, res, next) => {
|
const isIgnoredIP = async (ip) => {
|
||||||
// Extract the client's IP address from various possible headers (CF-Connecting-IP, X-Forwarded-For, X-Real-IP) or fallback to req.ip
|
if (ignoredIPs.includes(ip)) {
|
||||||
const ip = req.headers['cf-connecting-ip'] || req.headers['x-forwarded-for'] || req.headers['x-real-ip'] || req.ip;
|
return true; // Immediately return true if the IP is in the ignored IPs list
|
||||||
req.clientIp = ip; // Store the client's IP address in the request object for easy access later
|
|
||||||
|
|
||||||
// Log the incoming request along with the client's IP address and current timestamp
|
|
||||||
console.log(`${getTimestamp()} [INFO] Incoming request from IP: ${req.clientIp}`);
|
|
||||||
|
|
||||||
// If this IP address has not been seen before, initialize a new conversation history for it
|
|
||||||
if (!conversationHistory[req.clientIp]) {
|
|
||||||
console.log(`${getTimestamp()} [INFO] Initializing conversation history for new IP: ${req.clientIp}`);
|
|
||||||
// Start the conversation with the predefined prompt that instructs the AI on how to analyze the logs
|
|
||||||
conversationHistory[req.clientIp] = [
|
|
||||||
{ role: 'system', content: prompt }
|
|
||||||
];
|
|
||||||
}
|
}
|
||||||
next(); // Move on to the next middleware or route handler
|
const { default: CIDR } = await import('ip-cidr'); // Dynamically import the ip-cidr module for CIDR range checking
|
||||||
});
|
return ignoredSubnets.some((subnet) => new CIDR(subnet).contains(ip)); // Check if the IP is within any ignored subnets
|
||||||
|
};
|
||||||
|
|
||||||
// Function to count the number of tokens in a conversation history using the llama tokenizer
|
// Function to read and monitor log files in the specified directory using the Tail module
|
||||||
async function countLlamaTokens(messages) {
|
const readLogs = () => {
|
||||||
let totalTokens = 0; // Initialize a counter for the total number of tokens
|
log.info('Initiating log reading process...');
|
||||||
for (const message of messages) {
|
|
||||||
// Only count tokens for user and assistant messages, not system messages
|
|
||||||
if (message.role === 'user' || message.role === 'assistant') {
|
|
||||||
const encodedTokens = llamaTokenizer.encode(message.content); // Tokenize the message content
|
|
||||||
totalTokens += encodedTokens.length; // Add the number of tokens in the current message to the total
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return totalTokens; // Return the total number of tokens
|
|
||||||
}
|
|
||||||
|
|
||||||
// Function to trim the conversation history to fit within the token limit
|
// Stop and clear any existing Tail instances
|
||||||
async function trimConversationHistory(messages, maxLength, tolerance) {
|
logTails.forEach(tail => tail.unwatch());
|
||||||
let tokenLength = await countLlamaTokens(messages); // Get the total number of tokens in the conversation
|
logTails = [];
|
||||||
// Continue trimming the conversation history until it's within the allowed token limit
|
|
||||||
while (tokenLength > maxLength - tolerance && messages.length > 1) {
|
|
||||||
messages.splice(1, 1); // Remove the oldest user/assistant message (the second item in the array)
|
|
||||||
tokenLength = await countLlamaTokens(messages); // Recalculate the total number of tokens after trimming
|
|
||||||
console.log(`${getTimestamp()} [CLEANUP] Trimmed conversation history to ${tokenLength} tokens.`);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Function to scrape a web page and extract its content
|
// Read the directory to get all log files
|
||||||
async function scrapeWebPage(url) {
|
fs.readdir(LOG_DIRECTORY, (err, files) => {
|
||||||
console.log(`${getTimestamp()} [INFO] Starting to scrape URL: ${url}`);
|
if (err) {
|
||||||
try {
|
log.error(`Error reading directory: ${err}`); // Log an error if the directory cannot be read
|
||||||
// Perform an HTTP GET request to fetch the content of the specified URL
|
|
||||||
const res = await axios.get(url);
|
|
||||||
const html = res.data; // Extract the HTML content from the response
|
|
||||||
const $ = cheerio.load(html); // Load the HTML into Cheerio for parsing and manipulation
|
|
||||||
|
|
||||||
// Extract specific elements from the HTML: the page title, meta description, and body content
|
|
||||||
const pageTitle = $('head title').text().trim(); // Get the text of the <title> tag
|
|
||||||
const pageDescription = $('head meta[name="description"]').attr('content'); // Get the content of the meta description
|
|
||||||
const pageContent = $('body').text().trim(); // Get all text content within the <body> tag
|
|
||||||
|
|
||||||
// Construct a response message with the extracted details
|
|
||||||
let response = `Title: ${pageTitle}\n`; // Start with the page title
|
|
||||||
if (pageDescription) {
|
|
||||||
response += `Description: ${pageDescription}\n`; // Add the meta description if it exists
|
|
||||||
}
|
|
||||||
if (pageContent) {
|
|
||||||
const MAX_CONTENT_LENGTH = process.env.MAX_CONTENT_LENGTH || 2000; // Set a maximum length for the content
|
|
||||||
// Clean the page content to remove unnecessary whitespace and special characters
|
|
||||||
let plainTextContent = $('<div>').html(pageContent).text().trim().replace(/[\r\n\t]+/g, ' ');
|
|
||||||
// Define a regular expression pattern to identify code-like content
|
|
||||||
const codePattern = /\/\/|\/\*|\*\/|\{|\}|\[|\]|\bfunction\b|\bclass\b|\b0x[0-9A-Fa-f]+\b|\b0b[01]+\b/;
|
|
||||||
const isCode = codePattern.test(plainTextContent); // Check if the content resembles code
|
|
||||||
|
|
||||||
if (isCode) {
|
|
||||||
plainTextContent = plainTextContent.replace(codePattern, ''); // Remove code-like patterns if detected
|
|
||||||
}
|
|
||||||
// Further clean the content by removing text within parentheses
|
|
||||||
plainTextContent = plainTextContent.replace(/ *\([^)]*\) */g, '');
|
|
||||||
// If the content is too long, truncate it and add an ellipsis
|
|
||||||
if (plainTextContent.length > MAX_CONTENT_LENGTH) {
|
|
||||||
plainTextContent = plainTextContent.substring(0, MAX_CONTENT_LENGTH) + '...';
|
|
||||||
}
|
|
||||||
response += `Content: ${plainTextContent.trim()}`; // Add the cleaned and possibly truncated content to the response
|
|
||||||
}
|
|
||||||
response += `\nURL: ${url}`; // Include the original URL in the response
|
|
||||||
|
|
||||||
console.log(`${getTimestamp()} [INFO] Successfully scraped URL: ${url}`);
|
|
||||||
return response; // Return the constructed response
|
|
||||||
} catch (err) {
|
|
||||||
// If the scraping process fails, log an error with details and return null
|
|
||||||
console.error(`${getTimestamp()} [ERROR] Failed to scrape URL: ${url}`, err);
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Function to process incoming requests, handle AI interactions, and return a response
|
|
||||||
async function processRequest(req, res) {
|
|
||||||
const startTime = Date.now(); // Record the start time of the request processing for performance tracking
|
|
||||||
const ip = req.clientIp; // Retrieve the client's IP address from the request object
|
|
||||||
console.log(`${getTimestamp()} [INFO] Handling chat request from IP: ${ip}`); // Log the request details
|
|
||||||
|
|
||||||
// Set a timeout for the request processing, ensuring it doesn't hang indefinitely
|
|
||||||
const timeout = setTimeout(() => {
|
|
||||||
console.error(`${getTimestamp()} [ERROR] Request timed out for IP: ${ip}`);
|
|
||||||
res.status(408).json({ message: "Request timed out" }); // Send a timeout response if the processing takes too long
|
|
||||||
}, TIMEOUT_DURATION);
|
|
||||||
|
|
||||||
try {
|
|
||||||
let userMessage = req.body.message; // Extract the user's message from the request body
|
|
||||||
console.log(`${getTimestamp()} [INFO] Received user message: ${userMessage}`);
|
|
||||||
userMessage = req.body.message + `\nDate/Time:${getTimestamp()}`; // Append the current date and time to the user's message
|
|
||||||
|
|
||||||
// Initialize conversation history if it doesn't exist for the IP
|
|
||||||
if (!conversationHistory[ip]) {
|
|
||||||
console.log(`${getTimestamp()} [INFO] Initializing conversation history for new IP: ${ip}`);
|
|
||||||
conversationHistory[ip] = [{ role: 'system', content: prompt }]; // Start the conversation with the predefined prompt
|
|
||||||
}
|
|
||||||
// Add the user's message to the conversation history for the IP
|
|
||||||
conversationHistory[ip].push({ role: 'user', content: userMessage });
|
|
||||||
|
|
||||||
// Trim the conversation history if it exceeds the token limit
|
|
||||||
await trimConversationHistory(conversationHistory[ip], MAX_TOKENS, TOLERANCE);
|
|
||||||
|
|
||||||
// Split the user's message into individual log lines
|
|
||||||
const logLines = userMessage.split('\n');
|
|
||||||
// Define a regex pattern to identify lines containing client IP addresses
|
|
||||||
const clientIpRegex = /\[Client (\d{1,3}\.){3}\d{1,3}\]/;
|
|
||||||
// Filter the log lines to only include those with valid client IP addresses
|
|
||||||
const filteredLogLines = logLines.filter(line => clientIpRegex.test(line));
|
|
||||||
|
|
||||||
// If no valid IP addresses are found in the log lines, send a response indicating this
|
|
||||||
if (filteredLogLines.length === 0) {
|
|
||||||
console.log(`${getTimestamp()} [INFO] No valid client IP addresses found in the log.`);
|
|
||||||
res.json({ message: "No valid client IP addresses found in the log." });
|
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Join the filtered log lines back into a single string for processing
|
// Filter log files, excluding those in the ignoredFiles list
|
||||||
const filteredMessage = filteredLogLines.join('\n');
|
const logFiles = files.filter(file => file.endsWith('.log') && !ignoredFiles.includes(file));
|
||||||
|
if (logFiles.length === 0) {
|
||||||
|
log.warn(`No log files found in directory: ${LOG_DIRECTORY}`); // Warn if no log files are found
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
// Send the request to the llama API for processing and response generation
|
log.info(`Found ${logFiles.length} log files to tail.`); // Log the number of log files to be monitored
|
||||||
console.log(`${getTimestamp()} [INFO] Sending request to llama API for response`);
|
|
||||||
const response = await axios.post('http://127.0.0.1:8003/v1/chat/completions', {
|
// For each log file, start a new Tail instance to monitor the file
|
||||||
model: 'gpt-3.5-turbo', // Specify the AI model to use
|
logFiles.forEach(file => {
|
||||||
messages: [...conversationHistory[ip], { role: 'user', content: filteredMessage }] // Include the conversation history and the filtered message
|
const filePath = path.join(LOG_DIRECTORY, file); // Create the full path to the log file
|
||||||
|
log.info(`Starting to read log from: ${filePath}`); // Log the start of monitoring for this file
|
||||||
|
try {
|
||||||
|
const tail = new Tail(filePath); // Create a new Tail instance for the log file
|
||||||
|
|
||||||
|
// Event listener for new lines added to the log file
|
||||||
|
tail.on('line', async (line) => {
|
||||||
|
if (line.includes('git.ssh.surf')) {
|
||||||
|
log.debug(`Ignoring line involving git.ssh.surf: ${line}`); // Ignore lines related to specific domains
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
log.debug(`Read line: ${line}`); // Debug log for each line read
|
||||||
|
const ipMatch = line.match(/\[Client (\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\]/); // Regex to extract client IP from the log line
|
||||||
|
if (ipMatch) {
|
||||||
|
const ip = ipMatch[1];
|
||||||
|
const isIgnored = await isIgnoredIP(ip); // Check if the IP should be ignored
|
||||||
|
if (isIgnored) {
|
||||||
|
log.debug(`Ignored line with IP: ${ip}`); // Debug log for ignored IPs
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Add the line to the log buffer
|
||||||
|
logBuffer.push(line);
|
||||||
|
// If buffer reaches the limit, send logs to the backend
|
||||||
|
if (logBuffer.length >= LOG_BUFFER_LIMIT) {
|
||||||
|
await sendLogsToBackend();
|
||||||
|
}
|
||||||
});
|
});
|
||||||
// Extract the AI's response from the API's response data
|
|
||||||
const assistantMessage = response.data.choices[0].message;
|
|
||||||
// Add the AI's response to the conversation history
|
|
||||||
conversationHistory[ip].push(assistantMessage);
|
|
||||||
|
|
||||||
// Log the AI's response and additional details like the finish reason and token usage
|
// Event listener for errors in Tail instance
|
||||||
console.log(`${getTimestamp()} [INFO] Received response from llama API: ${assistantMessage.content}`);
|
tail.on('error', (error) => {
|
||||||
console.log(`${getTimestamp()} [DEBUG] Finish Reason: ${response.data.choices[0].finish_reason}`);
|
log.error(`Tail error: ${error}`); // Log errors that occur while tailing the file
|
||||||
console.log(`${getTimestamp()} [STATS] Usage: prompt_tokens=${response.data.usage.prompt_tokens}, completion_tokens=${response.data.usage.completion_tokens}, total_tokens=${response.data.usage.total_tokens}`);
|
});
|
||||||
|
|
||||||
clearTimeout(timeout); // Clear the timeout to prevent it from triggering
|
tail.watch(); // Start watching the log file for new lines
|
||||||
res.json(assistantMessage); // Send the AI's response back to the client
|
|
||||||
} catch (error) {
|
log.debug(`Started tailing file: ${filePath}`); // Debug log indicating the file is being monitored
|
||||||
// If an error occurs during request processing, log the error and send a 500 response
|
logTails.push(tail); // Add the Tail instance to the list of active Tails
|
||||||
console.error(`${getTimestamp()} [ERROR] An error occurred while handling chat request`, error);
|
} catch (ex) {
|
||||||
clearTimeout(timeout); // Clear the timeout to prevent it from triggering
|
log.error(`Failed to tail file ${filePath}: ${ex}`); // Log any exceptions that occur while starting the Tail
|
||||||
res.status(500).json({ message: "An error occurred", error: error.message }); // Send an error response
|
}
|
||||||
} finally {
|
});
|
||||||
// Record the end time and calculate the total processing time in seconds
|
});
|
||||||
const endTime = Date.now();
|
};
|
||||||
const processingTime = ((endTime - startTime) / 1000).toFixed(2); // Convert milliseconds to seconds
|
|
||||||
console.log(`${getTimestamp()} [STATS] Processing Time: ${processingTime} seconds`); // Log the processing time
|
// Function to count the number of tokens in a message using llama-tokenizer-js
|
||||||
console.log(`${getTimestamp()} [INFO] Finished processing chat request for IP: ${ip}`);
|
async function countLlamaTokens(messages) {
|
||||||
|
const llamaTokenizer = await import('llama-tokenizer-js'); // Dynamically import the tokenizer module
|
||||||
|
let totalTokens = 0; // Initialize token counter
|
||||||
|
for (const message of messages) {
|
||||||
|
if (message.role === 'user' || message.role === 'assistant') {
|
||||||
|
const encodedTokens = llamaTokenizer.default.encode(message.content); // Encode message content to count tokens
|
||||||
|
totalTokens += encodedTokens.length; // Accumulate the total number of tokens
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return totalTokens; // Return the total token count
|
||||||
|
}
|
||||||
|
|
||||||
|
// Function to trim conversation history to fit within token limits
|
||||||
|
async function trimConversationHistory(messages, maxLength, tolerance) {
|
||||||
|
let tokenLength = await countLlamaTokens(messages); // Get the current token length
|
||||||
|
if (tokenLength > maxLength + tolerance) {
|
||||||
|
const diff = tokenLength - (maxLength + tolerance); // Calculate how many tokens need to be removed
|
||||||
|
let removedTokens = 0;
|
||||||
|
|
||||||
|
// Iterate over the messages in reverse order to remove older messages first
|
||||||
|
for (let i = messages.length - 1; i >= 0; i--) {
|
||||||
|
const message = messages[i];
|
||||||
|
const messageTokens = await countLlamaTokens([message]); // Count tokens in the current message
|
||||||
|
|
||||||
|
if (removedTokens + messageTokens <= diff) {
|
||||||
|
messages.splice(i, 1); // Remove the message if it helps reduce the token count sufficiently
|
||||||
|
removedTokens += messageTokens;
|
||||||
|
console.log(`${getTimestamp()} [CLEANUP] ${removedTokens} removed | After Resize: ${await countLlamaTokens(messages)}`);
|
||||||
|
} else {
|
||||||
|
const messagesToRemove = Math.floor(diff / messageTokens); // Determine how many messages need to be removed
|
||||||
|
for (let j = 0; j < messagesToRemove; j++) {
|
||||||
|
messages.splice(i, 1); // Remove the determined number of messages
|
||||||
|
removedTokens += messageTokens;
|
||||||
|
}
|
||||||
|
break; // Exit the loop once enough tokens have been removed
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Route to handle incoming chat requests, trim the message content, and process the request
|
// Function to send accumulated log buffer to the backend server
|
||||||
app.post('/api/v1/chat', async (req, res) => {
|
const sendLogsToBackend = async () => {
|
||||||
// Trim the incoming message to fit within token limits
|
if (logBuffer.length === 0) {
|
||||||
const messageContent = req.body.message; // Get the user's message from the request body
|
log.info('Log buffer is empty, skipping sending to backend'); // Log if there are no logs to send
|
||||||
const encodedTokens = llamaTokenizer.encode(messageContent); // Tokenize the message to determine its length in tokens
|
return;
|
||||||
const MAX_MESSAGE_TOKENS = MAX_TOKENS - (await countLlamaTokens([{ role: 'system', content: prompt }])) - TOLERANCE; // Calculate the maximum allowed tokens for the user's message
|
|
||||||
|
|
||||||
// If the message exceeds the allowed token limit, trim it to fit
|
|
||||||
let trimmedMessageContent = messageContent;
|
|
||||||
if (encodedTokens.length > MAX_MESSAGE_TOKENS) {
|
|
||||||
trimmedMessageContent = llamaTokenizer.decode(encodedTokens.slice(0, MAX_MESSAGE_TOKENS)); // Truncate the message and decode it back to a string
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Process the trimmed message and send the response
|
if (isSendingLogs) {
|
||||||
await processRequest({ ...req, body: { message: trimmedMessageContent } }, res);
|
log.info('Log sending is already in progress, skipping...'); // Prevent concurrent log sending operations
|
||||||
});
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
// Route to fetch the conversation history for a specific IP address
|
isSendingLogs = true; // Set the flag to indicate logs are being sent
|
||||||
app.get('/api/v1/conversation-history', (req, res) => {
|
log.info('Sending logs to backend...'); // Log the start of the log sending process
|
||||||
const ip = req.clientIp; // Get the client's IP address from the request object
|
try {
|
||||||
console.log(`${getTimestamp()} [INFO] Fetching conversation history for IP: ${ip}`); // Log the request details
|
const messages = [{ role: 'user', content: logBuffer.join('\n') }]; // Combine the log buffer into a single message
|
||||||
res.json(conversationHistory[ip]); // Send the conversation history for the IP as a JSON response
|
await trimConversationHistory(messages, 2000, 100); // Trim the message if it exceeds token limits
|
||||||
});
|
const response = await axios.post(BACKEND_URL, { message: messages.map(msg => msg.content).join('\n') }); // Send the logs to the backend
|
||||||
|
|
||||||
// Route to restart the core AI service via Docker, typically used to refresh the model or resolve issues
|
// Check the response for any alerts, actions, or reports
|
||||||
app.post('/api/v1/restart-core', (req, res) => {
|
if (response.data.content.includes('ALERT') || response.data.content.includes('ACTION') || response.data.content.includes('REPORT')) {
|
||||||
console.log(`${getTimestamp()} [INFO] Restarting core service`); // Log the restart action
|
log.warn('ALERT detected in response'); // Log if an alert is detected
|
||||||
cmd(`docker restart llama-gpu-server`).then(out => { // Execute a shell command to restart the Docker container running the AI model
|
const ips = extractIPsFromAlert(response.data.content); // Extract IP addresses from the alert message
|
||||||
console.log(`${getTimestamp()} [INFO] Core service restarted`); // Log the successful restart
|
if (ips.length > 0) {
|
||||||
res.json(out.stdout); // Send the output of the restart command back to the client
|
const nonIgnoredIPs = [];
|
||||||
}).catch(err => { // Handle any errors that occur during the restart
|
for (const ip of ips) {
|
||||||
console.error(`${getTimestamp()} [ERROR] Failed to restart core service`, err); // Log the error
|
if (await isIgnoredIP(ip)) {
|
||||||
res.status(500).json({ message: "An error occurred while restarting the core service", error: err.message }); // Send an error response
|
log.debug(`Skipping banning for ignored IP: ${ip}`); // Skip banning if the IP is in the ignored list
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
log.info(`Detected IP for banning: ${ip}`); // Log the IP address that will be banned
|
||||||
|
await banIP(ip); // Execute the ban command for the IP
|
||||||
|
await delay(3000); // Add a 3-second delay between bans to avoid overloading the system
|
||||||
|
nonIgnoredIPs.push(ip); // Keep track of banned IPs that are not ignored
|
||||||
|
}
|
||||||
|
await sendAlertToDiscord(response.data.content, nonIgnoredIPs); // Send the alert message to Discord
|
||||||
|
} else {
|
||||||
|
log.warn('No IPs detected for banning.'); // Log if no IPs were found for banning
|
||||||
|
await sendAlertToDiscord(response.data.content, []); // Still send the alert to Discord, even without IPs
|
||||||
|
}
|
||||||
|
} else if (response.data.content.includes('GENERAL')) {
|
||||||
|
await sendGeneralToDiscord(response.data.content); // Send general information to Discord if present
|
||||||
|
} else {
|
||||||
|
log.info('No alerts detected in response'); // Log if no significant alerts are found
|
||||||
|
log.info(`Response:\n ${response.data.content}`); // Log the response content for review
|
||||||
|
}
|
||||||
|
|
||||||
|
// Clear the log buffer after successful sending
|
||||||
|
logBuffer = [];
|
||||||
|
log.info('Log buffer cleared');
|
||||||
|
|
||||||
|
// Reset the conversation history on the backend to start fresh
|
||||||
|
await resetConversationHistory();
|
||||||
|
} catch (error) {
|
||||||
|
log.error(`Error sending logs to backend: ${error.message}`); // Log any errors that occur during the process
|
||||||
|
} finally {
|
||||||
|
isSendingLogs = false; // Reset the flag to allow new log sending operations
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
// Function to extract IP addresses from the alert message using a regular expression
|
||||||
|
const extractIPsFromAlert = (message) => {
|
||||||
|
const ipPattern = /\|\|(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\|\|/g; // Regex to find IPs wrapped in ||...||
|
||||||
|
let matches;
|
||||||
|
const uniqueIPs = new Set(); // Use a Set to store unique IP addresses
|
||||||
|
|
||||||
|
while ((matches = ipPattern.exec(message)) !== null) {
|
||||||
|
uniqueIPs.add(matches[1]); // Add each found IP to the Set
|
||||||
|
}
|
||||||
|
|
||||||
|
return Array.from(uniqueIPs); // Convert the Set to an array of unique IPs
|
||||||
|
};
|
||||||
|
|
||||||
|
// Function to ban an IP address using a shell command
|
||||||
|
const banIP = (ip) => {
|
||||||
|
return new Promise((resolve, reject) => {
|
||||||
|
log.info(`Banning IP address: ${ip}`); // Log the IP address being banned
|
||||||
|
exec(`/usr/bin/banIP ${ip}`, (error, stdout, stderr) => {
|
||||||
|
if (error) {
|
||||||
|
log.error(`Error banning IP address: ${error.message}`); // Log any errors that occur during the ban
|
||||||
|
reject(error); // Reject the promise if an error occurs
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (stderr) {
|
||||||
|
log.warn(`stderr: ${stderr}`); // Log any warnings or errors from the command's stderr
|
||||||
|
}
|
||||||
|
log.success(`IP address ${ip} has been banned`); // Log a success message if the ban was successful
|
||||||
|
resolve(); // Resolve the promise to indicate success
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
};
|
||||||
|
|
||||||
// Route to reset the conversation history for a specific IP address, effectively starting a new session
|
// Function to send alert messages to a Discord channel via webhook
|
||||||
app.post('/api/v1/reset-conversation', (req, res) => {
|
const sendAlertToDiscord = async (alertMessage, ips) => {
|
||||||
const ip = req.clientIp; // Get the client's IP address from the request object
|
log.info('Sending alert to Discord...'); // Log the start of the Discord alert sending process
|
||||||
console.log(`${getTimestamp()} [INFO] Resetting conversation history for IP: ${ip}`); // Log the reset action
|
try {
|
||||||
|
await axios.post(DISCORD_WEBHOOK_URL, {
|
||||||
|
embeds: [
|
||||||
|
{
|
||||||
|
title: 'Alert Detected', // Title for the Discord embed
|
||||||
|
description: alertMessage, // The alert message content
|
||||||
|
color: 15158332, // Red color for alerts
|
||||||
|
fields: ips.filter(ip => !ignoredIPs.includes(ip)).map(ip => ({
|
||||||
|
name: 'Banned IP', // Field name in the Discord embed
|
||||||
|
value: ip, // The banned IP address
|
||||||
|
inline: true // Display fields inline for better readability
|
||||||
|
})),
|
||||||
|
timestamp: new Date() // Timestamp for when the alert was sent
|
||||||
|
}
|
||||||
|
]
|
||||||
|
});
|
||||||
|
log.success('Alert sent to Discord'); // Log a success message if the alert was successfully sent
|
||||||
|
} catch (error) {
|
||||||
|
log.error(`Error sending alert to Discord: ${error.message}`); // Log any errors that occur during the Discord alert sending process
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
// Reset the conversation history to its initial state for the given IP
|
// Function to send general information messages to a Discord channel via webhook
|
||||||
conversationHistory[ip] = [
|
const sendGeneralToDiscord = async (generalMessage) => {
|
||||||
{ role: 'system', content: prompt }
|
log.info('Sending general information to Discord...'); // Log the start of the general information sending process
|
||||||
];
|
try {
|
||||||
console.log(`${getTimestamp()} [INFO] Conversation history reset for IP: ${ip}`); // Log the successful reset
|
await axios.post(DISCORD_WEBHOOK_URL, {
|
||||||
res.json({ message: "Conversation history reset for IP: " + ip }); // Send a confirmation message back to the client
|
embeds: [
|
||||||
});
|
{
|
||||||
|
title: 'General Information', // Title for the Discord embed
|
||||||
|
description: generalMessage, // The general information content
|
||||||
|
color: 3066993, // Blue color for general information
|
||||||
|
timestamp: new Date() // Timestamp for when the information was sent
|
||||||
|
}
|
||||||
|
]
|
||||||
|
});
|
||||||
|
log.success('General information sent to Discord'); // Log a success message if the general information was successfully sent
|
||||||
|
} catch (error) {
|
||||||
|
log.error(`Error sending general information to Discord: ${error.message}`); // Log any errors that occur during the Discord general information sending process
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
// Start the Express server on the defined port, making the API available for requests
|
// Function to reset the conversation history in the backend
|
||||||
app.listen(port, () => {
|
const resetConversationHistory = async () => {
|
||||||
console.log(`${getTimestamp()} [INFO] Server running at http://localhost:${port}`); // Log the server startup and its URL
|
log.info('Resetting conversation history...'); // Log the start of the conversation history reset process
|
||||||
});
|
try {
|
||||||
|
await axios.post(`${BACKEND_URL.replace('/api/v1/chat', '')}/api/v1/reset-conversation`); // Send a request to reset the conversation history
|
||||||
|
log.success('Conversation history reset'); // Log a success message if the reset was successful
|
||||||
|
} catch (error) {
|
||||||
|
log.error(`Error resetting conversation history: ${error.message}`); // Log any errors that occur during the conversation history reset process
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
// Utility function to introduce a delay between operations, useful for rate limiting
|
||||||
|
const delay = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); // Create a promise that resolves after the specified delay
|
||||||
|
|
||||||
|
// Start reading logs continuously from the specified directory
|
||||||
|
readLogs();
|
||||||
|
|
||||||
|
// Set up an interval to send logs to the backend if buffer limit is reached or every 10 minutes
|
||||||
|
setInterval(sendLogsToBackend, TIME_LIMIT);
|
||||||
|
|||||||
279
ai_log_backend.js
Normal file
279
ai_log_backend.js
Normal file
@ -0,0 +1,279 @@
|
|||||||
|
// Import necessary modules for the application
|
||||||
|
import express from 'express'; // Express framework for building web server applications and handling HTTP requests and responses
|
||||||
|
import axios from 'axios'; // Axios is used to make HTTP requests to external APIs or services
|
||||||
|
import bodyParser from 'body-parser'; // Middleware for parsing incoming request bodies, specifically for handling JSON data
|
||||||
|
import cmd from 'cmd-promise'; // A module that allows execution of shell commands in a promise-based manner, making it easier to manage async operations
|
||||||
|
import cors from 'cors'; // Middleware to enable Cross-Origin Resource Sharing, allowing resources to be requested from another domain
|
||||||
|
import cheerio from 'cheerio'; // Cheerio is a server-side jQuery-like library for parsing and manipulating HTML content
|
||||||
|
import 'dotenv/config'; // Loads environment variables from a .env file into process.env, allowing secure storage of sensitive information
|
||||||
|
import llamaTokenizer from 'llama-tokenizer-js'; // A library for tokenizing text, which is crucial for managing the length of text inputs to the AI model
|
||||||
|
|
||||||
|
// Define a prompt that will guide the AI's behavior when analyzing NGINX logs for potential security issues
|
||||||
|
const prompt = `
|
||||||
|
You are a security AI responsible for analyzing web traffic from NGINX logs and blocking malicious IPs. Your task is to review the logs for potential attacks and issues. If you identify a verified problem, include [ALERT] followed by a detailed description of the issue in your response. Ensure your message is formatted in Markdown compatible with Discord.
|
||||||
|
|
||||||
|
**Guidelines:**
|
||||||
|
- Domains on the server: shells.lol, hehe.rest, dcord.rest, nodejs.lol, dht.rest, my-mc.link, ssh.surf, tcp.quest
|
||||||
|
- Use emojis to enhance communication.
|
||||||
|
- Do not report IP addresses for scraping or crawling.
|
||||||
|
- Ignore IPs: x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x. Do not mention these in reports.
|
||||||
|
- Ignore IP addresses with BOGONs such as 192.168.0.1 or 127.0.0.2, etc.
|
||||||
|
- Avoid alerting for false positives or irregular activity.
|
||||||
|
- If there are no alerts but you have interesting findings, write: GENERAL followed by your insights in Markdown.
|
||||||
|
- Only send GENERAL messages for noteworthy events, not for routine traffic reports.
|
||||||
|
- In a GENERAL message, feel free to provide a long explainer on your deductions.
|
||||||
|
- Be decisive. If an IP is being malicious, block it. Do not monitor IPs for further traffic.
|
||||||
|
- Do not ban an IP address without a corresponding log entry, provide this in your response.
|
||||||
|
- Block all bot IPs and information scanners except Google.
|
||||||
|
- Provide proof and reasoning for each ban.
|
||||||
|
- DO NOT BAN AN IP THAT IS NOT IN A LOG EVER! YOU MUST SEE THE IP ADDRESS!
|
||||||
|
- To ban an IP or flag it as a security risk, wrap it in a Markdown spoiler: ||IPHERE||
|
||||||
|
`;
|
||||||
|
|
||||||
|
// Initialize the Express application and define the port on which the server will run
|
||||||
|
const app = express(); // Create an instance of an Express application
|
||||||
|
const port = 3001; // Define the port number for the server, 3001 is commonly used for development
|
||||||
|
|
||||||
|
// Middleware to enable CORS for all routes
|
||||||
|
app.use(cors()); // This allows the server to accept requests from any origin, useful for APIs that may be accessed by web applications from different domains
|
||||||
|
|
||||||
|
// Set a larger limit for the request body to handle large data payloads
|
||||||
|
app.use(bodyParser.json({ limit: '50mb' })); // The JSON body parser is configured with a 50MB limit, suitable for handling large JSON payloads
|
||||||
|
|
||||||
|
// Define constants for the application, used to control various aspects of the server's behavior
|
||||||
|
const TIMEOUT_DURATION = 100000; // The maximum time (in milliseconds) the server will wait before timing out a request, set to 100 seconds
|
||||||
|
const MAX_TOKENS = 8000; // The maximum number of tokens (words and punctuation) allowed in a conversation, this limit helps manage API usage
|
||||||
|
const TOLERANCE = 100; // A buffer value used to prevent exceeding the MAX_TOKENS limit, ensuring the conversation stays within safe bounds
|
||||||
|
let conversationHistory = {}; // An object to store conversation history for each IP address, allowing the server to maintain context for each user
|
||||||
|
|
||||||
|
// Helper function to get the current timestamp in a formatted string
|
||||||
|
const getTimestamp = () => {
|
||||||
|
const now = new Date(); // Get the current date and time
|
||||||
|
const date = now.toLocaleDateString('en-US'); // Format the date in the US locale
|
||||||
|
const time = now.toLocaleTimeString('en-US'); // Format the time in the US locale
|
||||||
|
return `${date} [${time}]`; // Return the formatted date and time as a string
|
||||||
|
};
|
||||||
|
|
||||||
|
// Middleware to track conversation history based on the client's IP address
|
||||||
|
app.use((req, res, next) => {
|
||||||
|
// Extract the client's IP address from various possible headers (CF-Connecting-IP, X-Forwarded-For, X-Real-IP) or fallback to req.ip
|
||||||
|
const ip = req.headers['cf-connecting-ip'] || req.headers['x-forwarded-for'] || req.headers['x-real-ip'] || req.ip;
|
||||||
|
req.clientIp = ip; // Store the client's IP address in the request object for easy access later
|
||||||
|
|
||||||
|
// Log the incoming request along with the client's IP address and current timestamp
|
||||||
|
console.log(`${getTimestamp()} [INFO] Incoming request from IP: ${req.clientIp}`);
|
||||||
|
|
||||||
|
// If this IP address has not been seen before, initialize a new conversation history for it
|
||||||
|
if (!conversationHistory[req.clientIp]) {
|
||||||
|
console.log(`${getTimestamp()} [INFO] Initializing conversation history for new IP: ${req.clientIp}`);
|
||||||
|
// Start the conversation with the predefined prompt that instructs the AI on how to analyze the logs
|
||||||
|
conversationHistory[req.clientIp] = [
|
||||||
|
{ role: 'system', content: prompt }
|
||||||
|
];
|
||||||
|
}
|
||||||
|
next(); // Move on to the next middleware or route handler
|
||||||
|
});
|
||||||
|
|
||||||
|
// Function to count the number of tokens in a conversation history using the llama tokenizer
|
||||||
|
async function countLlamaTokens(messages) {
|
||||||
|
let totalTokens = 0; // Initialize a counter for the total number of tokens
|
||||||
|
for (const message of messages) {
|
||||||
|
// Only count tokens for user and assistant messages, not system messages
|
||||||
|
if (message.role === 'user' || message.role === 'assistant') {
|
||||||
|
const encodedTokens = llamaTokenizer.encode(message.content); // Tokenize the message content
|
||||||
|
totalTokens += encodedTokens.length; // Add the number of tokens in the current message to the total
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return totalTokens; // Return the total number of tokens
|
||||||
|
}
|
||||||
|
|
||||||
|
// Function to trim the conversation history to fit within the token limit
|
||||||
|
async function trimConversationHistory(messages, maxLength, tolerance) {
|
||||||
|
let tokenLength = await countLlamaTokens(messages); // Get the total number of tokens in the conversation
|
||||||
|
// Continue trimming the conversation history until it's within the allowed token limit
|
||||||
|
while (tokenLength > maxLength - tolerance && messages.length > 1) {
|
||||||
|
messages.splice(1, 1); // Remove the oldest user/assistant message (the second item in the array)
|
||||||
|
tokenLength = await countLlamaTokens(messages); // Recalculate the total number of tokens after trimming
|
||||||
|
console.log(`${getTimestamp()} [CLEANUP] Trimmed conversation history to ${tokenLength} tokens.`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Function to scrape a web page and extract its content
|
||||||
|
async function scrapeWebPage(url) {
|
||||||
|
console.log(`${getTimestamp()} [INFO] Starting to scrape URL: ${url}`);
|
||||||
|
try {
|
||||||
|
// Perform an HTTP GET request to fetch the content of the specified URL
|
||||||
|
const res = await axios.get(url);
|
||||||
|
const html = res.data; // Extract the HTML content from the response
|
||||||
|
const $ = cheerio.load(html); // Load the HTML into Cheerio for parsing and manipulation
|
||||||
|
|
||||||
|
// Extract specific elements from the HTML: the page title, meta description, and body content
|
||||||
|
const pageTitle = $('head title').text().trim(); // Get the text of the <title> tag
|
||||||
|
const pageDescription = $('head meta[name="description"]').attr('content'); // Get the content of the meta description
|
||||||
|
const pageContent = $('body').text().trim(); // Get all text content within the <body> tag
|
||||||
|
|
||||||
|
// Construct a response message with the extracted details
|
||||||
|
let response = `Title: ${pageTitle}\n`; // Start with the page title
|
||||||
|
if (pageDescription) {
|
||||||
|
response += `Description: ${pageDescription}\n`; // Add the meta description if it exists
|
||||||
|
}
|
||||||
|
if (pageContent) {
|
||||||
|
const MAX_CONTENT_LENGTH = process.env.MAX_CONTENT_LENGTH || 2000; // Set a maximum length for the content
|
||||||
|
// Clean the page content to remove unnecessary whitespace and special characters
|
||||||
|
let plainTextContent = $('<div>').html(pageContent).text().trim().replace(/[\r\n\t]+/g, ' ');
|
||||||
|
// Define a regular expression pattern to identify code-like content
|
||||||
|
const codePattern = /\/\/|\/\*|\*\/|\{|\}|\[|\]|\bfunction\b|\bclass\b|\b0x[0-9A-Fa-f]+\b|\b0b[01]+\b/;
|
||||||
|
const isCode = codePattern.test(plainTextContent); // Check if the content resembles code
|
||||||
|
|
||||||
|
if (isCode) {
|
||||||
|
plainTextContent = plainTextContent.replace(codePattern, ''); // Remove code-like patterns if detected
|
||||||
|
}
|
||||||
|
// Further clean the content by removing text within parentheses
|
||||||
|
plainTextContent = plainTextContent.replace(/ *\([^)]*\) */g, '');
|
||||||
|
// If the content is too long, truncate it and add an ellipsis
|
||||||
|
if (plainTextContent.length > MAX_CONTENT_LENGTH) {
|
||||||
|
plainTextContent = plainTextContent.substring(0, MAX_CONTENT_LENGTH) + '...';
|
||||||
|
}
|
||||||
|
response += `Content: ${plainTextContent.trim()}`; // Add the cleaned and possibly truncated content to the response
|
||||||
|
}
|
||||||
|
response += `\nURL: ${url}`; // Include the original URL in the response
|
||||||
|
|
||||||
|
console.log(`${getTimestamp()} [INFO] Successfully scraped URL: ${url}`);
|
||||||
|
return response; // Return the constructed response
|
||||||
|
} catch (err) {
|
||||||
|
// If the scraping process fails, log an error with details and return null
|
||||||
|
console.error(`${getTimestamp()} [ERROR] Failed to scrape URL: ${url}`, err);
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Function to process incoming requests, handle AI interactions, and return a response
|
||||||
|
async function processRequest(req, res) {
|
||||||
|
const startTime = Date.now(); // Record the start time of the request processing for performance tracking
|
||||||
|
const ip = req.clientIp; // Retrieve the client's IP address from the request object
|
||||||
|
console.log(`${getTimestamp()} [INFO] Handling chat request from IP: ${ip}`); // Log the request details
|
||||||
|
|
||||||
|
// Set a timeout for the request processing, ensuring it doesn't hang indefinitely
|
||||||
|
const timeout = setTimeout(() => {
|
||||||
|
console.error(`${getTimestamp()} [ERROR] Request timed out for IP: ${ip}`);
|
||||||
|
res.status(408).json({ message: "Request timed out" }); // Send a timeout response if the processing takes too long
|
||||||
|
}, TIMEOUT_DURATION);
|
||||||
|
|
||||||
|
try {
|
||||||
|
let userMessage = req.body.message; // Extract the user's message from the request body
|
||||||
|
console.log(`${getTimestamp()} [INFO] Received user message: ${userMessage}`);
|
||||||
|
userMessage = req.body.message + `\nDate/Time:${getTimestamp()}`; // Append the current date and time to the user's message
|
||||||
|
|
||||||
|
// Initialize conversation history if it doesn't exist for the IP
|
||||||
|
if (!conversationHistory[ip]) {
|
||||||
|
console.log(`${getTimestamp()} [INFO] Initializing conversation history for new IP: ${ip}`);
|
||||||
|
conversationHistory[ip] = [{ role: 'system', content: prompt }]; // Start the conversation with the predefined prompt
|
||||||
|
}
|
||||||
|
// Add the user's message to the conversation history for the IP
|
||||||
|
conversationHistory[ip].push({ role: 'user', content: userMessage });
|
||||||
|
|
||||||
|
// Trim the conversation history if it exceeds the token limit
|
||||||
|
await trimConversationHistory(conversationHistory[ip], MAX_TOKENS, TOLERANCE);
|
||||||
|
|
||||||
|
// Split the user's message into individual log lines
|
||||||
|
const logLines = userMessage.split('\n');
|
||||||
|
// Define a regex pattern to identify lines containing client IP addresses
|
||||||
|
const clientIpRegex = /\[Client (\d{1,3}\.){3}\d{1,3}\]/;
|
||||||
|
// Filter the log lines to only include those with valid client IP addresses
|
||||||
|
const filteredLogLines = logLines.filter(line => clientIpRegex.test(line));
|
||||||
|
|
||||||
|
// If no valid IP addresses are found in the log lines, send a response indicating this
|
||||||
|
if (filteredLogLines.length === 0) {
|
||||||
|
console.log(`${getTimestamp()} [INFO] No valid client IP addresses found in the log.`);
|
||||||
|
res.json({ message: "No valid client IP addresses found in the log." });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Join the filtered log lines back into a single string for processing
|
||||||
|
const filteredMessage = filteredLogLines.join('\n');
|
||||||
|
|
||||||
|
// Send the request to the llama API for processing and response generation
|
||||||
|
console.log(`${getTimestamp()} [INFO] Sending request to llama API for response`);
|
||||||
|
const response = await axios.post('http://127.0.0.1:8003/v1/chat/completions', {
|
||||||
|
model: 'gpt-3.5-turbo', // Specify the AI model to use
|
||||||
|
messages: [...conversationHistory[ip], { role: 'user', content: filteredMessage }] // Include the conversation history and the filtered message
|
||||||
|
});
|
||||||
|
// Extract the AI's response from the API's response data
|
||||||
|
const assistantMessage = response.data.choices[0].message;
|
||||||
|
// Add the AI's response to the conversation history
|
||||||
|
conversationHistory[ip].push(assistantMessage);
|
||||||
|
|
||||||
|
// Log the AI's response and additional details like the finish reason and token usage
|
||||||
|
console.log(`${getTimestamp()} [INFO] Received response from llama API: ${assistantMessage.content}`);
|
||||||
|
console.log(`${getTimestamp()} [DEBUG] Finish Reason: ${response.data.choices[0].finish_reason}`);
|
||||||
|
console.log(`${getTimestamp()} [STATS] Usage: prompt_tokens=${response.data.usage.prompt_tokens}, completion_tokens=${response.data.usage.completion_tokens}, total_tokens=${response.data.usage.total_tokens}`);
|
||||||
|
|
||||||
|
clearTimeout(timeout); // Clear the timeout to prevent it from triggering
|
||||||
|
res.json(assistantMessage); // Send the AI's response back to the client
|
||||||
|
} catch (error) {
|
||||||
|
// If an error occurs during request processing, log the error and send a 500 response
|
||||||
|
console.error(`${getTimestamp()} [ERROR] An error occurred while handling chat request`, error);
|
||||||
|
clearTimeout(timeout); // Clear the timeout to prevent it from triggering
|
||||||
|
res.status(500).json({ message: "An error occurred", error: error.message }); // Send an error response
|
||||||
|
} finally {
|
||||||
|
// Record the end time and calculate the total processing time in seconds
|
||||||
|
const endTime = Date.now();
|
||||||
|
const processingTime = ((endTime - startTime) / 1000).toFixed(2); // Convert milliseconds to seconds
|
||||||
|
console.log(`${getTimestamp()} [STATS] Processing Time: ${processingTime} seconds`); // Log the processing time
|
||||||
|
console.log(`${getTimestamp()} [INFO] Finished processing chat request for IP: ${ip}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Route to handle incoming chat requests, trim the message content, and process the request
|
||||||
|
app.post('/api/v1/chat', async (req, res) => {
|
||||||
|
// Trim the incoming message to fit within token limits
|
||||||
|
const messageContent = req.body.message; // Get the user's message from the request body
|
||||||
|
const encodedTokens = llamaTokenizer.encode(messageContent); // Tokenize the message to determine its length in tokens
|
||||||
|
const MAX_MESSAGE_TOKENS = MAX_TOKENS - (await countLlamaTokens([{ role: 'system', content: prompt }])) - TOLERANCE; // Calculate the maximum allowed tokens for the user's message
|
||||||
|
|
||||||
|
// If the message exceeds the allowed token limit, trim it to fit
|
||||||
|
let trimmedMessageContent = messageContent;
|
||||||
|
if (encodedTokens.length > MAX_MESSAGE_TOKENS) {
|
||||||
|
trimmedMessageContent = llamaTokenizer.decode(encodedTokens.slice(0, MAX_MESSAGE_TOKENS)); // Truncate the message and decode it back to a string
|
||||||
|
}
|
||||||
|
|
||||||
|
// Process the trimmed message and send the response
|
||||||
|
await processRequest({ ...req, body: { message: trimmedMessageContent } }, res);
|
||||||
|
});
|
||||||
|
|
||||||
|
// Route to fetch the conversation history for a specific IP address
|
||||||
|
app.get('/api/v1/conversation-history', (req, res) => {
|
||||||
|
const ip = req.clientIp; // Get the client's IP address from the request object
|
||||||
|
console.log(`${getTimestamp()} [INFO] Fetching conversation history for IP: ${ip}`); // Log the request details
|
||||||
|
res.json(conversationHistory[ip]); // Send the conversation history for the IP as a JSON response
|
||||||
|
});
|
||||||
|
|
||||||
|
// Route to restart the core AI service via Docker, typically used to refresh the model or resolve issues
|
||||||
|
app.post('/api/v1/restart-core', (req, res) => {
|
||||||
|
console.log(`${getTimestamp()} [INFO] Restarting core service`); // Log the restart action
|
||||||
|
cmd(`docker restart llama-gpu-server`).then(out => { // Execute a shell command to restart the Docker container running the AI model
|
||||||
|
console.log(`${getTimestamp()} [INFO] Core service restarted`); // Log the successful restart
|
||||||
|
res.json(out.stdout); // Send the output of the restart command back to the client
|
||||||
|
}).catch(err => { // Handle any errors that occur during the restart
|
||||||
|
console.error(`${getTimestamp()} [ERROR] Failed to restart core service`, err); // Log the error
|
||||||
|
res.status(500).json({ message: "An error occurred while restarting the core service", error: err.message }); // Send an error response
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// Route to reset the conversation history for a specific IP address, effectively starting a new session
|
||||||
|
app.post('/api/v1/reset-conversation', (req, res) => {
|
||||||
|
const ip = req.clientIp; // Get the client's IP address from the request object
|
||||||
|
console.log(`${getTimestamp()} [INFO] Resetting conversation history for IP: ${ip}`); // Log the reset action
|
||||||
|
|
||||||
|
// Reset the conversation history to its initial state for the given IP
|
||||||
|
conversationHistory[ip] = [
|
||||||
|
{ role: 'system', content: prompt }
|
||||||
|
];
|
||||||
|
console.log(`${getTimestamp()} [INFO] Conversation history reset for IP: ${ip}`); // Log the successful reset
|
||||||
|
res.json({ message: "Conversation history reset for IP: " + ip }); // Send a confirmation message back to the client
|
||||||
|
});
|
||||||
|
|
||||||
|
// Start the Express server on the defined port, making the API available for requests
|
||||||
|
app.listen(port, () => {
|
||||||
|
console.log(`${getTimestamp()} [INFO] Server running at http://localhost:${port}`); // Log the server startup and its URL
|
||||||
|
});
|
||||||
Loading…
Reference in New Issue
Block a user