update
This commit is contained in:
Raven Scott
@@ -0,0 +1,378 @@
|
||||
# autobase-discovery-cli - Service Discovery for Autobase
|
||||
|
||||
## Overview
|
||||
|
||||
autobase-discovery-cli is a command-line interface for autobase-discovery, providing tools to run discovery servers and query service information from Autobase databases. It enables decentralized service discovery by leveraging the Autobase multi-writer architecture.
|
||||
|
||||
### Key Features
|
||||
|
||||
- **Server mode**: Run a discovery service that indexes Autobase entries
|
||||
- **Client queries**: List services by name from the discovery database
|
||||
- **Secure RPC**: Public-key authenticated remote procedure calls
|
||||
- **Real-time updates**: Database key stabilizes as indexers process entries
|
||||
- **JSON logging**: Structured logs compatible with pino ecosystem
|
||||
|
||||
### Use Cases
|
||||
|
||||
- **Service registry**: Discover microservices in distributed systems
|
||||
- **Peer discovery**: Find nodes in P2P networks
|
||||
- **Dynamic configuration**: Query runtime service endpoints
|
||||
- **Multi-tenant discovery**: Isolate services by namespace
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Discovery Server"
|
||||
CLI[autodiscovery CLI]
|
||||
RPC[RPC Server<br/>Auth-required]
|
||||
INDEXER[Autobase Indexer]
|
||||
DB[Discovery Database]
|
||||
end
|
||||
|
||||
subgraph "Clients"
|
||||
CLIENT[autodiscovery-client]
|
||||
QUERY[List Services Query]
|
||||
end
|
||||
|
||||
subgraph "Autobase Network"
|
||||
AB[Autobase Writers]
|
||||
INPUT[Service Entries]
|
||||
end
|
||||
|
||||
AB -->|Replicate| INDEXER
|
||||
INPUT --> AB
|
||||
INDEXER --> DB
|
||||
DB --> CLI
|
||||
CLI --> RPC
|
||||
CLIENT --> QUERY
|
||||
QUERY -->|RPC| RPC
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
npm install -g autobase-discovery-cli
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Run Discovery Server
|
||||
|
||||
```bash
|
||||
# Generate a keypair for secure RPC
|
||||
# (Use hypercore-sign or similar to generate keys)
|
||||
autodiscovery run <rpc-allowed-public-key> | pino-pretty
|
||||
```
|
||||
|
||||
The server will output:
|
||||
- RPC server's public key
|
||||
- Discovery database key (updates as indexers process)
|
||||
|
||||
### Query Services (Client)
|
||||
|
||||
```bash
|
||||
# List all instances of a service
|
||||
autodiscovery-client list <database-key> <service-name>
|
||||
```
|
||||
|
||||
## CLI Reference
|
||||
|
||||
### Server Commands
|
||||
|
||||
#### `autodiscovery run <rpc-allowed-public-key>`
|
||||
|
||||
Start the discovery server.
|
||||
|
||||
**Parameters:**
|
||||
- `rpc-allowed-public-key`: Public key authorized to make RPC calls
|
||||
|
||||
**Output:**
|
||||
- RPC server public key
|
||||
- Database key (stabilizes after first indexer entry)
|
||||
|
||||
**Logging:**
|
||||
```bash
|
||||
# JSON format (default)
|
||||
autodiscovery run <pubkey>
|
||||
|
||||
# Human-readable
|
||||
autodiscovery run <pubkey> | pino-pretty
|
||||
```
|
||||
|
||||
**Note:** The database key updates as new indexers are processed. Add at least one entry to stabilize the initial database key.
|
||||
|
||||
### Client Commands
|
||||
|
||||
#### `autodiscovery-client list <database-key> <service-name>`
|
||||
|
||||
List all discovered instances of a service.
|
||||
|
||||
**Parameters:**
|
||||
- `database-key`: The discovery database key
|
||||
- `service-name`: Name of the service to query
|
||||
|
||||
**Output:**
|
||||
```
|
||||
service-instance-1 192.168.1.100:8080
|
||||
service-instance-2 192.168.1.101:8080
|
||||
```
|
||||
|
||||
#### `autodiscovery-client --help`
|
||||
|
||||
Show all available commands and options.
|
||||
|
||||
## Complete Examples
|
||||
|
||||
### Example 1: Service Discovery Setup
|
||||
|
||||
```bash
|
||||
# 1. Generate RPC keypair (signer)
|
||||
hypercore-sign generate-keys
|
||||
# Note the public key: ocmjxpzg...
|
||||
|
||||
# 2. Start discovery server
|
||||
autodiscovery run ocmjxpzghcx5gbhkky7qubn5pr4fpcxwr5mu4hjw43dqs3qhid3y | pino-pretty
|
||||
|
||||
# Output:
|
||||
# Server public key: abc123...
|
||||
# Database key: def456...
|
||||
# Waiting for indexers...
|
||||
|
||||
# 3. Add a service entry to Autobase (via your app)
|
||||
# (See autobase-discovery module for API)
|
||||
|
||||
# 4. Query for services
|
||||
autodiscovery-client list def456... api-gateway
|
||||
|
||||
# Output:
|
||||
# api-gateway-prod-1 10.0.1.10:3000
|
||||
# api-gateway-prod-2 10.0.1.11:3000
|
||||
```
|
||||
|
||||
### Example 2: Multi-Service Discovery
|
||||
|
||||
```bash
|
||||
# Terminal 1: Run discovery server
|
||||
export RPC_PUBKEY="ocmjxpzghcx5gbhkky7qubn5pr4fpcxwr5mu4hjw43dqs3qhid3y"
|
||||
autodiscovery run $RPC_PUBKEY 2>&1 | tee discovery.log | pino-pretty
|
||||
|
||||
# Extract database key from logs (after first entry)
|
||||
DB_KEY=$(grep "Database key" discovery.log | tail -1 | awk '{print $3}')
|
||||
|
||||
# Terminal 2: Query different services
|
||||
autodiscovery-client list $DB_KEY user-service
|
||||
autodiscovery-client list $DB_KEY payment-service
|
||||
autodiscovery-client list $DB_KEY notification-service
|
||||
```
|
||||
|
||||
### Example 3: Integration Script
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# start-discovery.sh
|
||||
|
||||
RPC_KEY="${RPC_KEY:-$(cat ~/.discovery-rpc-key.pub)}"
|
||||
LOG_FILE="/var/log/discovery.log"
|
||||
|
||||
# Start server with log rotation
|
||||
autodiscovery run "$RPC_KEY" >> "$LOG_FILE" 2>&1 &
|
||||
PID=$!
|
||||
|
||||
# Wait for database key
|
||||
for i in {1..30}; do
|
||||
DB_KEY=$(grep -o 'Database key: [a-f0-9]*' "$LOG_FILE" | tail -1 | awk '{print $3}')
|
||||
if [ -n "$DB_KEY" ]; then
|
||||
echo "Discovery server ready"
|
||||
echo "Database key: $DB_KEY"
|
||||
echo $DB_KEY > /run/discovery-db-key
|
||||
break
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
|
||||
echo $PID > /run/discovery.pid
|
||||
```
|
||||
|
||||
### Example 4: Client Wrapper
|
||||
|
||||
```js
|
||||
#!/usr/bin/env node
|
||||
const { execSync } = require('child_process')
|
||||
|
||||
class DiscoveryClient {
|
||||
constructor(dbKey) {
|
||||
this.dbKey = dbKey
|
||||
}
|
||||
|
||||
list(serviceName) {
|
||||
try {
|
||||
const output = execSync(
|
||||
`autodiscovery-client list ${this.dbKey} ${serviceName}`,
|
||||
{ encoding: 'utf8' }
|
||||
)
|
||||
|
||||
return output.trim().split('\n').map(line => {
|
||||
const [name, address] = line.trim().split(/\s+/)
|
||||
return { name, address }
|
||||
})
|
||||
} catch (err) {
|
||||
console.error('Query failed:', err.message)
|
||||
return []
|
||||
}
|
||||
}
|
||||
|
||||
getServiceUrl(serviceName, index = 0) {
|
||||
const instances = this.list(serviceName)
|
||||
if (instances.length === 0) {
|
||||
throw new Error(`No instances found for service: ${serviceName}`)
|
||||
}
|
||||
return instances[index % instances.length].address
|
||||
}
|
||||
}
|
||||
|
||||
// Usage
|
||||
const client = new DiscoveryClient(process.env.DISCOVERY_DB_KEY)
|
||||
|
||||
const userServiceUrl = client.getServiceUrl('user-service')
|
||||
console.log(`User service: http://${userServiceUrl}`)
|
||||
|
||||
const allGateways = client.list('api-gateway')
|
||||
console.log(`Found ${allGateways.length} API gateways`)
|
||||
```
|
||||
|
||||
## Security
|
||||
|
||||
### RPC Authentication
|
||||
|
||||
The discovery server requires a public key for RPC authentication:
|
||||
|
||||
```bash
|
||||
# Generate keypair
|
||||
hypercore-sign generate-keys
|
||||
|
||||
# Use public key when starting server
|
||||
autodiscovery run <public-key>
|
||||
|
||||
# Only clients with matching secret key can query
|
||||
```
|
||||
|
||||
### Network Security
|
||||
|
||||
- Run behind firewall for internal networks
|
||||
- Use TLS for public deployments
|
||||
- Rotate RPC keys periodically
|
||||
|
||||
## Integration with Autobase
|
||||
|
||||
### Service Entry Format
|
||||
|
||||
Entries written to Autobase for discovery:
|
||||
|
||||
```js
|
||||
{
|
||||
type: 'service-announcement',
|
||||
name: 'user-service',
|
||||
instance: 'user-service-prod-1',
|
||||
address: '10.0.1.10:3000',
|
||||
metadata: {
|
||||
version: '2.1.0',
|
||||
region: 'us-east-1',
|
||||
health: 'healthy'
|
||||
},
|
||||
timestamp: Date.now()
|
||||
}
|
||||
```
|
||||
|
||||
### Indexer Configuration
|
||||
|
||||
The discovery server indexes these entries:
|
||||
|
||||
```js
|
||||
// In your Autobase setup
|
||||
const Autobase = require('autobase')
|
||||
const DiscoveryIndexer = require('autobase-discovery')
|
||||
|
||||
const base = new Autobase(store, key, {
|
||||
async apply(nodes, view, base) {
|
||||
for (const node of nodes) {
|
||||
if (node.value.type === 'service-announcement') {
|
||||
await view.insert(node.value)
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Database Key Management
|
||||
|
||||
```bash
|
||||
# Wait for key stabilization
|
||||
start_server() {
|
||||
autodiscovery run $RPC_KEY &
|
||||
SERVER_PID=$!
|
||||
|
||||
# Wait for database key
|
||||
while [ -z "$DB_KEY" ]; do
|
||||
sleep 1
|
||||
DB_KEY=$(get_db_key_from_logs)
|
||||
done
|
||||
|
||||
# Store for clients
|
||||
echo $DB_KEY > /etc/discovery/db-key
|
||||
}
|
||||
```
|
||||
|
||||
### Monitoring
|
||||
|
||||
```bash
|
||||
# Check server health
|
||||
check_discovery() {
|
||||
if pgrep -f "autodiscovery run" > /dev/null; then
|
||||
echo "Server running"
|
||||
|
||||
# Test query
|
||||
DB_KEY=$(cat /etc/discovery/db-key)
|
||||
autodiscovery-client list $DB_KEY health-check > /dev/null && \
|
||||
echo "Query OK" || echo "Query failed"
|
||||
else
|
||||
echo "Server not running"
|
||||
fi
|
||||
}
|
||||
```
|
||||
|
||||
### Log Analysis
|
||||
|
||||
```bash
|
||||
# Parse discovery events
|
||||
cat /var/log/discovery.log | pino-pretty | grep -E "(announce|unannounce|query)"
|
||||
|
||||
# Monitor for errors
|
||||
tail -f /var/log/discovery.log | jq 'select(.level >= 50)'
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Database Key Keeps Changing
|
||||
|
||||
**Cause:** Not enough entries processed by indexers
|
||||
**Solution:** Add at least one entry to stabilize the key
|
||||
|
||||
### RPC Authentication Fails
|
||||
|
||||
**Cause:** Wrong public key or key mismatch
|
||||
**Solution:** Verify keys match between server and client
|
||||
|
||||
### No Services Found
|
||||
|
||||
**Cause:** Services not announcing or wrong database key
|
||||
**Solution:** Check Autobase entries and database key
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0
|
||||
|
||||
---
|
||||
**Module Type**: CLI Tool | **Ecosystem Role**: Service Discovery | **Dependencies**: autobase-discovery, Autobase
|
||||
@@ -0,0 +1,68 @@
|
||||
# blind-peer-cli - Blind Peer CLI
|
||||
|
||||
## Overview
|
||||
|
||||
blind-peer-cli provides CLI commands to run blind peers. It supports standard node and Bare environments, with options for storage, trusted peers, and monitoring.
|
||||
|
||||
### Key Features
|
||||
|
||||
- **CLI runner**: Start blind peers easily
|
||||
- **Bare runtime support**: `blind-peer-bare`
|
||||
- **Trusted peers**: Allow announce permissions
|
||||
- **Autodiscovery**: Register in discovery service
|
||||
- **Prometheus**: Optional metrics integration
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
npm install -g blind-peer-cli
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
blind-peer
|
||||
```
|
||||
|
||||
### Bare Runtime
|
||||
|
||||
```bash
|
||||
blind-peer-bare
|
||||
```
|
||||
|
||||
## Command Line Options
|
||||
|
||||
- `--storage, -s [path]` Storage path (default: ./blind-peer)
|
||||
- `--port, -p [int]` DHT port (default: random)
|
||||
- `--trusted-peer, -t [key]` Trusted peer key (repeatable)
|
||||
- `--debug, -d` Enable debug logs (repeatable)
|
||||
- `--max-storage, -m [int]` Max storage in MB (default: 100000)
|
||||
- `--autodiscovery-rpc-key` Autodiscovery RPC public key
|
||||
- `--autodiscovery-seed` Seed for autodiscovery auth
|
||||
- `--autodiscovery-service-name` Service name (default: blind-peer)
|
||||
- `--scraper-public-key` Prometheus scraper public key
|
||||
- `--scraper-secret` Prometheus scraper secret
|
||||
- `--scraper-alias` Scraper alias
|
||||
|
||||
## Output
|
||||
|
||||
Emits NDJSON (pino) logs, e.g.:
|
||||
|
||||
```jsonl
|
||||
{"level":30,"msg":"Starting blind peer"}
|
||||
{"level":30,"msg":"Using storage 'blind-peer'"}
|
||||
{"level":30,"msg":"Blind peer listening"}
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Use `--storage` on persistent disks
|
||||
- Configure trusted peers if enabling announce
|
||||
- Pipe to `pino-pretty` for readability
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0
|
||||
|
||||
---
|
||||
**Module Type**: Tooling | **Ecosystem Role**: Blind Peer CLI | **Dependencies**: blind-peer
|
||||
@@ -0,0 +1,44 @@
|
||||
# blind-peering-cli - Blind Peering CLI
|
||||
|
||||
## Overview
|
||||
|
||||
blind-peering-cli provides a command-line interface for interacting with blind peers. It supports seeding Hypercores and Hyperdrives via blind-peer services.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
npm install -g blind-peering-cli
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
blind-peering --help
|
||||
```
|
||||
|
||||
### Seed a Hypercore
|
||||
|
||||
```bash
|
||||
blind-peering seed --core --blind-peer-key <blind-peer-rpc-key> <hypercore-key>
|
||||
```
|
||||
|
||||
### Seed via Autodiscovery
|
||||
|
||||
```bash
|
||||
blind-peering seed --core \
|
||||
--auto-disc-db <autobase-discovery-db-key> \
|
||||
--service-name <service-name> \
|
||||
<hypercore-key>
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Only trusted peers can request seeding
|
||||
- Use `blind-peering identity` to get your DHT public key
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0
|
||||
|
||||
---
|
||||
**Module Type**: Tooling | **Ecosystem Role**: Blind Peer Client | **Dependencies**: blind-peering
|
||||
Reference in New Issue
Block a user