Agent Data Collector

A minimal, fast Go application for collecting data from AI agents with resumable uploads and agent-based access control.

Components

• agentd - Server (Go, runs in Docker)
• agentc - Client CLI (Go, single binary)

Features

• Resumable uploads: Chunked uploads with automatic resume on interruption
• Agent authentication: Pre-shared secrets per agent
• Cross-agent data access: Agents can download data from other agents
• Fast: Parallel chunk uploads, minimal overhead
• Small: Single binary, no external dependencies (except SQLite)
• Docker ready: Multi-stage build, ~15MB image

Quick Start

Build

go build -o bin/agentd ./cmd/agentd
go build -o bin/agentc ./cmd/agentc

Run Server

# Direct
AGENT_SECRETS='{"agent1":"secret1","agent2":"secret2"}' ./bin/agentd -data-dir ./data -port 8080

# Docker
docker run -d -p 8080:8080 -v $(pwd)/data:/data -e AGENT_SECRETS='{"agent1":"secret1"}' agentd

Client Usage

# Upload file
./bin/agentc -server http://localhost:8080 -agent agent1 -secret secret1 -file /path/to/data.bin

# Upload with custom chunk size and workers
./bin/agentc -server http://localhost:8080 -agent agent1 -secret secret1 -file data.bin -chunk-size 1024 -workers 4

# Resume interrupted upload (automatic)
./bin/agentc -server http://localhost:8080 -agent agent1 -secret secret1 -file data.bin

# List files from another agent
./bin/agentc -server http://localhost:8080 -agent agent1 -secret secret1 -list agent2

# Download file from another agent
./bin/agentc -server http://localhost:8080 -agent agent1 -secret secret1 -download <file_id> -from-agent agent2 -output /path/to/save.bin

API

Server Endpoints

Method	Path	Description
GET	/health	Health check
POST	/upload	Initialize upload (returns upload_id)
PUT	/upload/{id}/chunk	Upload chunk (requires Content-Range header)
POST	/upload/{id}/complete	Complete upload
GET	/upload/{id}	Get upload status
GET	/data/{agent_id}	List files from agent
GET	/data/{agent_id}/{file_id}	Download file

Authentication

All endpoints (except /health and GET /upload/{id}) require Authorization: Bearer <agent_id>:<secret> header.

Chunk Upload

Chunks must be uploaded with Content-Range: bytes <start>-<end>/<total> header.

Example:

Content-Range: bytes 0-1048575/10485760

Configuration

Server (Environment Variables)

Variable	Default	Description
PORT	8080	HTTP port
DATA_DIR	/data	Data directory
AGENT_SECRETS	-	JSON map of agent_id:secret
AGENT_SECRETS_FILE	-	Path to JSON secrets file
MAX_FILE_SIZE	10GB	Maximum upload size

Client Flags

Flag	Default	Description
-server	(required)	Server URL
-agent	(required)	Agent ID
-secret	(required)	Agent secret
-file	-	File to upload
-chunk-size	1024	Chunk size in KB
-workers	4	Parallel upload workers
-resume	true	Resume interrupted upload
-list	-	List files from agent
-download	-	Download file by ID
-from-agent	(your agent)	Source agent for download
-output	-	Output file for download

Architecture

• Storage: Files stored on disk in chunks, metadata in SQLite
• Resumability: Each chunk stored separately, progress tracked in DB
• Concurrency: Per-upload mutex for chunk serialization, parallel across uploads
• Security: Bcrypt-hashed secrets, agent isolation

License

MIT