Log Ingester - RAG Data Pipeline
Custom RAG (Retrieval-Augmented Generation) service. Ingests data from four sources every 5 minutes, creates embeddings via Ollama, and stores them in ChromaDB. Provides semantic search API for Open WebUI.
Architecture
[Frigate SQLite] --\
[HA PostgreSQL] ---+--> [ingest.py] --> [Ollama nomic-embed-text] --> [ChromaDB]
[home-assistant.log] -/
[Docker logs] --/
[Open WebUI Tool] --> [search_api.py] --> [ChromaDB] --> semantic results
Files
| File |
Description |
ingest.py |
Main ingestion loop, runs every INGEST_INTERVAL seconds |
search_api.py |
FastAPI semantic search endpoint |
openwebui_tool.py |
Open WebUI tool plugin |
state/state.json |
Persistent state (last processed ID, log position, timestamp) |
Environment Variables
| Variable |
Default |
Description |
CHROMA_HOST |
chromadb |
ChromaDB host |
CHROMA_PORT |
8000 |
ChromaDB port |
OLLAMA_URL |
http://ollama:11434 |
Ollama URL |
FRIGATE_DB |
/data/frigate/frigate.db |
Frigate database path |
HA_LOG |
/data/ha/home-assistant.log |
HA log file path |
STATE_FILE |
/data/state/state.json |
State file path |
INGEST_INTERVAL |
300 |
Interval in seconds |
PG_HOST |
postgres18 |
PostgreSQL host (HA recorder) |
PG_DB |
homeassistant |
HA database name |
PG_PASS |
- |
PostgreSQL password (required) |
ChromaDB Collections
| Collection |
Contents |
frigate_events |
Detection events (label, camera, timestamp, score) |
ha_history |
HA entity state changes (rolling 7-day window) |
ha_logs |
25-line blocks from home-assistant.log |
docker_logs |
Logs from: homeassistant, frigate, mosquitto, esphome, traefik, crowdsec, immich_server, ha-dashboard |
Useful Commands
# Follow ingestion logs
docker compose logs -f log-ingester
# Check processed state
cat /home/seba/mydocker/log-ingester/state/state.json
# Test search API
curl -X POST http://localhost:8765/search \
-H "Content-Type: application/json" \
-d '{"query": "who was at the door yesterday", "n_results": 5}'
# Collection statistics
curl http://localhost:8765/stats
# Rebuild
docker compose build log-ingester && docker compose up -d log-ingester