Initial release of Claude Vision Auto v1.0.0
Vision-based auto-approval system for Claude Code CLI using MiniCPM-V vision model. Features: - Automatic detection and response to approval prompts - Screenshot capture and vision analysis via Ollama - Support for multiple screenshot tools (scrot, gnome-screenshot, etc.) - Configurable timing and behavior - Debug mode for troubleshooting - Comprehensive documentation Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Jean-Philippe Brule <jp@svrnty.io>
This commit is contained in:
Svrnty
commit
41cecca0e2
46
.gitignore
vendored
Normal file
46
.gitignore
vendored
Normal file
@ -0,0 +1,46 @@
|
|||||||
|
# Python
|
||||||
|
__pycache__/
|
||||||
|
*.py[cod]
|
||||||
|
*$py.class
|
||||||
|
*.so
|
||||||
|
.Python
|
||||||
|
build/
|
||||||
|
develop-eggs/
|
||||||
|
dist/
|
||||||
|
downloads/
|
||||||
|
eggs/
|
||||||
|
.eggs/
|
||||||
|
lib/
|
||||||
|
lib64/
|
||||||
|
parts/
|
||||||
|
sdist/
|
||||||
|
var/
|
||||||
|
wheels/
|
||||||
|
*.egg-info/
|
||||||
|
.installed.cfg
|
||||||
|
*.egg
|
||||||
|
|
||||||
|
# Virtual environments
|
||||||
|
venv/
|
||||||
|
ENV/
|
||||||
|
env/
|
||||||
|
|
||||||
|
# IDE
|
||||||
|
.vscode/
|
||||||
|
.idea/
|
||||||
|
*.swp
|
||||||
|
*.swo
|
||||||
|
*~
|
||||||
|
|
||||||
|
# Testing
|
||||||
|
.pytest_cache/
|
||||||
|
.coverage
|
||||||
|
htmlcov/
|
||||||
|
|
||||||
|
# Cache
|
||||||
|
.cache/
|
||||||
|
*.log
|
||||||
|
|
||||||
|
# Screenshots
|
||||||
|
screenshots/
|
||||||
|
*.png
|
||||||
45
CHANGELOG.md
Normal file
45
CHANGELOG.md
Normal file
@ -0,0 +1,45 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
All notable changes to Claude Vision Auto will be documented in this file.
|
||||||
|
|
||||||
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
||||||
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||||
|
|
||||||
|
## [1.0.0] - 2025-10-29
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- Initial release of Claude Vision Auto
|
||||||
|
- Vision-based auto-approval using MiniCPM-V
|
||||||
|
- Support for multiple screenshot tools (scrot, gnome-screenshot, ImageMagick, maim)
|
||||||
|
- Configurable timing and behavior via environment variables
|
||||||
|
- Debug mode for troubleshooting
|
||||||
|
- Comprehensive documentation (README, INSTALLATION, USAGE)
|
||||||
|
- MIT License
|
||||||
|
- Example usage scripts
|
||||||
|
- Basic test suite
|
||||||
|
|
||||||
|
### Features
|
||||||
|
- Automatic detection of approval prompts
|
||||||
|
- Screenshot capture of terminal window
|
||||||
|
- Vision analysis via Ollama API
|
||||||
|
- Intelligent response submission (1, y, WAIT)
|
||||||
|
- Configurable idle threshold and response delay
|
||||||
|
- Support for multiple vision models (MiniCPM-V, Llama 3.2 Vision, LLaVA)
|
||||||
|
- Automatic screenshot cleanup
|
||||||
|
- Connection testing and validation
|
||||||
|
|
||||||
|
### Supported Platforms
|
||||||
|
- Linux (Debian/Ubuntu tested)
|
||||||
|
- X11 display server
|
||||||
|
- Python 3.8+
|
||||||
|
|
||||||
|
## [Unreleased]
|
||||||
|
|
||||||
|
### Planned
|
||||||
|
- Wayland support
|
||||||
|
- macOS support
|
||||||
|
- Headless mode (API-only)
|
||||||
|
- Configurable response patterns
|
||||||
|
- Multi-terminal support
|
||||||
|
- Session recording and replay
|
||||||
|
- Windows support (WSL)
|
||||||
21
LICENSE
Normal file
21
LICENSE
Normal file
@ -0,0 +1,21 @@
|
|||||||
|
MIT License
|
||||||
|
|
||||||
|
Copyright (c) 2025 Svrnty (Jean-Philippe Brule)
|
||||||
|
|
||||||
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||||
|
of this software and associated documentation files (the "Software"), to deal
|
||||||
|
in the Software without restriction, including without limitation the rights
|
||||||
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||||
|
copies of the Software, and to permit persons to whom the Software is
|
||||||
|
furnished to do so, subject to the following conditions:
|
||||||
|
|
||||||
|
The above copyright notice and this permission notice shall be included in all
|
||||||
|
copies or substantial portions of the Software.
|
||||||
|
|
||||||
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||||
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||||
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||||
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||||
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||||
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||||
|
SOFTWARE.
|
||||||
37
Makefile
Normal file
37
Makefile
Normal file
@ -0,0 +1,37 @@
|
|||||||
|
.PHONY: install install-dev uninstall test clean help
|
||||||
|
|
||||||
|
help:
|
||||||
|
@echo "Claude Vision Auto - Makefile"
|
||||||
|
@echo ""
|
||||||
|
@echo "Available targets:"
|
||||||
|
@echo " install - Install the package"
|
||||||
|
@echo " install-dev - Install in development mode"
|
||||||
|
@echo " uninstall - Uninstall the package"
|
||||||
|
@echo " test - Run tests"
|
||||||
|
@echo " clean - Clean build artifacts"
|
||||||
|
@echo " deps - Install system dependencies (Debian/Ubuntu)"
|
||||||
|
@echo ""
|
||||||
|
|
||||||
|
install:
|
||||||
|
pip3 install -e .
|
||||||
|
|
||||||
|
install-dev:
|
||||||
|
pip3 install -e ".[dev]"
|
||||||
|
|
||||||
|
uninstall:
|
||||||
|
pip3 uninstall -y claude-vision-auto
|
||||||
|
|
||||||
|
deps:
|
||||||
|
@echo "Installing system dependencies..."
|
||||||
|
sudo apt-get update
|
||||||
|
sudo apt-get install -y scrot python3-pip
|
||||||
|
|
||||||
|
test:
|
||||||
|
python3 -m pytest tests/
|
||||||
|
|
||||||
|
clean:
|
||||||
|
rm -rf build/
|
||||||
|
rm -rf dist/
|
||||||
|
rm -rf *.egg-info
|
||||||
|
find . -type d -name __pycache__ -exec rm -rf {} +
|
||||||
|
find . -type f -name "*.pyc" -delete
|
||||||
294
README.md
Normal file
294
README.md
Normal file
@ -0,0 +1,294 @@
|
|||||||
|
# Claude Vision Auto
|
||||||
|
|
||||||
|
Vision-based auto-approval system for Claude Code CLI using MiniCPM-V vision model.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Claude Vision Auto automatically detects and responds to approval prompts in Claude Code by:
|
||||||
|
1. Monitoring terminal output for approval keywords
|
||||||
|
2. Taking screenshots when idle (waiting for input)
|
||||||
|
3. Analyzing screenshots with MiniCPM-V vision model via Ollama
|
||||||
|
4. Automatically submitting appropriate responses
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
- **Zero Pattern Matching**: Uses vision AI instead of fragile regex patterns
|
||||||
|
- **Universal Compatibility**: Works with any Claude Code prompt format
|
||||||
|
- **Intelligent Detection**: Only activates when approval keywords are present
|
||||||
|
- **Configurable**: Environment variables for all settings
|
||||||
|
- **Lightweight**: Minimal dependencies (only `requests`)
|
||||||
|
- **Debug Mode**: Verbose logging for troubleshooting
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
### Required
|
||||||
|
|
||||||
|
1. **Claude Code CLI** - Anthropic's official CLI tool
|
||||||
|
```bash
|
||||||
|
npm install -g @anthropic-ai/claude-code
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Ollama with MiniCPM-V** - Vision model server
|
||||||
|
```bash
|
||||||
|
docker pull ollama/ollama
|
||||||
|
docker run -d -p 11434:11434 --name ollama ollama/ollama
|
||||||
|
docker exec ollama ollama pull minicpm-v:latest
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Screenshot Tool** (one of):
|
||||||
|
- `scrot` (recommended)
|
||||||
|
- `gnome-screenshot`
|
||||||
|
- `imagemagick` (import command)
|
||||||
|
- `maim`
|
||||||
|
|
||||||
|
### Install Screenshot Tool
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Debian/Ubuntu (recommended)
|
||||||
|
sudo apt-get install scrot
|
||||||
|
|
||||||
|
# Alternative options
|
||||||
|
sudo apt-get install gnome-screenshot
|
||||||
|
sudo apt-get install imagemagick
|
||||||
|
sudo apt-get install maim
|
||||||
|
```
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
### Quick Install
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd claude-vision-auto
|
||||||
|
make deps # Install system dependencies
|
||||||
|
make install # Install the package
|
||||||
|
```
|
||||||
|
|
||||||
|
### Manual Install
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Install system dependencies
|
||||||
|
sudo apt-get update
|
||||||
|
sudo apt-get install -y scrot python3-pip
|
||||||
|
|
||||||
|
# Install Python package
|
||||||
|
pip3 install -e .
|
||||||
|
```
|
||||||
|
|
||||||
|
### Verify Installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check if command is available
|
||||||
|
which claude-vision
|
||||||
|
|
||||||
|
# Test Ollama connection
|
||||||
|
curl http://localhost:11434/api/tags
|
||||||
|
```
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
### Basic Usage
|
||||||
|
|
||||||
|
Replace `claude` with `claude-vision`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Instead of:
|
||||||
|
claude
|
||||||
|
|
||||||
|
# Use:
|
||||||
|
claude-vision
|
||||||
|
```
|
||||||
|
|
||||||
|
### With Prompts
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Pass prompts directly
|
||||||
|
claude-vision "create a test file in /tmp"
|
||||||
|
|
||||||
|
# Interactive session
|
||||||
|
claude-vision
|
||||||
|
```
|
||||||
|
|
||||||
|
### Configuration
|
||||||
|
|
||||||
|
Set environment variables to customize behavior:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Ollama URL (default: http://localhost:11434/api/generate)
|
||||||
|
export OLLAMA_URL="http://custom-host:11434/api/generate"
|
||||||
|
|
||||||
|
# Vision model (default: minicpm-v:latest)
|
||||||
|
export VISION_MODEL="llama3.2-vision:latest"
|
||||||
|
|
||||||
|
# Idle threshold in seconds (default: 3.0)
|
||||||
|
export IDLE_THRESHOLD="5.0"
|
||||||
|
|
||||||
|
# Response delay in seconds (default: 1.0)
|
||||||
|
export RESPONSE_DELAY="2.0"
|
||||||
|
|
||||||
|
# Enable debug mode
|
||||||
|
export DEBUG="true"
|
||||||
|
|
||||||
|
# Run with custom settings
|
||||||
|
claude-vision
|
||||||
|
```
|
||||||
|
|
||||||
|
## How It Works
|
||||||
|
|
||||||
|
1. **Launch**: Spawns `claude` as subprocess
|
||||||
|
2. **Monitor**: Watches output for approval keywords (Yes, No, Approve, etc.)
|
||||||
|
3. **Detect Idle**: When output stops for `IDLE_THRESHOLD` seconds
|
||||||
|
4. **Screenshot**: Captures terminal window with `scrot`
|
||||||
|
5. **Analyze**: Sends to MiniCPM-V via Ollama API
|
||||||
|
6. **Respond**: Vision model returns "1", "y", or "WAIT"
|
||||||
|
7. **Submit**: Automatically sends response if not "WAIT"
|
||||||
|
|
||||||
|
## Configuration Options
|
||||||
|
|
||||||
|
| Variable | Default | Description |
|
||||||
|
|----------|---------|-------------|
|
||||||
|
| `OLLAMA_URL` | `http://localhost:11434/api/generate` | Ollama API endpoint |
|
||||||
|
| `VISION_MODEL` | `minicpm-v:latest` | Vision model to use |
|
||||||
|
| `IDLE_THRESHOLD` | `3.0` | Seconds of idle before screenshot |
|
||||||
|
| `RESPONSE_DELAY` | `1.0` | Seconds to wait before responding |
|
||||||
|
| `OUTPUT_BUFFER_SIZE` | `4096` | Bytes of output to buffer |
|
||||||
|
| `SCREENSHOT_TIMEOUT` | `5` | Screenshot capture timeout |
|
||||||
|
| `VISION_TIMEOUT` | `30` | Vision analysis timeout |
|
||||||
|
| `DEBUG` | `false` | Enable verbose logging |
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Ollama Not Connected
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check if Ollama is running
|
||||||
|
docker ps | grep ollama
|
||||||
|
|
||||||
|
# Check if model is available
|
||||||
|
curl http://localhost:11434/api/tags
|
||||||
|
```
|
||||||
|
|
||||||
|
### Screenshot Fails
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Test screenshot tool
|
||||||
|
scrot /tmp/test.png
|
||||||
|
ls -lh /tmp/test.png
|
||||||
|
|
||||||
|
# Install if missing
|
||||||
|
sudo apt-get install scrot
|
||||||
|
```
|
||||||
|
|
||||||
|
### Debug Mode
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run with verbose logging
|
||||||
|
DEBUG=true claude-vision "test command"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Vision Model Not Found
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Pull the model
|
||||||
|
docker exec ollama ollama pull minicpm-v:latest
|
||||||
|
|
||||||
|
# Or use alternative vision model
|
||||||
|
export VISION_MODEL="llava:latest"
|
||||||
|
claude-vision
|
||||||
|
```
|
||||||
|
|
||||||
|
## Development
|
||||||
|
|
||||||
|
### Setup Development Environment
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Clone repository
|
||||||
|
git clone https://git.openharbor.io/svrnty/claude-vision-auto.git
|
||||||
|
cd claude-vision-auto
|
||||||
|
|
||||||
|
# Install in development mode
|
||||||
|
pip3 install -e .
|
||||||
|
|
||||||
|
# Run tests
|
||||||
|
make test
|
||||||
|
```
|
||||||
|
|
||||||
|
### Project Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
claude-vision-auto/
|
||||||
|
├── README.md # This file
|
||||||
|
├── LICENSE # MIT License
|
||||||
|
├── setup.py # Package setup
|
||||||
|
├── requirements.txt # Python dependencies
|
||||||
|
├── Makefile # Build automation
|
||||||
|
├── .gitignore # Git ignore rules
|
||||||
|
├── claude_vision_auto/ # Main package
|
||||||
|
│ ├── __init__.py # Package initialization
|
||||||
|
│ ├── main.py # CLI entry point
|
||||||
|
│ ├── config.py # Configuration
|
||||||
|
│ ├── screenshot.py # Screenshot capture
|
||||||
|
│ └── vision_analyzer.py # Vision analysis
|
||||||
|
├── bin/
|
||||||
|
│ └── claude-vision # CLI wrapper script
|
||||||
|
├── tests/ # Test suite
|
||||||
|
│ └── test_vision.py
|
||||||
|
├── docs/ # Documentation
|
||||||
|
│ ├── INSTALLATION.md
|
||||||
|
│ └── USAGE.md
|
||||||
|
└── examples/ # Usage examples
|
||||||
|
└── example_usage.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
## Supported Vision Models
|
||||||
|
|
||||||
|
Tested and working:
|
||||||
|
|
||||||
|
- **minicpm-v:latest** (Recommended) - Best for structured output
|
||||||
|
- **llama3.2-vision:latest** - Good alternative
|
||||||
|
- **llava:latest** - Fallback option
|
||||||
|
|
||||||
|
## Performance
|
||||||
|
|
||||||
|
- **Startup**: < 1 second
|
||||||
|
- **Screenshot**: ~100ms
|
||||||
|
- **Vision Analysis**: 2-5 seconds (depends on model)
|
||||||
|
- **Total Response Time**: 3-7 seconds per approval
|
||||||
|
|
||||||
|
## Limitations
|
||||||
|
|
||||||
|
- **X11 Only**: Requires X11 display server (no Wayland support for scrot)
|
||||||
|
- **Linux Only**: Currently only tested on Debian/Ubuntu
|
||||||
|
- **Vision Dependency**: Requires Ollama and vision model
|
||||||
|
- **Screen Required**: Must have GUI session (no headless support)
|
||||||
|
|
||||||
|
## Future Enhancements
|
||||||
|
|
||||||
|
- [ ] Wayland support (alternative screenshot tools)
|
||||||
|
- [ ] macOS support
|
||||||
|
- [ ] Headless mode (API-only, no screenshots)
|
||||||
|
- [ ] Configurable response patterns
|
||||||
|
- [ ] Multi-terminal support
|
||||||
|
- [ ] Session recording and replay
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
MIT License - See LICENSE file
|
||||||
|
|
||||||
|
## Author
|
||||||
|
|
||||||
|
**Svrnty**
|
||||||
|
- Email: jp@svrnty.io
|
||||||
|
- Repository: https://git.openharbor.io/svrnty/claude-vision-auto
|
||||||
|
|
||||||
|
## Contributing
|
||||||
|
|
||||||
|
Contributions welcome! Please:
|
||||||
|
1. Fork the repository
|
||||||
|
2. Create a feature branch
|
||||||
|
3. Submit a pull request
|
||||||
|
|
||||||
|
## Acknowledgments
|
||||||
|
|
||||||
|
- **Anthropic** - Claude Code CLI
|
||||||
|
- **MiniCPM-V** - Vision model
|
||||||
|
- **Ollama** - Model serving infrastructure
|
||||||
10
bin/claude-vision
Executable file
10
bin/claude-vision
Executable file
@ -0,0 +1,10 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Claude Vision Auto - CLI wrapper
|
||||||
|
"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
from claude_vision_auto.main import main
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
main()
|
||||||
14
claude_vision_auto/__init__.py
Normal file
14
claude_vision_auto/__init__.py
Normal file
@ -0,0 +1,14 @@
|
|||||||
|
"""
|
||||||
|
Claude Vision Auto - Vision-based auto-approval for Claude Code
|
||||||
|
|
||||||
|
Automatically analyzes terminal screenshots using MiniCPM-V vision model
|
||||||
|
to respond to Claude Code approval prompts.
|
||||||
|
"""
|
||||||
|
|
||||||
|
__version__ = "1.0.0"
|
||||||
|
__author__ = "Svrnty"
|
||||||
|
__license__ = "MIT"
|
||||||
|
|
||||||
|
from .main import run_claude_with_vision
|
||||||
|
|
||||||
|
__all__ = ["run_claude_with_vision"]
|
||||||
46
claude_vision_auto/config.py
Normal file
46
claude_vision_auto/config.py
Normal file
@ -0,0 +1,46 @@
|
|||||||
|
"""
|
||||||
|
Configuration for Claude Vision Auto
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
# Ollama configuration
|
||||||
|
OLLAMA_URL = os.getenv("OLLAMA_URL", "http://localhost:11434/api/generate")
|
||||||
|
VISION_MODEL = os.getenv("VISION_MODEL", "minicpm-v:latest")
|
||||||
|
|
||||||
|
# Timing configuration
|
||||||
|
IDLE_THRESHOLD = float(os.getenv("IDLE_THRESHOLD", "3.0")) # seconds of no output before screenshot
|
||||||
|
RESPONSE_DELAY = float(os.getenv("RESPONSE_DELAY", "1.0")) # seconds to wait before sending response
|
||||||
|
|
||||||
|
# Buffer configuration
|
||||||
|
OUTPUT_BUFFER_SIZE = int(os.getenv("OUTPUT_BUFFER_SIZE", "4096")) # bytes
|
||||||
|
|
||||||
|
# Keywords that suggest we're waiting for approval
|
||||||
|
APPROVAL_KEYWORDS = [
|
||||||
|
"Yes",
|
||||||
|
"No",
|
||||||
|
"(y/n)",
|
||||||
|
"[y/n]",
|
||||||
|
"Approve",
|
||||||
|
"Do you want to",
|
||||||
|
"create",
|
||||||
|
"edit",
|
||||||
|
"delete"
|
||||||
|
]
|
||||||
|
|
||||||
|
# Screenshot configuration
|
||||||
|
SCREENSHOT_TIMEOUT = int(os.getenv("SCREENSHOT_TIMEOUT", "5")) # seconds
|
||||||
|
SCREENSHOT_TOOLS = ["scrot", "gnome-screenshot", "import", "maim"]
|
||||||
|
|
||||||
|
# Vision analysis timeout
|
||||||
|
VISION_TIMEOUT = int(os.getenv("VISION_TIMEOUT", "30")) # seconds
|
||||||
|
|
||||||
|
# Debug mode
|
||||||
|
DEBUG = os.getenv("DEBUG", "false").lower() in ("true", "1", "yes")
|
||||||
|
|
||||||
|
def get_cache_dir() -> Path:
|
||||||
|
"""Get cache directory for screenshots"""
|
||||||
|
cache_dir = Path.home() / ".cache" / "claude-vision-auto"
|
||||||
|
cache_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
return cache_dir
|
||||||
164
claude_vision_auto/main.py
Normal file
164
claude_vision_auto/main.py
Normal file
@ -0,0 +1,164 @@
|
|||||||
|
"""
|
||||||
|
Main entry point for Claude Vision Auto
|
||||||
|
"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
import select
|
||||||
|
import subprocess
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from . import config
|
||||||
|
from .screenshot import take_screenshot, cleanup_old_screenshots
|
||||||
|
from .vision_analyzer import VisionAnalyzer
|
||||||
|
|
||||||
|
|
||||||
|
def run_claude_with_vision(args: list = None):
|
||||||
|
"""
|
||||||
|
Run Claude Code with vision-based auto-approval
|
||||||
|
|
||||||
|
Args:
|
||||||
|
args: Command line arguments to pass to claude
|
||||||
|
"""
|
||||||
|
args = args or []
|
||||||
|
|
||||||
|
# Initialize vision analyzer
|
||||||
|
analyzer = VisionAnalyzer()
|
||||||
|
|
||||||
|
# Test connection
|
||||||
|
print("[Claude Vision Auto] Testing Ollama connection...")
|
||||||
|
if not analyzer.test_connection():
|
||||||
|
print("[ERROR] Cannot connect to Ollama or model not available")
|
||||||
|
print(f"Make sure Ollama is running and '{config.VISION_MODEL}' is installed")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
print(f"[Claude Vision Auto] Connected to Ollama")
|
||||||
|
print(f"[Claude Vision Auto] Using model: {config.VISION_MODEL}")
|
||||||
|
print(f"[Claude Vision Auto] Idle threshold: {config.IDLE_THRESHOLD}s")
|
||||||
|
print()
|
||||||
|
|
||||||
|
# Build command
|
||||||
|
cmd = ['claude'] + args
|
||||||
|
|
||||||
|
# Start Claude Code process
|
||||||
|
try:
|
||||||
|
process = subprocess.Popen(
|
||||||
|
cmd,
|
||||||
|
stdin=subprocess.PIPE,
|
||||||
|
stdout=subprocess.PIPE,
|
||||||
|
stderr=subprocess.STDOUT,
|
||||||
|
bufsize=0
|
||||||
|
)
|
||||||
|
except FileNotFoundError:
|
||||||
|
print("[ERROR] 'claude' command not found")
|
||||||
|
print("Make sure Claude Code CLI is installed")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
last_output_time = time.time()
|
||||||
|
output_buffer = bytearray()
|
||||||
|
|
||||||
|
# Cleanup old screenshots
|
||||||
|
cleanup_old_screenshots()
|
||||||
|
|
||||||
|
try:
|
||||||
|
while True:
|
||||||
|
# Check if there's data to read
|
||||||
|
readable, _, _ = select.select([process.stdout], [], [], 0.1)
|
||||||
|
|
||||||
|
if readable:
|
||||||
|
char = process.stdout.read(1)
|
||||||
|
if not char:
|
||||||
|
# Process ended
|
||||||
|
break
|
||||||
|
|
||||||
|
# Print to terminal
|
||||||
|
sys.stdout.buffer.write(char)
|
||||||
|
sys.stdout.buffer.flush()
|
||||||
|
|
||||||
|
output_buffer.extend(char)
|
||||||
|
last_output_time = time.time()
|
||||||
|
|
||||||
|
# Keep buffer reasonable size
|
||||||
|
if len(output_buffer) > config.OUTPUT_BUFFER_SIZE:
|
||||||
|
output_buffer = output_buffer[-config.OUTPUT_BUFFER_SIZE:]
|
||||||
|
|
||||||
|
# Check if idle (no output for threshold seconds)
|
||||||
|
idle_time = time.time() - last_output_time
|
||||||
|
|
||||||
|
if idle_time >= config.IDLE_THRESHOLD:
|
||||||
|
# Check if buffer suggests we're waiting for input
|
||||||
|
buffer_str = output_buffer.decode('utf-8', errors='ignore')
|
||||||
|
|
||||||
|
# Look for approval keywords
|
||||||
|
has_keywords = any(
|
||||||
|
keyword in buffer_str
|
||||||
|
for keyword in config.APPROVAL_KEYWORDS
|
||||||
|
)
|
||||||
|
|
||||||
|
if has_keywords:
|
||||||
|
if config.DEBUG:
|
||||||
|
print("\n[DEBUG] Approval keywords detected in buffer")
|
||||||
|
|
||||||
|
print("\n[Vision] Analyzing prompt...", file=sys.stderr)
|
||||||
|
|
||||||
|
# Take screenshot
|
||||||
|
screenshot_path = take_screenshot()
|
||||||
|
|
||||||
|
if screenshot_path:
|
||||||
|
# Analyze with vision
|
||||||
|
response = analyzer.analyze_screenshot(screenshot_path)
|
||||||
|
|
||||||
|
if response:
|
||||||
|
print(f"[Vision] Response: {response}", file=sys.stderr)
|
||||||
|
|
||||||
|
if response and response.upper() != "WAIT":
|
||||||
|
# Send response
|
||||||
|
time.sleep(config.RESPONSE_DELAY)
|
||||||
|
process.stdin.write(f"{response}\n".encode('utf-8'))
|
||||||
|
process.stdin.flush()
|
||||||
|
|
||||||
|
# Clear buffer
|
||||||
|
output_buffer.clear()
|
||||||
|
last_output_time = time.time()
|
||||||
|
|
||||||
|
print("[Vision] Response sent", file=sys.stderr)
|
||||||
|
else:
|
||||||
|
print("[Vision] No action needed (WAIT)", file=sys.stderr)
|
||||||
|
else:
|
||||||
|
print("[Vision] Analysis failed, waiting for manual input", file=sys.stderr)
|
||||||
|
|
||||||
|
# Clean up screenshot
|
||||||
|
try:
|
||||||
|
Path(screenshot_path).unlink()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
else:
|
||||||
|
print("[Vision] Screenshot failed, waiting for manual input", file=sys.stderr)
|
||||||
|
|
||||||
|
# Reset idle detection
|
||||||
|
last_output_time = time.time()
|
||||||
|
|
||||||
|
# Check if process is still running
|
||||||
|
if process.poll() is not None:
|
||||||
|
break
|
||||||
|
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
print("\n[Claude Vision Auto] Interrupted by user")
|
||||||
|
process.terminate()
|
||||||
|
process.wait()
|
||||||
|
sys.exit(130)
|
||||||
|
|
||||||
|
finally:
|
||||||
|
# Wait for process to finish
|
||||||
|
exit_code = process.wait()
|
||||||
|
sys.exit(exit_code)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
"""CLI entry point"""
|
||||||
|
args = sys.argv[1:]
|
||||||
|
run_claude_with_vision(args)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
main()
|
||||||
123
claude_vision_auto/screenshot.py
Normal file
123
claude_vision_auto/screenshot.py
Normal file
@ -0,0 +1,123 @@
|
|||||||
|
"""
|
||||||
|
Screenshot capture functionality
|
||||||
|
"""
|
||||||
|
|
||||||
|
import subprocess
|
||||||
|
import tempfile
|
||||||
|
import shutil
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
from . import config
|
||||||
|
|
||||||
|
|
||||||
|
class ScreenshotError(Exception):
|
||||||
|
"""Raised when screenshot capture fails"""
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def find_screenshot_tool() -> Optional[str]:
|
||||||
|
"""Find available screenshot tool on the system"""
|
||||||
|
for tool in config.SCREENSHOT_TOOLS:
|
||||||
|
if shutil.which(tool):
|
||||||
|
return tool
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def take_screenshot() -> Optional[str]:
|
||||||
|
"""
|
||||||
|
Take screenshot of active terminal window
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Path to screenshot file, or None if capture failed
|
||||||
|
"""
|
||||||
|
tool = find_screenshot_tool()
|
||||||
|
|
||||||
|
if not tool:
|
||||||
|
if config.DEBUG:
|
||||||
|
print(f"[DEBUG] No screenshot tool found. Tried: {', '.join(config.SCREENSHOT_TOOLS)}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
# Create temporary file
|
||||||
|
cache_dir = config.get_cache_dir()
|
||||||
|
screenshot_path = cache_dir / f"screenshot_{int(subprocess.check_output(['date', '+%s']).decode().strip())}.png"
|
||||||
|
|
||||||
|
try:
|
||||||
|
if tool == "scrot":
|
||||||
|
# Capture active window
|
||||||
|
subprocess.run(
|
||||||
|
["scrot", "-u", str(screenshot_path)],
|
||||||
|
check=True,
|
||||||
|
capture_output=True,
|
||||||
|
timeout=config.SCREENSHOT_TIMEOUT
|
||||||
|
)
|
||||||
|
elif tool == "gnome-screenshot":
|
||||||
|
# Capture active window
|
||||||
|
subprocess.run(
|
||||||
|
["gnome-screenshot", "-w", "-f", str(screenshot_path)],
|
||||||
|
check=True,
|
||||||
|
capture_output=True,
|
||||||
|
timeout=config.SCREENSHOT_TIMEOUT
|
||||||
|
)
|
||||||
|
elif tool == "import":
|
||||||
|
# ImageMagick - capture root window
|
||||||
|
subprocess.run(
|
||||||
|
["import", "-window", "root", str(screenshot_path)],
|
||||||
|
check=True,
|
||||||
|
capture_output=True,
|
||||||
|
timeout=config.SCREENSHOT_TIMEOUT
|
||||||
|
)
|
||||||
|
elif tool == "maim":
|
||||||
|
# Capture active window
|
||||||
|
subprocess.run(
|
||||||
|
["maim", "-i", "$(xdotool getactivewindow)", str(screenshot_path)],
|
||||||
|
shell=True,
|
||||||
|
check=True,
|
||||||
|
capture_output=True,
|
||||||
|
timeout=config.SCREENSHOT_TIMEOUT
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
return None
|
||||||
|
|
||||||
|
if screenshot_path.exists():
|
||||||
|
if config.DEBUG:
|
||||||
|
print(f"[DEBUG] Screenshot saved to {screenshot_path}")
|
||||||
|
return str(screenshot_path)
|
||||||
|
else:
|
||||||
|
return None
|
||||||
|
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
if config.DEBUG:
|
||||||
|
print(f"[DEBUG] Screenshot timeout with tool: {tool}")
|
||||||
|
return None
|
||||||
|
except subprocess.CalledProcessError as e:
|
||||||
|
if config.DEBUG:
|
||||||
|
print(f"[DEBUG] Screenshot failed: {e}")
|
||||||
|
return None
|
||||||
|
except Exception as e:
|
||||||
|
if config.DEBUG:
|
||||||
|
print(f"[DEBUG] Unexpected error: {e}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def cleanup_old_screenshots(max_age_seconds: int = 3600):
|
||||||
|
"""
|
||||||
|
Clean up old screenshots from cache directory
|
||||||
|
|
||||||
|
Args:
|
||||||
|
max_age_seconds: Maximum age of screenshots to keep (default 1 hour)
|
||||||
|
"""
|
||||||
|
import time
|
||||||
|
|
||||||
|
cache_dir = config.get_cache_dir()
|
||||||
|
current_time = time.time()
|
||||||
|
|
||||||
|
for screenshot in cache_dir.glob("screenshot_*.png"):
|
||||||
|
if current_time - screenshot.stat().st_mtime > max_age_seconds:
|
||||||
|
try:
|
||||||
|
screenshot.unlink()
|
||||||
|
if config.DEBUG:
|
||||||
|
print(f"[DEBUG] Cleaned up old screenshot: {screenshot}")
|
||||||
|
except Exception as e:
|
||||||
|
if config.DEBUG:
|
||||||
|
print(f"[DEBUG] Failed to cleanup {screenshot}: {e}")
|
||||||
133
claude_vision_auto/vision_analyzer.py
Normal file
133
claude_vision_auto/vision_analyzer.py
Normal file
@ -0,0 +1,133 @@
|
|||||||
|
"""
|
||||||
|
Vision analysis using MiniCPM-V via Ollama
|
||||||
|
"""
|
||||||
|
|
||||||
|
import base64
|
||||||
|
import requests
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
from . import config
|
||||||
|
|
||||||
|
|
||||||
|
VISION_PROMPT = """You are analyzing a terminal screenshot showing a Claude Code approval prompt.
|
||||||
|
|
||||||
|
Look for:
|
||||||
|
- Menu options like "1. Yes", "2. Yes, allow all", "3. No"
|
||||||
|
- Questions asking for approval (create/edit/delete files)
|
||||||
|
- Yes/No questions
|
||||||
|
|
||||||
|
If you see an approval prompt with numbered options:
|
||||||
|
- Respond ONLY with the number to select "Yes" (usually "1")
|
||||||
|
- Output format: Just the number, nothing else
|
||||||
|
|
||||||
|
If you see a yes/no question:
|
||||||
|
- Respond with: y
|
||||||
|
|
||||||
|
If you don't see any prompt requiring input:
|
||||||
|
- Respond with: WAIT
|
||||||
|
|
||||||
|
Your response (one word only):"""
|
||||||
|
|
||||||
|
|
||||||
|
class VisionAnalyzer:
|
||||||
|
"""Analyzes screenshots using vision model"""
|
||||||
|
|
||||||
|
def __init__(self, ollama_url: str = None, model: str = None):
|
||||||
|
"""
|
||||||
|
Initialize vision analyzer
|
||||||
|
|
||||||
|
Args:
|
||||||
|
ollama_url: Ollama API URL (default from config)
|
||||||
|
model: Vision model name (default from config)
|
||||||
|
"""
|
||||||
|
self.ollama_url = ollama_url or config.OLLAMA_URL
|
||||||
|
self.model = model or config.VISION_MODEL
|
||||||
|
|
||||||
|
def analyze_screenshot(self, image_path: str) -> Optional[str]:
|
||||||
|
"""
|
||||||
|
Analyze screenshot and determine what response to give
|
||||||
|
|
||||||
|
Args:
|
||||||
|
image_path: Path to screenshot image
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Response to send ("1", "y", "WAIT", etc.) or None on error
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
# Read and encode image
|
||||||
|
with open(image_path, 'rb') as f:
|
||||||
|
image_data = base64.b64encode(f.read()).decode('utf-8')
|
||||||
|
|
||||||
|
# Send to Ollama
|
||||||
|
payload = {
|
||||||
|
"model": self.model,
|
||||||
|
"prompt": VISION_PROMPT,
|
||||||
|
"images": [image_data],
|
||||||
|
"stream": False
|
||||||
|
}
|
||||||
|
|
||||||
|
if config.DEBUG:
|
||||||
|
print(f"[DEBUG] Sending to Ollama: {self.ollama_url}")
|
||||||
|
print(f"[DEBUG] Model: {self.model}")
|
||||||
|
|
||||||
|
response = requests.post(
|
||||||
|
self.ollama_url,
|
||||||
|
json=payload,
|
||||||
|
timeout=config.VISION_TIMEOUT
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
|
||||||
|
result = response.json()
|
||||||
|
answer = result.get('response', '').strip()
|
||||||
|
|
||||||
|
if config.DEBUG:
|
||||||
|
print(f"[DEBUG] Vision model response: {answer}")
|
||||||
|
|
||||||
|
return answer
|
||||||
|
|
||||||
|
except requests.Timeout:
|
||||||
|
if config.DEBUG:
|
||||||
|
print("[DEBUG] Vision analysis timeout")
|
||||||
|
return None
|
||||||
|
except requests.RequestException as e:
|
||||||
|
if config.DEBUG:
|
||||||
|
print(f"[DEBUG] Vision API error: {e}")
|
||||||
|
return None
|
||||||
|
except Exception as e:
|
||||||
|
if config.DEBUG:
|
||||||
|
print(f"[DEBUG] Unexpected error in vision analysis: {e}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
def test_connection(self) -> bool:
|
||||||
|
"""
|
||||||
|
Test if Ollama is accessible and model is available
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True if connection successful, False otherwise
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
# Try to list tags
|
||||||
|
tags_url = self.ollama_url.replace('/api/generate', '/api/tags')
|
||||||
|
response = requests.get(tags_url, timeout=5)
|
||||||
|
response.raise_for_status()
|
||||||
|
|
||||||
|
data = response.json()
|
||||||
|
models = [m['name'] for m in data.get('models', [])]
|
||||||
|
|
||||||
|
if config.DEBUG:
|
||||||
|
print(f"[DEBUG] Available models: {models}")
|
||||||
|
|
||||||
|
# Check if our model is available
|
||||||
|
model_available = any(self.model in m for m in models)
|
||||||
|
|
||||||
|
if not model_available:
|
||||||
|
print(f"Warning: Model '{self.model}' not found in Ollama")
|
||||||
|
print(f"Available models: {', '.join(models)}")
|
||||||
|
|
||||||
|
return model_available
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
if config.DEBUG:
|
||||||
|
print(f"[DEBUG] Connection test failed: {e}")
|
||||||
|
return False
|
||||||
358
docs/INSTALLATION.md
Normal file
358
docs/INSTALLATION.md
Normal file
@ -0,0 +1,358 @@
|
|||||||
|
# Installation Guide
|
||||||
|
|
||||||
|
Detailed installation instructions for Claude Vision Auto.
|
||||||
|
|
||||||
|
## Table of Contents
|
||||||
|
|
||||||
|
1. [Prerequisites](#prerequisites)
|
||||||
|
2. [System Dependencies](#system-dependencies)
|
||||||
|
3. [Ollama Setup](#ollama-setup)
|
||||||
|
4. [Package Installation](#package-installation)
|
||||||
|
5. [Verification](#verification)
|
||||||
|
6. [Troubleshooting](#troubleshooting)
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
### 1. Claude Code CLI
|
||||||
|
|
||||||
|
Install Anthropic's official CLI:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install -g @anthropic-ai/claude-code
|
||||||
|
```
|
||||||
|
|
||||||
|
Verify installation:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
claude --version
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Python 3.8+
|
||||||
|
|
||||||
|
Check Python version:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 --version
|
||||||
|
```
|
||||||
|
|
||||||
|
If not installed:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo apt-get update
|
||||||
|
sudo apt-get install python3 python3-pip
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Docker (for Ollama)
|
||||||
|
|
||||||
|
Install Docker:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -fsSL https://get.docker.com | sh
|
||||||
|
sudo usermod -aG docker $USER
|
||||||
|
```
|
||||||
|
|
||||||
|
Log out and back in for group changes to take effect.
|
||||||
|
|
||||||
|
## System Dependencies
|
||||||
|
|
||||||
|
### Screenshot Tool
|
||||||
|
|
||||||
|
Install `scrot` (recommended):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo apt-get update
|
||||||
|
sudo apt-get install -y scrot
|
||||||
|
```
|
||||||
|
|
||||||
|
Alternative screenshot tools:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# GNOME Screenshot
|
||||||
|
sudo apt-get install -y gnome-screenshot
|
||||||
|
|
||||||
|
# ImageMagick
|
||||||
|
sudo apt-get install -y imagemagick
|
||||||
|
|
||||||
|
# Maim
|
||||||
|
sudo apt-get install -y maim xdotool
|
||||||
|
```
|
||||||
|
|
||||||
|
### Additional Dependencies
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo apt-get install -y \
|
||||||
|
python3-pip \
|
||||||
|
git \
|
||||||
|
curl
|
||||||
|
```
|
||||||
|
|
||||||
|
## Ollama Setup
|
||||||
|
|
||||||
|
### 1. Pull Ollama Docker Image
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker pull ollama/ollama:latest
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Start Ollama Container
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker run -d \
|
||||||
|
-p 11434:11434 \
|
||||||
|
--name ollama \
|
||||||
|
--restart unless-stopped \
|
||||||
|
ollama/ollama:latest
|
||||||
|
```
|
||||||
|
|
||||||
|
For GPU support (NVIDIA):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker run -d \
|
||||||
|
-p 11434:11434 \
|
||||||
|
--name ollama \
|
||||||
|
--gpus all \
|
||||||
|
--restart unless-stopped \
|
||||||
|
ollama/ollama:latest
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Pull Vision Model
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# MiniCPM-V (recommended - 5.5GB)
|
||||||
|
docker exec ollama ollama pull minicpm-v:latest
|
||||||
|
|
||||||
|
# Alternative: Llama 3.2 Vision (7.8GB)
|
||||||
|
docker exec ollama ollama pull llama3.2-vision:latest
|
||||||
|
|
||||||
|
# Alternative: LLaVA (4.5GB)
|
||||||
|
docker exec ollama ollama pull llava:latest
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Verify Ollama
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check container status
|
||||||
|
docker ps | grep ollama
|
||||||
|
|
||||||
|
# Test API
|
||||||
|
curl http://localhost:11434/api/tags
|
||||||
|
|
||||||
|
# List installed models
|
||||||
|
curl -s http://localhost:11434/api/tags | python3 -m json.tool
|
||||||
|
```
|
||||||
|
|
||||||
|
## Package Installation
|
||||||
|
|
||||||
|
### Method 1: Using Makefile (Recommended)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd claude-vision-auto
|
||||||
|
|
||||||
|
# Install system dependencies
|
||||||
|
make deps
|
||||||
|
|
||||||
|
# Install package
|
||||||
|
make install
|
||||||
|
```
|
||||||
|
|
||||||
|
### Method 2: Manual Installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd claude-vision-auto
|
||||||
|
|
||||||
|
# Install system dependencies
|
||||||
|
sudo apt-get update
|
||||||
|
sudo apt-get install -y scrot python3-pip
|
||||||
|
|
||||||
|
# Install Python package
|
||||||
|
pip3 install -e .
|
||||||
|
```
|
||||||
|
|
||||||
|
### Method 3: From Git
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Clone repository
|
||||||
|
git clone https://git.openharbor.io/svrnty/claude-vision-auto.git
|
||||||
|
cd claude-vision-auto
|
||||||
|
|
||||||
|
# Install
|
||||||
|
pip3 install -e .
|
||||||
|
```
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
### 1. Check Command Installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
which claude-vision
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected output: `/home/username/.local/bin/claude-vision`
|
||||||
|
|
||||||
|
### 2. Test Ollama Connection
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl http://localhost:11434/api/tags
|
||||||
|
```
|
||||||
|
|
||||||
|
Should return JSON with list of models.
|
||||||
|
|
||||||
|
### 3. Test Screenshot
|
||||||
|
|
||||||
|
```bash
|
||||||
|
scrot /tmp/test_screenshot.png
|
||||||
|
ls -lh /tmp/test_screenshot.png
|
||||||
|
```
|
||||||
|
|
||||||
|
Should create a screenshot file.
|
||||||
|
|
||||||
|
### 4. Run Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Start claude-vision
|
||||||
|
claude-vision
|
||||||
|
|
||||||
|
# You should see:
|
||||||
|
# [Claude Vision Auto] Testing Ollama connection...
|
||||||
|
# [Claude Vision Auto] Connected to Ollama
|
||||||
|
# [Claude Vision Auto] Using model: minicpm-v:latest
|
||||||
|
```
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### "claude-vision: command not found"
|
||||||
|
|
||||||
|
Add to PATH in `~/.bashrc` or `~/.zshrc`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export PATH="$HOME/.local/bin:$PATH"
|
||||||
|
```
|
||||||
|
|
||||||
|
Then reload:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
source ~/.bashrc # or source ~/.zshrc
|
||||||
|
```
|
||||||
|
|
||||||
|
### "Cannot connect to Ollama"
|
||||||
|
|
||||||
|
Check if Ollama container is running:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker ps | grep ollama
|
||||||
|
|
||||||
|
# If not running, start it:
|
||||||
|
docker start ollama
|
||||||
|
```
|
||||||
|
|
||||||
|
Check if port 11434 is open:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
netstat -tulpn | grep 11434
|
||||||
|
# or
|
||||||
|
ss -tulpn | grep 11434
|
||||||
|
```
|
||||||
|
|
||||||
|
### "Model not found"
|
||||||
|
|
||||||
|
Pull the model:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker exec ollama ollama pull minicpm-v:latest
|
||||||
|
```
|
||||||
|
|
||||||
|
List available models:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker exec ollama ollama list
|
||||||
|
```
|
||||||
|
|
||||||
|
### "Screenshot failed"
|
||||||
|
|
||||||
|
Install scrot:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo apt-get install scrot
|
||||||
|
```
|
||||||
|
|
||||||
|
Test screenshot:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
scrot -u /tmp/test.png
|
||||||
|
```
|
||||||
|
|
||||||
|
If error persists, try alternative tools in config:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export SCREENSHOT_TOOL="gnome-screenshot"
|
||||||
|
claude-vision
|
||||||
|
```
|
||||||
|
|
||||||
|
### Permission Issues
|
||||||
|
|
||||||
|
If pip install fails with permissions:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Install for user only
|
||||||
|
pip3 install --user -e .
|
||||||
|
|
||||||
|
# Or use virtual environment
|
||||||
|
python3 -m venv venv
|
||||||
|
source venv/bin/activate
|
||||||
|
pip install -e .
|
||||||
|
```
|
||||||
|
|
||||||
|
### Docker Permission Denied
|
||||||
|
|
||||||
|
Add user to docker group:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo usermod -aG docker $USER
|
||||||
|
```
|
||||||
|
|
||||||
|
Log out and back in, then:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker ps # Should work without sudo
|
||||||
|
```
|
||||||
|
|
||||||
|
## Uninstallation
|
||||||
|
|
||||||
|
### Remove Package
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make uninstall
|
||||||
|
# or
|
||||||
|
pip3 uninstall claude-vision-auto
|
||||||
|
```
|
||||||
|
|
||||||
|
### Remove Ollama
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker stop ollama
|
||||||
|
docker rm ollama
|
||||||
|
docker rmi ollama/ollama
|
||||||
|
```
|
||||||
|
|
||||||
|
### Remove System Dependencies
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo apt-get remove scrot
|
||||||
|
```
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
After successful installation:
|
||||||
|
|
||||||
|
1. Read [USAGE.md](USAGE.md) for usage examples
|
||||||
|
2. Configure environment variables if needed
|
||||||
|
3. Test with a simple Claude Code command
|
||||||
|
|
||||||
|
## Getting Help
|
||||||
|
|
||||||
|
If you encounter issues not covered here:
|
||||||
|
|
||||||
|
1. Check the main [README.md](../README.md)
|
||||||
|
2. Enable debug mode: `DEBUG=true claude-vision`
|
||||||
|
3. Check logs: `~/.cache/claude-vision-auto/`
|
||||||
|
4. Report issues: https://git.openharbor.io/svrnty/claude-vision-auto/issues
|
||||||
439
docs/USAGE.md
Normal file
439
docs/USAGE.md
Normal file
@ -0,0 +1,439 @@
|
|||||||
|
# Usage Guide
|
||||||
|
|
||||||
|
Comprehensive usage guide for Claude Vision Auto.
|
||||||
|
|
||||||
|
## Table of Contents
|
||||||
|
|
||||||
|
1. [Basic Usage](#basic-usage)
|
||||||
|
2. [Configuration](#configuration)
|
||||||
|
3. [Common Scenarios](#common-scenarios)
|
||||||
|
4. [Advanced Usage](#advanced-usage)
|
||||||
|
5. [Tips and Best Practices](#tips-and-best-practices)
|
||||||
|
|
||||||
|
## Basic Usage
|
||||||
|
|
||||||
|
### Starting Interactive Session
|
||||||
|
|
||||||
|
Simply replace `claude` with `claude-vision`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
claude-vision
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected output:
|
||||||
|
|
||||||
|
```
|
||||||
|
[Claude Vision Auto] Testing Ollama connection...
|
||||||
|
[Claude Vision Auto] Connected to Ollama
|
||||||
|
[Claude Vision Auto] Using model: minicpm-v:latest
|
||||||
|
[Claude Vision Auto] Idle threshold: 3.0s
|
||||||
|
|
||||||
|
▐▛███▜▌ Claude Code v2.0.26
|
||||||
|
▝▜█████▛▘ Sonnet 4.5 · Claude Max
|
||||||
|
▘▘ ▝▝ /home/username/project
|
||||||
|
|
||||||
|
>
|
||||||
|
```
|
||||||
|
|
||||||
|
### With Initial Prompt
|
||||||
|
|
||||||
|
```bash
|
||||||
|
claude-vision "create a test.md file in /tmp"
|
||||||
|
```
|
||||||
|
|
||||||
|
### How Auto-Approval Works
|
||||||
|
|
||||||
|
When Claude asks for approval:
|
||||||
|
|
||||||
|
```
|
||||||
|
╭───────────────────────────────────────────╮
|
||||||
|
│ Create file │
|
||||||
|
│ ╭───────────────────────────────────────╮ │
|
||||||
|
│ │ /tmp/test.md │ │
|
||||||
|
│ │ │ │
|
||||||
|
│ │ # Test File │ │
|
||||||
|
│ │ │ │
|
||||||
|
│ │ This is a test. │ │
|
||||||
|
│ ╰───────────────────────────────────────╯ │
|
||||||
|
│ Do you want to create test.md? │
|
||||||
|
│ ❯ 1. Yes │
|
||||||
|
│ 2. Yes, allow all edits │
|
||||||
|
│ 3. No │
|
||||||
|
╰───────────────────────────────────────────╯
|
||||||
|
```
|
||||||
|
|
||||||
|
Claude Vision Auto will:
|
||||||
|
1. Detect idle state (3 seconds)
|
||||||
|
2. Take screenshot
|
||||||
|
3. Analyze with vision model
|
||||||
|
4. Automatically select option 1
|
||||||
|
|
||||||
|
Output:
|
||||||
|
|
||||||
|
```
|
||||||
|
[Vision] Analyzing prompt...
|
||||||
|
[Vision] Response: 1
|
||||||
|
[Vision] Response sent
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
### Environment Variables
|
||||||
|
|
||||||
|
Set before running `claude-vision`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Example: Custom configuration
|
||||||
|
export OLLAMA_URL="http://192.168.1.100:11434/api/generate"
|
||||||
|
export VISION_MODEL="llama3.2-vision:latest"
|
||||||
|
export IDLE_THRESHOLD="5.0"
|
||||||
|
export RESPONSE_DELAY="2.0"
|
||||||
|
export DEBUG="true"
|
||||||
|
|
||||||
|
claude-vision
|
||||||
|
```
|
||||||
|
|
||||||
|
### Persistent Configuration
|
||||||
|
|
||||||
|
Add to `~/.bashrc` or `~/.zshrc`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Claude Vision Auto Configuration
|
||||||
|
export CLAUDE_VISION_OLLAMA_URL="http://localhost:11434/api/generate"
|
||||||
|
export CLAUDE_VISION_MODEL="minicpm-v:latest"
|
||||||
|
export CLAUDE_VISION_IDLE_THRESHOLD="3.0"
|
||||||
|
export CLAUDE_VISION_RESPONSE_DELAY="1.0"
|
||||||
|
|
||||||
|
# Alias for convenience
|
||||||
|
alias cv="claude-vision"
|
||||||
|
```
|
||||||
|
|
||||||
|
Reload:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
source ~/.bashrc
|
||||||
|
```
|
||||||
|
|
||||||
|
### Configuration Options Reference
|
||||||
|
|
||||||
|
| Variable | Type | Default | Description |
|
||||||
|
|----------|------|---------|-------------|
|
||||||
|
| `OLLAMA_URL` | URL | `http://localhost:11434/api/generate` | Ollama API endpoint |
|
||||||
|
| `VISION_MODEL` | String | `minicpm-v:latest` | Vision model name |
|
||||||
|
| `IDLE_THRESHOLD` | Float | `3.0` | Seconds to wait before screenshot |
|
||||||
|
| `RESPONSE_DELAY` | Float | `1.0` | Seconds to wait before responding |
|
||||||
|
| `OUTPUT_BUFFER_SIZE` | Integer | `4096` | Buffer size in bytes |
|
||||||
|
| `SCREENSHOT_TIMEOUT` | Integer | `5` | Screenshot timeout (seconds) |
|
||||||
|
| `VISION_TIMEOUT` | Integer | `30` | Vision analysis timeout (seconds) |
|
||||||
|
| `DEBUG` | Boolean | `false` | Enable debug logging |
|
||||||
|
|
||||||
|
## Common Scenarios
|
||||||
|
|
||||||
|
### Scenario 1: File Creation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
claude-vision "create a new Python script called hello.py"
|
||||||
|
```
|
||||||
|
|
||||||
|
Auto-approves file creation.
|
||||||
|
|
||||||
|
### Scenario 2: File Editing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
claude-vision "add error handling to main.py"
|
||||||
|
```
|
||||||
|
|
||||||
|
Auto-approves file edits.
|
||||||
|
|
||||||
|
### Scenario 3: Multiple Operations
|
||||||
|
|
||||||
|
```bash
|
||||||
|
claude-vision "refactor the authentication module and add tests"
|
||||||
|
```
|
||||||
|
|
||||||
|
Auto-approves each operation sequentially.
|
||||||
|
|
||||||
|
### Scenario 4: Longer Wait Time
|
||||||
|
|
||||||
|
For slower systems or models:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
IDLE_THRESHOLD="5.0" RESPONSE_DELAY="2.0" claude-vision
|
||||||
|
```
|
||||||
|
|
||||||
|
### Scenario 5: Different Vision Model
|
||||||
|
|
||||||
|
```bash
|
||||||
|
VISION_MODEL="llama3.2-vision:latest" claude-vision
|
||||||
|
```
|
||||||
|
|
||||||
|
### Scenario 6: Remote Ollama
|
||||||
|
|
||||||
|
```bash
|
||||||
|
OLLAMA_URL="http://192.168.1.100:11434/api/generate" claude-vision
|
||||||
|
```
|
||||||
|
|
||||||
|
### Scenario 7: Debug Mode
|
||||||
|
|
||||||
|
When troubleshooting:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
DEBUG=true claude-vision "test command"
|
||||||
|
```
|
||||||
|
|
||||||
|
Output will include:
|
||||||
|
|
||||||
|
```
|
||||||
|
[DEBUG] Screenshot saved to /home/user/.cache/claude-vision-auto/screenshot_1234567890.png
|
||||||
|
[DEBUG] Sending to Ollama: http://localhost:11434/api/generate
|
||||||
|
[DEBUG] Model: minicpm-v:latest
|
||||||
|
[DEBUG] Vision model response: 1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Advanced Usage
|
||||||
|
|
||||||
|
### Using Different Models
|
||||||
|
|
||||||
|
#### MiniCPM-V (Default)
|
||||||
|
|
||||||
|
Best for structured responses:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
VISION_MODEL="minicpm-v:latest" claude-vision
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Llama 3.2 Vision
|
||||||
|
|
||||||
|
Good alternative:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
VISION_MODEL="llama3.2-vision:latest" claude-vision
|
||||||
|
```
|
||||||
|
|
||||||
|
#### LLaVA
|
||||||
|
|
||||||
|
Lightweight option:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
VISION_MODEL="llava:latest" claude-vision
|
||||||
|
```
|
||||||
|
|
||||||
|
### Shell Aliases
|
||||||
|
|
||||||
|
Add to `~/.bashrc`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Quick aliases
|
||||||
|
alias cv="claude-vision"
|
||||||
|
alias cvd="DEBUG=true claude-vision" # Debug mode
|
||||||
|
alias cvs="IDLE_THRESHOLD=5.0 claude-vision" # Slower
|
||||||
|
|
||||||
|
# Model-specific
|
||||||
|
alias cv-mini="VISION_MODEL=minicpm-v:latest claude-vision"
|
||||||
|
alias cv-llama="VISION_MODEL=llama3.2-vision:latest claude-vision"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Integration with Scripts
|
||||||
|
|
||||||
|
```bash
|
||||||
|
#!/bin/bash
|
||||||
|
# auto-refactor.sh
|
||||||
|
|
||||||
|
export DEBUG="false"
|
||||||
|
export IDLE_THRESHOLD="3.0"
|
||||||
|
|
||||||
|
claude-vision "refactor all JavaScript files to use modern ES6 syntax"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Conditional Auto-Approval
|
||||||
|
|
||||||
|
Create wrapper script:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
#!/bin/bash
|
||||||
|
# conditional-claude.sh
|
||||||
|
|
||||||
|
if [ "$AUTO_APPROVE" = "true" ]; then
|
||||||
|
claude-vision "$@"
|
||||||
|
else
|
||||||
|
claude "$@"
|
||||||
|
fi
|
||||||
|
```
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
AUTO_APPROVE=true ./conditional-claude.sh "create files"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Multiple Terminal Support
|
||||||
|
|
||||||
|
Each terminal needs separate instance:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Terminal 1
|
||||||
|
claude-vision # Project A
|
||||||
|
|
||||||
|
# Terminal 2
|
||||||
|
claude-vision # Project B
|
||||||
|
```
|
||||||
|
|
||||||
|
## Tips and Best Practices
|
||||||
|
|
||||||
|
### 1. Adjust Idle Threshold
|
||||||
|
|
||||||
|
- **Fast system**: `IDLE_THRESHOLD=2.0`
|
||||||
|
- **Slow system**: `IDLE_THRESHOLD=5.0`
|
||||||
|
- **Remote Ollama**: `IDLE_THRESHOLD=4.0`
|
||||||
|
|
||||||
|
### 2. Model Selection
|
||||||
|
|
||||||
|
- **Accuracy**: MiniCPM-V > Llama 3.2 Vision > LLaVA
|
||||||
|
- **Speed**: LLaVA > MiniCPM-V > Llama 3.2 Vision
|
||||||
|
- **Size**: LLaVA (4.5GB) < MiniCPM-V (5.5GB) < Llama 3.2 (7.8GB)
|
||||||
|
|
||||||
|
### 3. Debug When Needed
|
||||||
|
|
||||||
|
Enable debug mode if responses are incorrect:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
DEBUG=true claude-vision
|
||||||
|
```
|
||||||
|
|
||||||
|
Check vision model output to see what it's detecting.
|
||||||
|
|
||||||
|
### 4. Screenshot Quality
|
||||||
|
|
||||||
|
Ensure terminal is visible and not obscured by other windows.
|
||||||
|
|
||||||
|
### 5. Performance Optimization
|
||||||
|
|
||||||
|
For faster responses:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Pre-warm the model
|
||||||
|
docker exec ollama ollama run minicpm-v:latest "test" < /dev/null
|
||||||
|
|
||||||
|
# Then use normally
|
||||||
|
claude-vision
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6. Clean Up Old Screenshots
|
||||||
|
|
||||||
|
Screenshots are auto-cleaned after 1 hour, but manual cleanup:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rm -rf ~/.cache/claude-vision-auto/*.png
|
||||||
|
```
|
||||||
|
|
||||||
|
### 7. Check Model Status
|
||||||
|
|
||||||
|
Before starting long sessions:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Verify Ollama is responsive
|
||||||
|
curl -s http://localhost:11434/api/tags | python3 -m json.tool
|
||||||
|
|
||||||
|
# Check model is loaded
|
||||||
|
docker exec ollama ollama ps
|
||||||
|
```
|
||||||
|
|
||||||
|
## Troubleshooting During Use
|
||||||
|
|
||||||
|
### Issue: Not Auto-Responding
|
||||||
|
|
||||||
|
**Symptoms**: Vision analyzes but doesn't respond
|
||||||
|
|
||||||
|
**Solutions**:
|
||||||
|
|
||||||
|
1. Check if "WAIT" is returned:
|
||||||
|
```bash
|
||||||
|
DEBUG=true claude-vision
|
||||||
|
# Look for: [DEBUG] Vision model response: WAIT
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Increase idle threshold:
|
||||||
|
```bash
|
||||||
|
IDLE_THRESHOLD="5.0" claude-vision
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Try different model:
|
||||||
|
```bash
|
||||||
|
VISION_MODEL="llama3.2-vision:latest" claude-vision
|
||||||
|
```
|
||||||
|
|
||||||
|
### Issue: Wrong Response
|
||||||
|
|
||||||
|
**Symptoms**: Selects wrong option or types wrong answer
|
||||||
|
|
||||||
|
**Solutions**:
|
||||||
|
|
||||||
|
1. Enable debug to see what vision model sees:
|
||||||
|
```bash
|
||||||
|
DEBUG=true claude-vision
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Check screenshot manually:
|
||||||
|
```bash
|
||||||
|
# Don't auto-clean
|
||||||
|
ls ~/.cache/claude-vision-auto/
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Adjust response delay:
|
||||||
|
```bash
|
||||||
|
RESPONSE_DELAY="2.0" claude-vision
|
||||||
|
```
|
||||||
|
|
||||||
|
### Issue: Slow Response
|
||||||
|
|
||||||
|
**Symptoms**: Takes too long to respond
|
||||||
|
|
||||||
|
**Solutions**:
|
||||||
|
|
||||||
|
1. Use faster model:
|
||||||
|
```bash
|
||||||
|
VISION_MODEL="llava:latest" claude-vision
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Reduce vision timeout:
|
||||||
|
```bash
|
||||||
|
VISION_TIMEOUT="15" claude-vision
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Check Ollama performance:
|
||||||
|
```bash
|
||||||
|
docker stats ollama
|
||||||
|
```
|
||||||
|
|
||||||
|
### Issue: Too Aggressive
|
||||||
|
|
||||||
|
**Symptoms**: Responds to non-approval prompts
|
||||||
|
|
||||||
|
**Solutions**:
|
||||||
|
|
||||||
|
1. Increase idle threshold:
|
||||||
|
```bash
|
||||||
|
IDLE_THRESHOLD="5.0" claude-vision
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Check approval keywords in config
|
||||||
|
3. Use manual mode for sensitive operations:
|
||||||
|
```bash
|
||||||
|
claude # No auto-approval
|
||||||
|
```
|
||||||
|
|
||||||
|
## Getting Help
|
||||||
|
|
||||||
|
If issues persist:
|
||||||
|
|
||||||
|
1. Check logs with `DEBUG=true`
|
||||||
|
2. Review documentation
|
||||||
|
3. Report issues with debug output
|
||||||
|
4. Include screenshot samples
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
- Explore [examples/](../examples/) directory
|
||||||
|
- Customize configuration for your workflow
|
||||||
|
- Create shell aliases for common tasks
|
||||||
|
- Integrate with CI/CD pipelines (if applicable)
|
||||||
53
examples/example_usage.sh
Executable file
53
examples/example_usage.sh
Executable file
@ -0,0 +1,53 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Example usage scenarios for Claude Vision Auto
|
||||||
|
|
||||||
|
echo "=== Claude Vision Auto - Example Usage ==="
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Example 1: Basic usage
|
||||||
|
echo "Example 1: Basic Interactive Session"
|
||||||
|
echo "Command: claude-vision"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Example 2: With prompt
|
||||||
|
echo "Example 2: With Initial Prompt"
|
||||||
|
echo "Command: claude-vision \"create a test.md file\""
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Example 3: Debug mode
|
||||||
|
echo "Example 3: Debug Mode"
|
||||||
|
echo "Command: DEBUG=true claude-vision"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Example 4: Custom timing
|
||||||
|
echo "Example 4: Custom Timing (slower system)"
|
||||||
|
echo "Command: IDLE_THRESHOLD=5.0 RESPONSE_DELAY=2.0 claude-vision"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Example 5: Different model
|
||||||
|
echo "Example 5: Use Llama 3.2 Vision"
|
||||||
|
echo "Command: VISION_MODEL=\"llama3.2-vision:latest\" claude-vision"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Example 6: Remote Ollama
|
||||||
|
echo "Example 6: Remote Ollama Server"
|
||||||
|
echo "Command: OLLAMA_URL=\"http://192.168.1.100:11434/api/generate\" claude-vision"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Example 7: Combined configuration
|
||||||
|
echo "Example 7: Combined Custom Configuration"
|
||||||
|
cat << 'EOF'
|
||||||
|
export OLLAMA_URL="http://localhost:11434/api/generate"
|
||||||
|
export VISION_MODEL="minicpm-v:latest"
|
||||||
|
export IDLE_THRESHOLD="4.0"
|
||||||
|
export RESPONSE_DELAY="1.5"
|
||||||
|
export DEBUG="true"
|
||||||
|
|
||||||
|
claude-vision "refactor authentication module"
|
||||||
|
EOF
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
echo "=== End of Examples ==="
|
||||||
|
echo ""
|
||||||
|
echo "Run any of these commands to test Claude Vision Auto"
|
||||||
|
echo "For more information, see docs/USAGE.md"
|
||||||
4
requirements-dev.txt
Normal file
4
requirements-dev.txt
Normal file
@ -0,0 +1,4 @@
|
|||||||
|
-r requirements.txt
|
||||||
|
pytest>=7.4.0
|
||||||
|
pytest-cov>=4.1.0
|
||||||
|
pytest-mock>=3.11.1
|
||||||
1
requirements.txt
Normal file
1
requirements.txt
Normal file
@ -0,0 +1 @@
|
|||||||
|
requests>=2.31.0
|
||||||
49
setup.py
Normal file
49
setup.py
Normal file
@ -0,0 +1,49 @@
|
|||||||
|
"""
|
||||||
|
Setup script for Claude Vision Auto
|
||||||
|
"""
|
||||||
|
|
||||||
|
from setuptools import setup, find_packages
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
# Read README
|
||||||
|
readme_file = Path(__file__).parent / "README.md"
|
||||||
|
long_description = readme_file.read_text() if readme_file.exists() else ""
|
||||||
|
|
||||||
|
setup(
|
||||||
|
name="claude-vision-auto",
|
||||||
|
version="1.0.0",
|
||||||
|
author="Svrnty",
|
||||||
|
author_email="jp@svrnty.io",
|
||||||
|
description="Vision-based auto-approval for Claude Code using MiniCPM-V",
|
||||||
|
long_description=long_description,
|
||||||
|
long_description_content_type="text/markdown",
|
||||||
|
url="https://git.openharbor.io/svrnty/claude-vision-auto",
|
||||||
|
packages=find_packages(),
|
||||||
|
classifiers=[
|
||||||
|
"Development Status :: 4 - Beta",
|
||||||
|
"Intended Audience :: Developers",
|
||||||
|
"License :: OSI Approved :: MIT License",
|
||||||
|
"Programming Language :: Python :: 3",
|
||||||
|
"Programming Language :: Python :: 3.8",
|
||||||
|
"Programming Language :: Python :: 3.9",
|
||||||
|
"Programming Language :: Python :: 3.10",
|
||||||
|
"Programming Language :: Python :: 3.11",
|
||||||
|
"Programming Language :: Python :: 3.12",
|
||||||
|
"Programming Language :: Python :: 3.13",
|
||||||
|
"Operating System :: POSIX :: Linux",
|
||||||
|
"Topic :: Software Development :: Libraries :: Python Modules",
|
||||||
|
"Topic :: Terminals",
|
||||||
|
"Topic :: Scientific/Engineering :: Artificial Intelligence",
|
||||||
|
],
|
||||||
|
python_requires=">=3.8",
|
||||||
|
install_requires=[
|
||||||
|
"requests>=2.31.0",
|
||||||
|
],
|
||||||
|
entry_points={
|
||||||
|
"console_scripts": [
|
||||||
|
"claude-vision=claude_vision_auto.main:main",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
include_package_data=True,
|
||||||
|
zip_safe=False,
|
||||||
|
)
|
||||||
79
tests/test_vision.py
Normal file
79
tests/test_vision.py
Normal file
@ -0,0 +1,79 @@
|
|||||||
|
"""
|
||||||
|
Basic tests for Claude Vision Auto
|
||||||
|
"""
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from unittest.mock import Mock, patch, MagicMock
|
||||||
|
from claude_vision_auto import config
|
||||||
|
from claude_vision_auto.vision_analyzer import VisionAnalyzer
|
||||||
|
from claude_vision_auto.screenshot import find_screenshot_tool
|
||||||
|
|
||||||
|
|
||||||
|
def test_config_defaults():
|
||||||
|
"""Test default configuration values"""
|
||||||
|
assert config.VISION_MODEL == "minicpm-v:latest"
|
||||||
|
assert config.IDLE_THRESHOLD == 3.0
|
||||||
|
assert config.RESPONSE_DELAY == 1.0
|
||||||
|
assert config.OUTPUT_BUFFER_SIZE == 4096
|
||||||
|
|
||||||
|
|
||||||
|
def test_find_screenshot_tool():
|
||||||
|
"""Test screenshot tool detection"""
|
||||||
|
tool = find_screenshot_tool()
|
||||||
|
# Should find at least one tool or return None
|
||||||
|
assert tool is None or isinstance(tool, str)
|
||||||
|
|
||||||
|
|
||||||
|
@patch('requests.get')
|
||||||
|
def test_vision_analyzer_connection(mock_get):
|
||||||
|
"""Test Ollama connection check"""
|
||||||
|
mock_response = Mock()
|
||||||
|
mock_response.json.return_value = {
|
||||||
|
'models': [
|
||||||
|
{'name': 'minicpm-v:latest'},
|
||||||
|
{'name': 'llama3.2-vision:latest'}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
mock_response.raise_for_status = Mock()
|
||||||
|
mock_get.return_value = mock_response
|
||||||
|
|
||||||
|
analyzer = VisionAnalyzer()
|
||||||
|
assert analyzer.test_connection() is True
|
||||||
|
|
||||||
|
|
||||||
|
@patch('requests.post')
|
||||||
|
def test_vision_analyzer_analyze(mock_post):
|
||||||
|
"""Test vision analysis"""
|
||||||
|
mock_response = Mock()
|
||||||
|
mock_response.json.return_value = {
|
||||||
|
'response': '1'
|
||||||
|
}
|
||||||
|
mock_response.raise_for_status = Mock()
|
||||||
|
mock_post.return_value = mock_response
|
||||||
|
|
||||||
|
analyzer = VisionAnalyzer()
|
||||||
|
|
||||||
|
# Create a temporary test image
|
||||||
|
import tempfile
|
||||||
|
with tempfile.NamedTemporaryFile(suffix='.png', delete=False) as tmp:
|
||||||
|
# Write a minimal PNG header
|
||||||
|
tmp.write(b'\x89PNG\r\n\x1a\n')
|
||||||
|
tmp_path = tmp.name
|
||||||
|
|
||||||
|
result = analyzer.analyze_screenshot(tmp_path)
|
||||||
|
assert result == '1'
|
||||||
|
|
||||||
|
# Cleanup
|
||||||
|
import os
|
||||||
|
os.unlink(tmp_path)
|
||||||
|
|
||||||
|
|
||||||
|
def test_approval_keywords():
|
||||||
|
"""Test approval keyword configuration"""
|
||||||
|
assert 'Yes' in config.APPROVAL_KEYWORDS
|
||||||
|
assert 'No' in config.APPROVAL_KEYWORDS
|
||||||
|
assert '(y/n)' in config.APPROVAL_KEYWORDS
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
pytest.main([__file__, '-v'])
|
||||||
Loading…
Reference in New Issue
Block a user