Commit Graph

108 Commits

Author SHA1 Message Date
Pouzor 0863c2db94 feat(zigbee): import as background scan run
Mirrors IP scan flow: POST /zigbee/import-pending now creates a
ScanRun(kind=zigbee, status=running) and returns immediately.
Networkmap fetch + pending upsert run in the background, status
transitions to done/error when finished.

Frontend: import modal closes on submit, scan history shows the
run with a ZIG/IP kind chip and toasts on completion. Pending
modal auto-refreshes when run finishes.

scan_runs.kind column added (default 'ip', idempotent migration).
Existing zigbee tests refactored to exercise _persist_pending_import
directly (background tasks don't see the test session); route test
verifies the run is created with kind=zigbee.
2026-05-10 00:45:21 +02:00
Pouzor 277879488f feat(pending): full-screen modal with grid cards, filters, bulk restore
Replaces sidebar pending/hidden panels with a wide modal showing
devices as cards. Adds search, segmented source/status filters,
type filter, select mode for bulk approve/hide/restore, and
keyboard shortcuts. Hidden cards click-to-restore (no approval
detour); approval no longer pops the edit modal.

Backend: new restore + bulk-restore endpoints (hidden -> pending).
2026-05-09 23:40:44 +02:00
Pouzor 456f26a85a fix(ci): null-safe pending IP in SearchModal + bump python-multipart
- SearchModal.tsx: same null-IP guard as SearchBar; also search by
  friendly_name and ieee_address. Fixes Docker CI typecheck.
- python-multipart 0.0.26 -> 0.0.27 (CVE-2026-42561). Fixes
  Security pip-audit.
2026-05-08 00:53:18 +02:00
Pouzor 2a6140a6b1 chore(db): log instead of silently suppress new Zigbee migrations
Pre-push code review flagged the inline `with suppress(OperationalError)`
ALTERs as silently eating real errors. The project still uses inline
init_db migrations (CLAUDE.md aspires to Alembic but the runtime path
is inline), so as a minimum: add a `_try_migrate` helper that
distinguishes already-applied (debug log) from genuine failures
(warning log). Apply it to the Zigbee-related migrations only — the
legacy ALTERs above remain unchanged in this commit's scope.
2026-05-07 23:57:52 +02:00
Pouzor 3ae159d8d6 feat(zigbee): import to pending section with edge persistence
Coordinator auto-approves to a canvas Node; routers/end devices land in
pending_devices keyed by IEEE. Discovered parent->child edges are stored
in pending_device_links so that approving a pending device later
auto-creates the Edge once both endpoints exist as canvas Nodes.

- new POST /api/v1/zigbee/import-pending (default mode in modal)
- new pending_device_links table; ieee_address on nodes + pending_devices
- pending_devices.ip migrated to nullable (table rebuild on existing DBs)
- approve / bulk-approve return auto-created edges; sidebar pushes them
  into the canvas store with bottom -> top-t handles
- ZigbeeImportModal: radio toggle pending vs canvas; reset on close
- PendingDeviceModal: zigbee badge, IEEE/LQI/vendor/model rows, services
  hidden for zigbee
- Sidebar pending row: ZIG source badge, LQI badge, friendly_name fallback
- SearchBar: null-safe IP, also searches friendly_name and ieee_address
- Tooltip trigger uses asChild to avoid nested-button hydration error
2026-05-07 23:18:29 +02:00
Pouzor 5e567d4628 feat(zigbee): add slow-fetch hint in modal + bump timeout to 300s
- Modal: small italic note under Test/Fetch buttons warning users that
  large meshes can take several minutes
- Service: _NETWORKMAP_TIMEOUT 180s -> 300s (5 min) for very large meshes
2026-05-07 00:27:48 +02:00
Pouzor 05db9a59f5 fix(zigbee): strict tree edges + auto-select imported nodes
Edges
- Z2M `links` is bidirectional and includes router mesh paths, which
  caused duplicate edges and edges entering the coordinator from the
  bottom. Walk `links` only to derive parent_id + LQI; build final
  edges strictly from the parent->child hierarchy (one edge per
  non-coordinator node). Result: parent bottom -> child top, every time.
- Tests: bidirectional pair collapses to one edge, router-mesh siblings
  dropped, coordinator never receives an edge.

Auto-select
- After import, deselect existing canvas nodes and mark only the
  freshly-imported ones as selected, so the user can drag the whole
  subtree as a group.
2026-05-06 23:53:31 +02:00
Pouzor a46e505505 fix(zigbee): parse real Z2M networkmap shape (data.value.nodes/links)
The previous parser read `data.routes` which is just an echo of the
`routes` request flag (a boolean). On real brokers this caused
`TypeError: 'bool' object is not iterable` and 500s during /import.

- Rewrite parse_networkmap to read data.value.nodes + data.value.links
  with fallback to data.{nodes,links} for legacy variants
- Defensive: drop links to unknown nodes, propagate lqi from link to
  target node, extract model/vendor from definition block
- Bump networkmap timeout 10s -> 180s (large meshes are slow)
- Tests: rewrite fixture builders + sample payload to real Z2M shape;
  add cases for legacy shape, routes:false echo (regression), malformed
  list, link to unknown node, lqi propagation, definition extraction
- Update docs to mention 60s+ wait window

53 backend tests pass, mypy + ruff clean.
2026-05-06 22:38:15 +02:00
Pouzor 9970780e7a fix(zigbee): sanitize MQTT error messages to prevent credential leakage
aiomqtt/paho exception strings can include the broker URI with embedded
credentials (mqtt://user:pass@host) or auth detail. The 502 response
from /import and the message field on /test-connection echoed these
verbatim via str(exc).

- Add _sanitize_mqtt_error() that maps known patterns (auth, refused,
  DNS, TLS, timeout) to coarse, credential-free categories
- Original exception still logged at WARNING level for operator debug
- Drop hostname:port from TimeoutError messages
- /test-connection unexpected-error path no longer interpolates exc

Tests: 6 new (auth/refused/DNS/TLS/unknown sanitization, end-to-end
fetch_networkmap leak check).
2026-05-06 14:55:51 +00:00
Pouzor d49610163f feat(zigbee): add MQTT TLS support with optional cert verify skip
- Schema: mqtt_tls + mqtt_tls_insecure flags on import + test-connection
  requests; model_validator enforces insecure requires tls
- Service: _build_tls_context() using ssl.create_default_context();
  logger.warning when verification disabled; tls_context plumbed into
  aiomqtt.Client for both fetch_networkmap and test_mqtt_connection
- Route: passes tls flags through to service
- Frontend: TLS checkbox auto-toggles port 1883<->8883 unless user
  edited; insecure checkbox disabled until TLS on, red-tinted; password
  field marked autocomplete=new-password
- Tests: 7 new backend tests (TLS context build, client kwargs assertion,
  router happy path, insecure-without-tls 422)
2026-05-06 16:30:23 +02:00
pranjal-joshi 67b0620be2 fix: resolve all mypy errors in zigbee_service
- Add dict[str, Any] type parameters throughout (mypy type-arg)
- Fix str | None returns with explicit 'or None' coercion (no-any-return)
- Remove unused type: ignore[import] comment (unused-ignore)
- Decode message.payload bytes before json.loads (arg-type)
- Use X | Y syntax in isinstance check (ruff UP038)
- All 33 tests still passing, ruff + mypy clean

Co-authored-by: CyberKeys <noreply@openclaw.ai>
2026-05-04 14:25:15 +00:00
pranjal-joshi cc9c010002 fix: adhere to CONTRIBUTING.md — ruff clean + all tests passing
- Move aiomqtt to module-level import (enables proper patch() in tests)
- Remove unused variable (description) in zigbee_service — ruff F841
- Split long line (132 chars) to fit 120 char limit — ruff E501
- Fix import sort order in test files — ruff I001
- Remove unused imports (asyncio, AsyncMock, MagicMock) — ruff F401
- Rename test_mqtt_connection import alias to _test_mqtt_connection
  to avoid pytest fixture name collision (ERROR at setup)
- All 33 backend tests now pass (21 service + 12 router)
- TypeScript typecheck: 0 errors

Co-authored-by: CyberKeys <noreply@openclaw.ai>
2026-05-04 14:12:38 +00:00
pranjal-joshi 103e24e5fa feat: add Zigbee2MQTT network map importer
- Backend: async MQTT service (aiomqtt) to fetch Z2M networkmap via bridge API
- Backend: FastAPI router at /api/v1/zigbee with /import and /test-connection
- Backend: Pydantic v2 schemas for request/response validation
- Backend: coordinator → router → end-device parent_id hierarchy builder
- Frontend: ZigbeeImportModal with MQTT config form, Test Connection, Fetch Devices
- Frontend: device list grouped by type (coordinator/router/enddevice) with checkboxes
- Frontend: ZigbeeCoordinatorNode, ZigbeeRouterNode, ZigbeeEndDeviceNode canvas nodes
- Frontend: Zigbee Import button in sidebar alongside Scan Network
- Frontend: handleZigbeeAddToCanvas wires selected devices + edges onto canvas
- Tests: full unit test suite for parser, hierarchy builder, MQTT mocks
- Tests: API endpoint tests for /zigbee/import and /zigbee/test-connection
- Tests: Vitest component tests for ZigbeeImportModal
- Docs: docs/zigbee-import.md with full usage, MQTT config, troubleshooting guide
- Docs: README.md Zigbee2MQTT Import section

Co-authored-by: CyberKeys <noreply@openclaw.ai>
2026-05-04 13:58:58 +00:00
Pouzor babbcb1dc5 feat: add custom style theme with per-type node/edge style editor
- New 'custom' ThemeId with dedicated ThemeCard + pencil edit button
- CustomStyleModal: two-column editor for all node types (excl. groupRect/group) and edge types
  - Per node: border, background, icon color + opacity, default size (w/h)
  - Per edge: color + opacity, path style, animation
  - 'Apply to existing [Type]' per type, 'Apply All to Canvas' footer action
- canvasStore: applyTypeNodeStyle, applyTypeEdgeStyle, applyAllCustomStyles actions
- themeStore: customStyle state + setCustomStyle action
- Backend: custom_style JSON column on canvas_state (ALTER TABLE migration), saved/loaded with canvas
- App.tsx: custom_style included in save payload, restored on load (API + standalone)
- Tests: +8 frontend (themeStore + canvasStore), +3 backend (canvas API)
2026-04-24 11:42:31 +02:00
findthelorax c7be851c34 feature: add support for services to use a path 2026-04-20 14:37:08 +02:00
Pouzor 7608d07255 fix: flush before reading node IDs in bulk/single approve; 404/409 guards; catch scan errors
- db.flush() ensures node.id is populated before reading — fixes bulk approve
  where node_ids were null, causing frontend to skip addNode for every device
- approve_device raises 404 on missing device, 409 on already-processed device
- _background_scan rollbacks dirty session then marks run as "failed"
- Explicit Node() field mapping instead of **model_dump() to prevent injection
- update_scan_config rolls back in-memory change if save_overrides() fails
- clear_pending uses bulk DELETE instead of N individual row deletes
2026-04-20 10:41:02 +02:00
Pouzor 0193f933ce feat: bulk approve/hide pending devices (#70)
- Backend: POST /scan/pending/bulk-approve and /scan/pending/bulk-hide endpoints (registered before dynamic routes to avoid conflict); bulk-approve response includes device_ids for frontend mapping
- Frontend: PendingDevicesPanel gains per-row checkboxes, select-all, and a bulk action bar (Approve N / Hide N) that appears when ≥1 device is selected
- Tests: 6 new backend API tests + 7 new frontend UI tests for bulk selection flows
2026-04-19 22:13:40 +02:00
Pouzor ef96cafcc8 feat: IPv6 support and multi-IP per node (closes #60)
- maskIp handles IPv6 addresses (masks second and last group)
- maskIp handles comma-separated IP strings (masks each address)
- Add splitIps() helper to parse comma-separated IP field
- Add primaryIp() helper used by status checker (first IP wins)
- BaseNode renders each IP on its own line when comma-separated
- NodeModal placeholder shows comma-separated example
- Backend status_checker uses only first IP for connectivity checks
- Expand maskIp test suite: IPv6, comma-separated, splitIps, primaryIp
2026-04-19 22:13:40 +02:00
Pouzor 6c9974b357 bump version 1.10 2026-04-19 22:13:40 +02:00
Pouzor 0019c086cf feat: automatic DB backup before migrations using VERSION file
- Add VERSION file at repo root as single source of truth for app version
- frontend/vite.config.ts reads VERSION file instead of package.json
- backend config.py exposes APP_VERSION read from VERSION (dev) or /app/VERSION (Docker)
- database.py backs up DB to homelab.db.back-{version} before running migrations
  (skipped if DB doesn't exist or backup already exists — fully idempotent)
- Dockerfile.backend and Dockerfile.frontend copy VERSION into the image
- Add test_db_backup.py with 4 tests covering create/skip/idempotent/version cases
2026-04-19 22:09:43 +02:00
Pouzor a7b244502e fix: bump pytest-asyncio to 1.3.0 (first version supporting pytest 9) 2026-04-18 21:52:32 +02:00
Pouzor 72d5a51b44 fix: bump pytest-asyncio to 1.0.0 for pytest 9 compatibility
pytest-asyncio 0.26.0 requires pytest<9; 1.0.0 supports pytest 9.x
2026-04-18 21:45:10 +02:00
Pouzor 12f46715c1 fix: bump python-multipart, pytest, pytest-asyncio to fix CVEs
- python-multipart 0.0.22 → 0.0.26 (CVE-2026-40347)
- pytest 8.3.3 → 9.0.3 (CVE-2025-71176)
- pytest-asyncio 0.24.0 → 0.26.0 (pytest 9 compat)
2026-04-18 21:36:09 +02:00
Pouzor e666abefad chore: remove db backup from tracking and add *.db.back to gitignore 2026-04-09 16:34:25 +02:00
Pouzor 4844576c3b feat: add Basic animation type for edges
Adds a new "basic" animation mode that uses React Flow's native animated
dashes — the entire stroke moves as dashes, with no solid line underneath.
Distinct from "snake" (moving dot) and "flow" (overlay on solid line).

- Frontend: new Basic button in EdgeModal, animMode mapping, passes
  animated={true} to BaseEdge when mode is "basic"
- Backend: normalize_animated accepts "basic" as a valid value
- Tests: EdgeModal + canvas round-trip tests for basic mode
2026-04-09 15:46:52 +02:00
Pouzor 4976f2e694 test: improve canvas route coverage to 100% and fix async tracing
- Add 22 new tests for canvas save/load: properties round-trip,
  edge waypoints, edge handles, animated, parent/child nodes,
  group nodes, all node fields, bottom_handles, services/notes,
  check_method/target, viewport update branches, edge update branch
- Fix coverage.py async tracing by setting core = "sysmon" in
  pyproject.toml (Python 3.12+ sys.monitoring API), resolving the
  sys.settrace drop-after-await issue
2026-04-09 15:17:58 +02:00
Pouzor 94c6ac7fa7 fix: add type parameters to bare list annotations for mypy compliance 2026-04-09 14:23:13 +02:00
Pouzor e0f96001e2 chore: bump version to 1.9.0 2026-04-09 14:06:24 +02:00
Pouzor 3d89ba3b6f feat: replace static hardware fields with dynamic node properties
Replaces the 4 fixed hardware columns (cpu_count, cpu_model, ram_gb,
disk_gb) with a flexible properties system. Each property has a key,
value, icon (from a curated Lucide picker), and a visibility toggle
that controls whether it appears on the canvas node card.

- Backend: add `properties` JSON column to Node model; data migration
  converts existing hardware rows to properties with correct icons
  (idempotent, old columns kept for safety)
- Backend: add `properties` to NodeBase, NodeUpdate, NodeSave schemas
  and canvasSerializer so values survive canvas save/load
- Frontend: add NodeProperty type; new propertyIcons.ts registry (20
  icons); BaseNode renders visible properties with legacy hardware
  fallback for unmigrated nodes
- Frontend: DetailPanel gains interactive properties section (add /
  edit / remove / toggle visibility / icon picker) replacing the
  read-only hardware block; hardware section removed from NodeModal
- Tests: 6 migration tests, 7 API tests, 8 DetailPanel property tests,
  6 BaseNode render/fallback tests, 9 propertyIcons util tests
2026-04-09 13:47:36 +02:00
Pouzor 3afc8ed3d8 fix: persist edge waypoints in backend
Add waypoints JSON column to edges table, include it in all edge
schemas (EdgeBase, EdgeUpdate, canvas CanvasEdge) and add the
idempotent ALTER TABLE migration so existing databases are upgraded
on next startup.
2026-04-08 23:31:53 +02:00
Pouzor 75c7f25a30 feat: display app version in sidebar with GitHub release check 2026-04-08 12:22:39 +02:00
Pouzor 8bd1c48976 chore: bump version to 1.8.3 2026-04-07 01:04:10 +02:00
Pouzor 3bd18ab543 chore: bump version to 1.8.2 2026-04-05 15:01:57 +02:00
Pouzor 07da498d18 fix: use correct ping args on Windows (-n/-w instead of -c/-W) 2026-04-05 14:08:36 +02:00
Pouzor 0e59f15608 chore: bump version to 1.8.1 2026-04-05 10:32:37 +02:00
Pouzor 08a3a689c5 chore: bump version to 1.8.0 2026-04-04 23:29:20 +02:00
Pouzor 38c5bcb606 feat: configurable bottom handles, scanner rewrite, UI polish
## New features
- Configurable bottom connection points per node (1–4 handles)
- Fit view on load
- LiveView improvements
- Node modal: inline Type/Icon picker, default icon in trigger
- Remove redundant Save button from ScanConfigModal

## Scanner fixes
- Phase 1: replace nmap ARP sweep with concurrent asyncio ping sweep
  (50 parallel pings, 1s timeout). Zero false positives, works in any
  Docker network mode. Supplements with /proc/net/arp for ICMP-blocked devices.
- Phase 2: explicit -sS (root) / -sT (non-root) scan type; bump
  host-timeout to 60s; gather(return_exceptions=True) so one failing
  host doesn't abort the batch
- Fix 404 on missing device in hide/ignore
- Validate CIDR ranges to prevent nmap injection
- Thread-safe cancel set, pre-fetch canvas/hidden IPs (no N+1 queries)
- Logging: attach StreamHandler to root logger so app.* logs are visible

## Tests
- 21 backend scanner tests (ping sweep, ARP cache, Phase 2 tolerance)
- Full NodeModal coverage (53 tests)
- LiveView, store, edge label tests
2026-04-04 23:15:47 +02:00
Pouzor cf7777e0af feat: improve IoT device detection via two-phase scan and mDNS discovery
- Two-phase nmap: ARP sweep first to find all alive hosts (incl. IoT with
  no open TCP ports), then port scan only alive hosts
- mDNS/Bonjour discovery via zeroconf for Shelly, ESPHome, HomeKit devices
- Add CoAP ports (5683, 5684, 4915) to port scan and IoT type hints
- Expand MAC OUI table with Shelly, Espressif, Sonoff, Tapo, Hue, IKEA, Tuya
- IoT vendor MAC takes precedence over generic HTTP port type hints
- Reorder suggest_node_type priority: iot now beats server
2026-04-02 17:41:51 +02:00
Pouzor 255443b8e1 fix: harden scheduler startup and interval validation
- Wrap shutdown() in try/except to prevent double-scheduler if teardown fails
- Guard reschedule_status_checks with interval_seconds >= 10 validation
2026-04-02 17:39:52 +02:00
Pouzor 7e24878077 fix: pass scalars to _check_single_node to prevent DetachedInstanceError
- Refactor _check_single_node to accept plain scalar args (node_id,
  check_method, check_target, ip) instead of a detached ORM Node object
- Extract scalars inside the session in _run_status_checks before it closes
- Capture datetime.now() once per check to keep DB and WebSocket timestamps consistent
2026-04-02 17:32:42 +02:00
Pouzor ff1bc7340d fix: resolve mypy errors in scheduler (dict type params + str | None arg) 2026-04-02 17:24:26 +02:00
Pouzor e4bfab7e58 perf: run status checks concurrently to prevent scheduler overrun
Replace sequential node checks with asyncio.gather so all nodes are
checked in parallel. Add coalesce=True + max_instances=1 to suppress
APScheduler "maximum instances reached" log spam.
2026-04-02 11:41:08 +02:00
Pouzor 057891f7d5 feat: add stop scan button in UI with backend cancellation support
- POST /scan/{run_id}/stop endpoint signals running scan to cancel
- Scanner checks cancellation flag between CIDR ranges and hosts, exits early
- Cancelled scans get status 'cancelled' instead of 'done'
- Stop button (red StopCircle) shown in Scan History panel for running scans
- 6 new backend tests, 5 new frontend tests
2026-04-01 14:18:44 +02:00
Pouzor 1f884fd1db fix: guard scheduler against double-start and unguarded reschedule 2026-03-31 00:03:01 +02:00
Pouzor e9152df17a fix: remove stale reschedule call from scan config after settings endpoint split 2026-03-31 00:01:31 +02:00
Pouzor e7fc091701 fix: reschedule APScheduler job immediately when status check interval is updated
Interval was read once at startup — changing it via API had no effect
until server restart. Now calls reschedule_status_checks() after saving.
2026-03-30 23:59:47 +02:00
Pouzor 68c7672cea feat: split scan config and app settings into separate endpoints
- New GET/POST /api/v1/settings for status check interval
- Scan /api/v1/scan/config now handles ranges only
- Frontend: settingsApi client, SettingsPanel uses settingsApi
- ScanConfigModal no longer reads/writes interval
- 4 new backend tests for settings endpoint
2026-03-30 23:06:51 +02:00
Pouzor ce4af14ee6 feat: deduplicate pending devices and skip canvas/hidden nodes on scan
- At scan start, purge any pending entries whose IPs already exist in canvas
- Skip canvas nodes (approved) during scan — don't re-add to pending
- Skip hidden devices during scan — respect user's hide decision
- Add 4 tests covering all new behaviors
2026-03-30 22:01:47 +02:00
Pouzor 9c92d39629 fix: return 401 (not 500) when bcrypt hash is malformed (#21)
- verify_password catches ValueError from passlib so a mangled hash
  ($ signs stripped by shell/Docker) returns False instead of crashing
- Settings.check_password_hash logs a clear startup error with fix
  instructions when AUTH_PASSWORD_HASH doesn't start with '$2'
2026-03-29 16:03:40 +02:00
Pouzor 32b60a201b fix: persist edge animation mode (None/Snake/Flow) end-to-end
- canvasStore.onConnect: include animated in edge data object (was silently dropped)
- Backend schemas: normalize animated bool/int to string ('none'/'snake'/'flow') via field_validator
- ORM model: change animated column from Boolean to String
- DB migration: convert existing 0/1 boolean rows to 'none'/'snake' strings
2026-03-29 14:41:18 +02:00