nmap -sV sends plaintext probes to TLS ports (e.g. Proxmox 8006). They
stall until the host crosses --host-timeout, at which point nmap skips
the whole host and discards ports it already found — a reachable host is
returned with 0 open ports (issue #277).
Split Phase 2 into two passes: Pass A (-sS/-sT --open) discovers ports
fast with no version detection; its ports are authoritative. Pass B
(-sV --host-timeout, scoped to the found ports) enriches banners
best-effort — on timeout/failure the discovered ports are kept with
empty banners instead of losing the host.
Healthy hosts still get full-intensity -sV banners (no quality drop).
Version host-timeout is now configurable via SCANNER_VERSION_HOST_TIMEOUT
(default 60s).
ha-relevant: yes
Node.ip stores several comma-separated addresses once a user adds an
IPv6 (e.g. "fe80::1, 192.168.1.5"). All placement/inventory matching did
exact string equality on that field, so a device scanned as the plain
IPv4 looked absent from the canvas: canvas_count stayed 0, the "In N
canvas" badge and hide-on-canvas button vanished, and bulk-approve
re-placed a duplicate node.
Match per-address instead, and add MAC (a stable identifier immune to IP
edits) as a cumulative match key alongside ieee/ip:
- _canvas_correlation indexes ip per token + by_mac
- bulk_approve skip detection tokenizes ip + tracks mac
- find_duplicate_node narrows with Node.ip.contains then confirms per
token in Python (guards the 10.0.0.4 / 10.0.0.40 substring false match)
ha-relevant: maybe
Single-device approve now guards duplicates per-design the same way bulk
approve does, and asks the user instead of failing or silently merging:
- create_node and approve_device reject a same-design duplicate (ieee, ip
or mac) with 409 + the existing node; a force flag creates it anyway.
- Frontend shows a confirm dialog: go to existing node, add duplicate
anyway, or cancel.
- approve_device no longer rejects a device already on another canvas
(status is global, canvas membership is per-design) — it can be placed
on a new design, matching bulk approve.
- IEEE (Zigbee/Z-Wave) devices now use the same prompt as ip/mac instead
of auto-merging into the existing node.
- bulk approve reports which devices it skipped as duplicates.
Closes#260
ha-relevant: maybe
Reconcile the same physical device discovered by both the nmap IP scan and
the Proxmox importer into a single inventory row, keyed on MAC. Previously
each path only deduped by IP, and the importer captured no MAC (and no IP for
stopped guests), so most guests double-listed.
Backend:
- mac_utils.normalize_mac: canonical MAC (lowercase, ':'-separated), the
cross-source join key. Normalized on write and on compare.
- proxmox_service: capture the guest NIC MAC agent-free from the net0 config
(qemu virtio=<MAC>, lxc hwaddr=<MAC>); works for stopped guests. Resolver
now returns (ip, mac).
- proxmox persist: match existing Node/PendingDevice by ieee OR ip OR MAC;
fill mac, keep the vm/lxc type, union sources.
- scanner persist: match PendingDevice by ip OR MAC; fill the IP a Proxmox
import lacked, keep a pve row's type, union the scan source. Stamp query
matches raw + normalized MAC (legacy-safe).
- Multi-source tags: new PendingDevice.discovery_sources JSON column so a
merged device shows under both the IP and Proxmox filters. Idempotent
migration backfills from discovery_source (legacy NULL-scalar rows with an
IP become ["arp"]). _sources_after_merge preserves a scanned row's IP origin
through the merge without tagging a pure Proxmox guest.
- Import now broadcasts a scan update on completion so an open inventory
reloads without a manual refresh.
Frontend:
- pendingSources: sourceBuckets/orderedSources map discovery_sources to filter
buckets; a device with ["arp","proxmox"] matches both filters and renders
both badges. PendingDevicesModal filter + badges use them.
Tests: MAC normalization, config MAC capture, cross-source merge both
directions, legacy-row IP-tag preservation, no-false-IP-tag guard, refresh
broadcast, and the frontend bucket mapping.
ha-relevant: maybe
Scan History:
- Proxmox is now a first-class scan kind (badge, filter chip, Server icon,
completion toast) instead of being mislabeled as an IP scan.
- A done run carrying a non-fatal advisory renders amber (info) with a
warning toast, distinct from red failures.
Import diagnostics:
- test-connection probes /access/permissions and warns when the API token
has no ACL (VMs/LXC would be invisible) — points at the PVEAuditor grant.
- import surfaces an advisory when hosts import but no guests are visible
(privilege-separated token whose rights are the intersection with the user),
rather than a silent "done".
Node style:
- Proxmox container mode is now opt-in (container_mode === true), matching the
rest of the codebase (App.tsx nesting logic). Imported proxmox nodes leave
the flag unset and render like a manually-created node instead of an empty
group container.
Cluster edges:
- Hosts from one import are chained with 'cluster' edges via left/right handles,
distinct from the vertical host->guest 'virtual' edges. Wired for both the
direct "Add to Canvas" path and the pending -> approve path (host<->host
proxmox_cluster links, resolved to cluster edges on approve; cluster hosts
get left/right handles).
Tests added on both sides.
ha-relevant: maybe
Add a Proxmox VE importer that reads the /api2/json REST API with a read-only
API token and drops hosts (proxmox), VMs (vm) and LXC containers (lxc) onto the
canvas as typed nodes with run state and hardware specs (vCPU/RAM/disk).
- Backend: proxmox_service (httpx) + proxmox routes (test-connection, import,
import-pending, config). Two-tier dedupe — merge onto an existing scanned node
by IP, else synthetic pve-{host}-{vmid} identity. Update-in-place, never
deletes. Host->guest rendered as a 'virtual' edge via the pending-link flow.
- Security: token is env-only (PROXMOX_TOKEN_*), never written to disk by the
app, never returned by any endpoint; errors are credential-sanitized.
- Auto-sync: optional scheduled re-import into pending (APScheduler job).
- PendingDevice.properties carries specs through approve (+ migration).
- Frontend: ProxmoxImportModal, sidebar entry, pending inventory source filter,
Settings auto-sync section, proxmoxApi client.
- Docs: docs/proxmox-import.md, README + FEATURES sections, .env.example keys.
- Tests: backend service/router/scheduler, frontend modal/client/pending.
ha-relevant: maybe
Zigbee and Z-Wave imports looked up canvas nodes by ieee_address with
scalar_one_or_none(), assuming one node per IEEE globally. A device placed
on two designs (one Node per canvas — a supported feature) made re-import
crash with MultipleResultsFound.
- Both mesh imports now refresh properties on every matching node instead
of a single row (loop over .scalars().all()).
- approve_device guards against a true duplicate: same IEEE already on the
SAME design reuses that node instead of inserting a second one.
- New node_dedupe service: loss-free repair keyed on (ieee, design_id).
Collapses only genuine same-canvas duplicates (merges properties/services/
missing fields, re-points edges + parent_id, drops self-loops/parallel
edges). Cross-design placements preserved. Runs at start of both imports
and bulk-approve. No-op on healthy DBs.
- IP correlation path already handled multiple nodes; left unchanged.
Tests: dedupe unit tests (collapse, cross-design preservation, edge/parent
re-point, idempotent), zigbee + zwave multi-canvas regression, approve
no-dupe guard.
ha-relevant: yes
Stop button had no effect: run_scan only checked the cancel flag between
CIDR ranges and between hosts, never inside the per-range nmap call. For a
single /24 the whole scan is one blocking call, so cancel was ignored for
minutes and the run status stayed 'running'.
- thread run_id into _nmap_scan/_ping_sweep/_nmap_port_scan; check cancel
before each phase and skip queued hosts once cancelled
- flip ScanRun status to 'cancelled' eagerly in the stop endpoint so the UI
reacts immediately instead of waiting for a checkpoint
Fixes#218
ha-relevant: yes
Import a Z-Wave JS UI (zwavejs2mqtt) network over the MQTT gateway API,
mirroring the existing Zigbee pipeline:
- New Z-Wave Import modal + sidebar entry (broker, prefix, gateway name)
- coordinator/router/end-device typing with mesh tree from node neighbors
- import to Pending section or straight to canvas
- Pending Devices gains a Z-Wave source filter
- shared mqtt_common helpers extracted from the zigbee service
ha-relevant: yes
A re-scan of an already-approved device created a fresh pending row each
time (the upsert only matched status=pending), so one device could show
several times in the inventory. Now the scanner keeps one non-hidden row
per IP, refreshing it in place and preserving an approved status, and
collapses any pre-existing duplicates at scan start.
ha-relevant: maybe
Reworks the "Pending Devices" panel into a "Device Inventory": scanned
devices already placed on a canvas are no longer suppressed — they stay
listed and badged with how many canvases they appear on.
- scanner: stop deleting/skipping on-canvas IPs (hidden still suppressed)
- scan API: /pending returns all non-hidden devices; compute canvas_count
by correlating ip/ieee_address against nodes grouped by design
- frontend: rename to "Device Inventory", top-right canvas-count corner,
toggle to show/hide on-canvas devices (default show)
ha-relevant: maybe
run_scan() accepts DeepScanOptions: user port ranges are validated and appended
to the nmap -p list; when http_probe_enabled, open ports are probed for HTTP
signals before fingerprinting. Defaults (no options) reproduce the standard
scan exactly — probe is never called, port list unchanged.
ha-relevant: yes
probe_port() GETs https:// then http:// for an open port and returns the page
<title> plus Server/X-Powered-By headers as identifying signals. Non-web ports
(SSH, DB, etc.) are skipped; body read is capped at 64KB; 3s timeout.
probe_open_ports() fans out over all open ports with a bounded semaphore.
ha-relevant: yes
Adds match_service() priority walk for fingerprinting:
1. port + http_regex confirmed
2. port + banner_regex confirmed
3. port:null + http_regex confirmed (custom-port services)
4. port-only fallback
http_regex is strict only once a probe has run; with no probe signals it
degrades to port-only matching, preserving pre-probe behaviour. match_port
kept as a probe-less alias.
ha-relevant: yes
Moves the OUI mapping from an inline dict in fingerprint.py into
data/oui_database.json grouped by vendor, matching the service_signatures
data-file pattern. Adds ~100 curated OUIs covering MikroTik, Ubiquiti,
Synology, QNAP, Cisco, Aruba, Juniper, Hikvision, Dahua, Reolink, Axis,
Raspberry Pi, Dell, Supermicro, and others. Existing IoT vendors and
hypervisor OUIs are preserved.
For multi-product vendors the OUI is tagged with the most common homelab
category (e.g. Ubiquiti -> ap) and port hints in suggest_node_type
continue to upgrade ambiguous matches (Ubiquiti + BGP -> router).
Per-service checks now only probe HTTP(S)-reachable services. SSH (22) and
other non-web ports (DB, mail, DNS, raw TCP) stay 'unknown' (grey category
colour) rather than going red — an open TCP socket doesn't prove the service
is healthy, and a firewalled port flapped red misleadingly.
ha-relevant: yes
Adds optional live status checking per service (not just per node),
requested as a follow-up to issue #196.
Backend:
- New check_service / check_services: HTTP(S) GET for web services, TCP
connect otherwise; UDP and port-less non-web services stay 'unknown'.
- New scheduler job 'service_checks', independent interval (default 300s),
added/removed live via set_service_checks_enabled.
- Settings gain service_check_enabled + service_check_interval (>=30s),
persisted to scan_config.json. New WS message type 'service_status'.
Frontend:
- Live per-service status overlay in canvasStore (not persisted, so it
never round-trips through canvas save), fed by the WS message.
- DetailPanel + canvas node service rows: offline service turns red
(#f85149), otherwise keeps its category colour.
- SettingsModal: toggle + interval input (default 300s / 5 min).
Off by default — no behaviour change until enabled.
ha-relevant: yes
Addresses three reports from issue #196:
- Ping now sends 2 probes with a ~2s timeout (was 1 probe / 1s) so a
single dropped packet or a slow IoT/ESPHome device no longer flaps a
node offline (#196.1, #196.2).
- IPv6-only devices (e.g. Alexa) are now pinged over IPv6: ping6 on
macOS, -6 flag on Linux/Windows, detected via inet_pton (#196.3).
- Manually-added services carry no category and so always rendered grey
even when they were reachable HTTP/HTTPS. A resolvable web URL now
falls back to the web colour (#196.9).
ha-relevant: yes
- Replace passlib.CryptContext with bcrypt 4.2.1 directly (passlib is
unmaintained and only emitted warnings when paired with bcrypt 4+).
hash_password / verify_password now call bcrypt.checkpw / .hashpw, and
verify_password is hardened against empty inputs without raising.
scripts/hash_password.py + tests/conftest.py updated to match.
- status_checker.check_node() now rejects targets starting with '-' before
any subprocess invocation, defending ping/tcp paths against arg-injection
if an admin sets a check_target like '-O'.
Tests added:
- test_auth: expired JWT, malformed JWT, wrong-secret JWT, empty password
vs empty server hash, verify_password edge cases.
- test_scan: _background_scan failure path marks ScanRun failed, leaves
non-running terminal status alone, success path invokes run_scan.
- test_status_checker: ping/tcp invocations are bypassed when target or
ip starts with '-'.
Backend coverage 89% → 90%.
- New zigbee props (approve + first-time re-import) ship with visible=false
so the canvas card stays clean. User opts in from the right panel.
- On re-import of an already-approved node, merge instead of overwrite:
keys already present keep their visible flag (and any user-edited value
is replaced with the freshly imported one), brand-new keys are appended
hidden. Non-zigbee custom properties are preserved untouched.
- Approval (single + bulk) of zigbee pending devices now writes IEEE, Vendor,
Model, LQI into Node.properties so they show in the right panel.
- Zigbee re-import refreshes properties on existing canvas Nodes and skips
creating a pending row when the device was already approved — keeps
approved devices out of pending/hidden modals.
- Coordinator Node also receives the same properties on first creation and
on re-import.
- Remove the Parent Container selector from the add/edit node modal.
ping(8) -W flag is milliseconds on macOS, seconds on Linux. Code used
-W 1 for both, giving macOS a 1ms timeout that fails any host with
RTT >1ms (typical wifi/mesh is 2-10ms).
Use -W 1000 on darwin to match the intended 1s timeout. Add regression
test asserting the platform-specific value.
- Modal: small italic note under Test/Fetch buttons warning users that
large meshes can take several minutes
- Service: _NETWORKMAP_TIMEOUT 180s -> 300s (5 min) for very large meshes
Edges
- Z2M `links` is bidirectional and includes router mesh paths, which
caused duplicate edges and edges entering the coordinator from the
bottom. Walk `links` only to derive parent_id + LQI; build final
edges strictly from the parent->child hierarchy (one edge per
non-coordinator node). Result: parent bottom -> child top, every time.
- Tests: bidirectional pair collapses to one edge, router-mesh siblings
dropped, coordinator never receives an edge.
Auto-select
- After import, deselect existing canvas nodes and mark only the
freshly-imported ones as selected, so the user can drag the whole
subtree as a group.
The previous parser read `data.routes` which is just an echo of the
`routes` request flag (a boolean). On real brokers this caused
`TypeError: 'bool' object is not iterable` and 500s during /import.
- Rewrite parse_networkmap to read data.value.nodes + data.value.links
with fallback to data.{nodes,links} for legacy variants
- Defensive: drop links to unknown nodes, propagate lqi from link to
target node, extract model/vendor from definition block
- Bump networkmap timeout 10s -> 180s (large meshes are slow)
- Tests: rewrite fixture builders + sample payload to real Z2M shape;
add cases for legacy shape, routes:false echo (regression), malformed
list, link to unknown node, lqi propagation, definition extraction
- Update docs to mention 60s+ wait window
53 backend tests pass, mypy + ruff clean.
aiomqtt/paho exception strings can include the broker URI with embedded
credentials (mqtt://user:pass@host) or auth detail. The 502 response
from /import and the message field on /test-connection echoed these
verbatim via str(exc).
- Add _sanitize_mqtt_error() that maps known patterns (auth, refused,
DNS, TLS, timeout) to coarse, credential-free categories
- Original exception still logged at WARNING level for operator debug
- Drop hostname:port from TimeoutError messages
- /test-connection unexpected-error path no longer interpolates exc
Tests: 6 new (auth/refused/DNS/TLS/unknown sanitization, end-to-end
fetch_networkmap leak check).
- Schema: mqtt_tls + mqtt_tls_insecure flags on import + test-connection
requests; model_validator enforces insecure requires tls
- Service: _build_tls_context() using ssl.create_default_context();
logger.warning when verification disabled; tls_context plumbed into
aiomqtt.Client for both fetch_networkmap and test_mqtt_connection
- Route: passes tls flags through to service
- Frontend: TLS checkbox auto-toggles port 1883<->8883 unless user
edited; insecure checkbox disabled until TLS on, red-tinted; password
field marked autocomplete=new-password
- Tests: 7 new backend tests (TLS context build, client kwargs assertion,
router happy path, insecure-without-tls 422)
- maskIp handles IPv6 addresses (masks second and last group)
- maskIp handles comma-separated IP strings (masks each address)
- Add splitIps() helper to parse comma-separated IP field
- Add primaryIp() helper used by status checker (first IP wins)
- BaseNode renders each IP on its own line when comma-separated
- NodeModal placeholder shows comma-separated example
- Backend status_checker uses only first IP for connectivity checks
- Expand maskIp test suite: IPv6, comma-separated, splitIps, primaryIp
- Two-phase nmap: ARP sweep first to find all alive hosts (incl. IoT with
no open TCP ports), then port scan only alive hosts
- mDNS/Bonjour discovery via zeroconf for Shelly, ESPHome, HomeKit devices
- Add CoAP ports (5683, 5684, 4915) to port scan and IoT type hints
- Expand MAC OUI table with Shelly, Espressif, Sonoff, Tapo, Hue, IKEA, Tuya
- IoT vendor MAC takes precedence over generic HTTP port type hints
- Reorder suggest_node_type priority: iot now beats server
- POST /scan/{run_id}/stop endpoint signals running scan to cancel
- Scanner checks cancellation flag between CIDR ranges and hosts, exits early
- Cancelled scans get status 'cancelled' instead of 'done'
- Stop button (red StopCircle) shown in Scan History panel for running scans
- 6 new backend tests, 5 new frontend tests
The file was in /app/data/ which gets overwritten by the Docker volume
mount (backend_data:/app/data), making it invisible at runtime and
causing scan failures on fresh installs. Moved to /app/app/data/ so
it stays baked into the image.
Backend:
- Add MAC OUI lookup table (QEMU 52:54:00, Proxmox bc:24:11, VMware, VBox, Hyper-V)
- suggest_node_type() now accepts mac param and uses OUI as a low-priority hint
Frontend:
- Detect VM type from MAC prefix on pending device cards
- Show colored badge (QEMU / PVE / VMware / VBox / Hyper-V) in orange with tooltip
datetime.UTC was introduced in Python 3.11. Users on 3.10 get an ImportError.
Also lower ruff/mypy target-version from py313 to py310 to match minimum supported version.
- New 'camera' node type with Camera icon (frontend + types)
- Scanner fingerprint: camera ports (554, 8554, 37777, 34567, 2020) now suggest 'camera' instead of 'iot'
- service_signatures.json: all camera/NVR entries updated to suggested_node_type=camera
- 'camera' added to priority list in suggest_node_type (above iot)
Banner-specific signatures (e.g. AdGuard on port 3000) were incorrectly
matching when nmap returned no banner, causing wrong suggested_node_type
(e.g. 'router' for a media server). Now a signature with banner_regex is
only matched when a banner is present AND matches the regex.
Also make _PORT_TYPE_HINTS always contribute to suggest_node_type found set
(not just as elif fallback) so well-known ports like 8006 still resolve to
proxmox even when a generic sig also matched.
- Expand service_signatures.json from 35 → ~120 entries covering *arr apps,
smart-home (HA, MQTT, ESPHome), cameras (RTSP, Dahua, Tapo, Reolink),
network gear (MikroTik, UniFi, Pi-hole), auth (Authelia, Authentik, Vault),
monitoring (Grafana, InfluxDB, Loki, Uptime Kuma), containers (Portainer),
and many more home lab services
- Expand nmap port range to cover home lab ports (8989, 7878, 8123, 554, 1883, etc.)
and increase --host-timeout from 30s to 120s for reliable -sV detection
- Add _PORT_TYPE_HINTS fallback in suggest_node_type for ports without signatures
(cameras→iot, MQTT→iot, Proxmox→proxmox, MikroTik→router, etc.)
- Show actual port/protocol (e.g. TCP/9999) instead of "unknown_service" so
all open ports are visible even when not matched
- Update tests to reflect new unknown-port label format
Backend:
- asyncio.to_thread(_nmap_scan) — nmap no longer blocks the event loop
- Commit each discovered device immediately (previously one bulk commit at end)
- Update ScanRun.devices_found after each device so history panel shows live count
- broadcast_scan_update() pushes {type: scan_device_found} WS event per device
- Refactor broadcast_status to shared _broadcast() helper, adds type: "status" field
Frontend:
- useStatusPolling handles both WS message types (status / scan_device_found)
- canvasStore: scanEventTs + notifyScanDeviceFound() action
- PendingDevicesPanel auto-refreshes when WS scan event is received
- scanner.py: re-raise nmap exceptions so they reach ScanRun.error
(previously silently returned [] hiding the root cause)
- Sidebar: after triggering scan, auto-switch to History tab
- ScanHistoryPanel: auto-refresh every 3s while any run is 'running';
show error toast when a run transitions running→error; show spinner
on running runs; error message fully visible (no truncation)
- scripts/run_scan.py: standalone scan script to run with sudo for
nmap OS detection / SYN scans on macOS
Pouzor