Collecting data safely

Practical guidance on ethics, consent and storage when running measurements.

Ethical basics

Consider whether your probes could be mistaken for malicious scanning — throttle request rates, identify your measurement endpoint (if applicable), and follow local regulations. Provide clear contact information if you operate public probes and consider a page describing your measurement policy so network operators can quickly understand and contact you.

Consent and transparency

Where measurements involve volunteers or user data, obtain informed consent and describe what is measured and why. For network-level measurements that touch third-party infrastructure, be conservative and prefer passive, non-invasive collection where possible.

Data minimisation and pseudonymisation

Keep only the fields you need. Avoid storing raw payloads and consider hashing or truncating IP addresses when a full address is not required. Keep a data dictionary that documents fields, formats, and any transformations applied.

Logging, storage and access control

Store logs securely with strict access controls. Rotate keys, encrypt data at rest, and log administrative access. Maintain a retention policy and automate deletion where possible.

Rate limiting and abuse prevention

Implement client-side rate limiting and exponential backoff for retries. Monitor outbound probing for anomalies and set conservative defaults to avoid accidentally generating high traffic or appearing as a scanner.