We currently have approximately 50 low cost monitors installed and recording measurements in NSW, most of which are in Sydney and can be seen on the live data map. These monitors measure two sizes of particulate pollution (PM 2.5 and PM 10), as well as humidity and temperature. We have installed them as part of the global sensor.community network that was initiated in Stuttgart and now has more than 14,000 citizen science monitors around the world.
This post explains how to download the data files that contain all the measurements taken by our monitors (or any sensor.community monitor for that matter), so that you can share and analyse the results for your own research, projects or whatever you like!
The data from these monitors is free for you to access, republish, analyse, remix and so on, made available by sensor.community, under the Open Data Commons, Database Contents license.
Getting the raw data
The sensor.community network provides two ways to access the measurement data from sensors in the network, via API calls or CSV file downloads. In this post we will focus on the CSV (comma separated values) files option, because its more straight forward for most people.
The full list of our Community Environmental Monitoring sensors is available in a public spreadsheet. The spreadsheet lists the ‘sensorID’, the unique identifying number for sensor.community monitors, for each monitor we have installed. It also provides the direct links to the raw data CSV files and the charts dashboards.
The raw data CSV files are provided by Madavi.de, a data service set up by sensor.community, which has a CSV file with all the daily readings from a particular sensor. Sensor.community have also provided a dashboard of charts for each sensor, using a service called Grafana.
While our spreadsheet provides the links to the raw data CSV files and the charts, it is still useful to know how to construct these links/URLs using the ID number for any monitor on the map.
Let’s take a sensor in Haberfield, Sydney as an example to demonstrate how to access the data. This sensor’s sensorID is 13947371.
The structure of the URL to access the raw data CSV files is:
https://www.madavi.de/sensor/csvfiles.php?sensor=esp8266-[sensorID]so, for sensor 13947371, the URL is:
https://www.madavi.de/sensor/csvfiles.php?sensor=esp8266-13947371
This link returns a list of CSV files, one for every day that the sensor has been running. Some sensors encounter hardware issues or their wifi connection gets disrupted, and so you might find gaps in the expected list where measurements were not taken or recorded.
Download a file and open it in spreadsheet software like Calc (free), Excel or Google Sheets. From there, you can do all kinds of analysis.
The raw data in a given CSV file is shown below. As you can see, it is a little difficult to understand the salient information in this format. Our intention in the future is to condition this data to make it more easily digestible.
Time;durP1;ratioP1;P1;durP2;ratioP2;P2;SDS_P1;SDS_P2;PMS_P1;PMS_P2;Temp;Humidity;BMP_temperature;BMP_pressure;BME280_temperature;BME280_humidity;BME280_pressure;Samples;Min_cycle;Max_cycle;Signal;HPM_P1;HPM_P2
2023/02/19 00:02:04;;;;;;;13.05;2.13;;;25.20;99.90;;;;;;5054819;28;20614;-73
2023/02/19 00:04:38;;;;;;;13.45;2.33;;;25.20;99.90;;;;;;5057665;28;20193;-76
2023/02/19 00:07:09;;;;;;;12.63;2.00;;;25.30;99.90;;;;;;5074330;28;20209;-73
2023/02/19 00:09:40;;;;;;;8.63;1.58;;;25.30;99.90;;;;;;5061282;28;20211;-74
2023/02/19 00:12:11;;;;;;;14.25;2.00;;;25.20;99.90;;;;;;5069793;28;20562;-76
2023/02/19 00:14:42;;;;;;;10.80;2.00;;;25.30;99.90;;;;;;5071828;28;20189;-75
2023/02/19 00:17:13;;;;;;;9.35;2.00;;;25.40;99.90;;;;;;5072805;28;20667;-73
2023/02/19 00:19:46;;;;;;;11.35;1.80;;;25.40;99.90;;;;;;5064199;28;20251;-74
2023/02/19 00:22:18;;;;;;;8.57;1.73;;;25.40;99.90;;;;;;5057900;28;20218;-75
2023/02/19 00:24:49;;;;;;;12.90;2.05;;;25.50;99.90;;;;;;5053111;28;20198;-74
2023/02/19 00:27:20;;;;;;;10.07;2.03;;;25.60;99.90;;;;;;5054639;28;20189;-75
2023/02/19 00:29:52;;;;;;;8.60;1.95;;;25.70;99.90;;;;;;5052513;28;20197;-74
2023/02/19 00:32:23;;;;;;;11.63;2.10;;;25.70;99.90;;;;;;5061231;28;20537;-73
2023/02/19 00:34:58;;;;;;;12.80;1.80;;;25.70;99.90;;;;;;5058976;28;20199;-74As mentioned above, there are also dashboards with charts for each sensor. The URL for these dashboards is also available in the Google Sheets file, and has a similar format to the one to retrieve CSV files:
https://maps.sensor.community/grafana/d/GUaL5aZMz/pm-sensors?orgId=1&var-chipID=esp8266-[sensorID]So for the Haberfield example, sensor 13947371, the URL is:
https://maps.sensor.community/grafana/d/GUaL5aZMz/pm-sensors?orgId=1&var-chipID=esp8266-13947371
Future work
Having a method to download the data for any given sensor is a great start. Our next goals are to firstly make the data more accessible by conditioning the CSV files to contain only the relevant information (i.e. time and clearly labelled data readings). Once we have the data in a simpler format, we can work towards creating more flexible ways to query and download data over any location and time period, and explore more ways to make community generated environmental data useful and meaningful.
We would love to hear about how you go using the data and what you do with it. Please let us know via our Contact page.