After finally getting everything to work on the database side of things I wanted to explore dashboarding and being able to quickly pull up a visual showing what the current values are for the sensor data. After spending a few hours with Grafana and discovering that it doesn’t play nicely enough with JSON for my tastes(but pretty nicely with SQL type DBs…) I decided to just code up a pretty simple dashboard which shows the last four or so odd hours of sensor data. Its fairly simple using Chart.js and pulling the JSON data from ThingSpeak(last 6,000 entries @ 2s avg sampling = 200 min = 3 hours 20 min). I’m currently “pressure testing” the raspberry pi 4 I have and trying to ensure I have at least a week of uptime before I move to an ESP32 based solution which should let me get the rate of sampling up. The visual is below and I’ll explain the code afterwards:
Pi Environment Test Data
Code
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Dynamic Plot</title>
<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
<script src="https://cdn.jsdelivr.net/npm/chartjs-adapter-date-fns"></script>
<style>
table {
width: 100%;
border-collapse: collapse;
margin-top: 20px;
}
th, td {
border: 1px solid #ddd;
padding: 8px;
text-align: center;
}
th {
background-color: #f2f2f2;
}
</style>
</head>
<body>
<h2>Pi Environment Test Data</h2>
<button onclick="loadChart()">Load Chart</button>
<button onclick="refreshData()">Refresh</button>
<canvas id="myChart" width="400" height="200"></canvas>
<div id="latestValues"></div>
<script>
async function fetchData() {
const response = await fetch('https://api.thingspeak.com/channels/2545447/feeds.json?results=6000');
const data = await response.json();
return data.feeds;
}
function processData(feeds) {
const labels = feeds.map(feed => new Date(feed.created_at));
const tempC = feeds.map(feed => parseFloat(feed.field1));
const tempF = feeds.map(feed => parseFloat(feed.field2));
const humidity = feeds.map(feed => parseFloat(feed.field3));
return { labels, tempC, tempF, humidity };
}
function displayLatestValues(labels, tempC, tempF, humidity) {
const latestTime = labels[labels.length - 1];
const latestTempC = tempC[tempC.length - 1];
const latestTempF = tempF[tempF.length - 1];
const latestHumidity = humidity[humidity.length - 1];
const tableHTML = `
<table>
<tr>
<th>Time</th>
<th>Temperature C</th>
<th>Temperature F</th>
<th>Humidity</th>
</tr>
<tr>
<td>${latestTime.toLocaleDateString('en-US')} ${latestTime.toLocaleTimeString('en-US')}</td>
<td>${latestTempC.toFixed(2)}</td>
<td>${latestTempF.toFixed(2)}</td>
<td>${latestHumidity.toFixed(2)}</td>
</tr>
</table>
`;
document.getElementById('latestValues').innerHTML = tableHTML;
}
async function createChart() {
const feeds = await fetchData();
const { labels, tempC, tempF, humidity } = processData(feeds);
const ctx = document.getElementById('myChart').getContext('2d');
const myChart = new Chart(ctx, {
type: 'line',
data: {
labels: labels,
datasets: [
{
label: 'Temperature C',
data: tempC,
borderColor: 'rgba(255, 99, 132, 1)',
backgroundColor: 'rgba(255, 99, 132, 0.2)',
borderWidth: 1,
fill: true
},
{
label: 'Temperature F',
data: tempF,
borderColor: 'rgba(255, 165, 0, 1)',
backgroundColor: 'rgba(255, 165, 0, 0.2)',
borderWidth: 1,
fill: true
},
{
label: 'Humidity',
data: humidity,
borderColor: 'rgba(54, 162, 235, 1)',
backgroundColor: 'rgba(54, 162, 235, 0.2)',
borderWidth: 1,
fill: true
}
]
},
options: {
scales: {
x: {
type: 'time',
time: {
unit: 'hour'
}
},
y: {
beginAtZero: true
}
}
}
});
displayLatestValues(labels, tempC, tempF, humidity);
return myChart;
}
let chartInstance;
async function refreshData() {
if (chartInstance) {
chartInstance.destroy();
}
chartInstance = await createChart();
}
async function loadChart() {
if (!chartInstance) {
chartInstance = await createChart();
}
}
window.onload = async () => {
chartInstance = await createChart();
};
</script>
</body>
</html>Breaking down the code:
Code Summary
Fetch Data:
fetchData(): Fetches the last 6,000 entries from the ThingSpeak API. Theawaitkeyword ensures the function waits for the data to be fetched before moving on.
Process Data:
processData(feeds): Processes the raw data to extract timestamps, temperatures in Celsius and Fahrenheit, and humidity. It returns these as arrays.
Display Latest Values:
displayLatestValues(labels, tempC, tempF, humidity): Extracts the latest values from the arrays and formats them into an HTML table. ThetoLocaleDateStringandtoLocaleTimeStringmethods format the date and time in MM/DD/YYYY and 12-hour format respectively.
Create Chart:
createChart(): Fetches and processes the data, then uses Chart.js to create a line chart with three datasets: Temperature C, Temperature F, and Humidity. It configures the x-axis to display time and the y-axis to start at zero. It also callsdisplayLatestValuesto update the latest values table.
Refresh Data:
refreshData(): Destroys the existing chart instance and creates a new one with the latest data.
Window Onload:
window.onload: Ensures the chart is created when the page loads.
By breaking down the code into these sections, we can see how each part works together to fetch data, process it, display it in a chart, and show the latest values in a table. This simple dashboard provides a visual representation of sensor data over the last few hours, making it easier to monitor and analyze the data in real time.
Overall, this is a lot more lightweight than Grafana and given the fact that I’m not generating large amounts of traffic to Thingspeak this works very well as a simple logger/check-in for myself. I’m sure there’s a lot more convoluted ways of checking on the sensors, but this web interface allows me to quickly assess if I have downtime or the sensors are toast.
Additionally, I downloaded RaspController on iOS to quickly check on my sensors when at home. Overall, the ads were a bit intrusive and even reading from the simple DHT11 sensor was a bit goofy with the sensor readout being read then a few seconds later the readings disconnecting. I believe there’s a PRO version, but I doubt the functionality boost warrants the price as the web display is robust enough.
Regardless I plan on implementing an ESP32 based solution moving forward. Some comparisons are noted below based on what I’ve read online, but of course that can change once experience/pain sets in. I’m assuming that the 2s delay I’m experiencing will get worse as I add more sensors so I’m assuming the worst. I also plan on implementing the sensors outlined in my first raspberry pi post.
Overview of Sensors
PMS5003: Measures particulate matter (PM2.5 and PM10) with a typical update interval of 1 second.
MH-Z19: Measures CO2 levels with an update interval of every 2-3 seconds.
DHT22: Measures temperature and humidity with a sampling rate of 2 seconds.
BME680: Measures temperature, humidity, pressure, and VOC, with data output intervals ranging from 1 to 3 seconds.
MQ135: Measures air quality (VOC levels), requires analog-to-digital conversion and calibration for accurate readings.
1. Sampling Speed and Sensor Handling
Raspberry Pi:
Capabilities: Equipped with a multi-core CPU and up to 8GB RAM, allowing concurrent handling of multiple sensors.
PMS5003: Capable of continuous 1-second updates.
MH-Z19: Efficiently handles 2-3 second intervals.
DHT22: Manages 2-second intervals smoothly.
BME680: Processes complex outputs (temperature, humidity, pressure, VOC) every 1-3 seconds.
MQ135: Utilizes an ADC for continuous data reading.
Overall Sampling: Can read from all sensors simultaneously, processing and displaying data in real-time.
ESP32:
Capabilities: Dual-core processor designed for real-time processing, but might need optimization for simultaneous high-frequency data handling(C++).
PMS5003: Handles 1 Hz output effectively.
MH-Z19: Manages 2-3 second intervals efficiently.
DHT22: Handles 2-second intervals without issues.
BME680: Efficiently processes complex outputs within the 1-3 second range.
MQ135: Uses built-in ADC for continuous data but might have fewer channels than an external ADC.
Overall Sampling: Can handle the cumulative sampling rate with careful resource management.
2. Overhead from Operating System
Raspberry Pi:
OS: Runs a full Linux-based OS (e.g., Raspbian) with a graphical user interface, background processes, and system services.
Overhead: Significant but manageable, thanks to the Pi’s processing power.
Advantages: Multi-threading, extensive libraries, ability to run heavy applications (e.g., databases, servers).
ESP32:
OS: Operates with a minimal OS like FreeRTOS or can run bare-metal.
Overhead: Minimal, making it highly efficient for real-time applications.
Advantages: Direct hardware control, low latency, efficient resource use.
3. Speed of Data Transfer Wirelessly
Raspberry Pi:
Wi-Fi: Supports 2.4 GHz and 5 GHz bands, with data transfer rates up to several hundred Mbps.
Use Cases: Suitable for fast, reliable data transfer, streaming large datasets, frequent updates to a remote server.
Bluetooth: Available for short-range communication.
ESP32:
Wi-Fi: Primarily operates on the 2.4 GHz band, with data rates up to 150 Mbps.
Use Cases: Adequate for sensor data transmission, suitable for IoT applications.
Bluetooth: Supports Bluetooth (classic and BLE), useful for short-range data transfer and low-power communication.
4. Programming Language Choice
Raspberry Pi:
Languages: Supports Python, C, C++, JavaScript, Java, and more.
Development: Python is favored for its simplicity and extensive library support for hardware interaction.
Flexibility: Extensive libraries and frameworks for data processing, machine learning, web servers.
ESP32:
Languages: Commonly programmed using Arduino IDE (C/C++) or MicroPython.
Development: Arduino IDE provides robust sensor libraries; MicroPython offers ease of use and rapid prototyping.
Flexibility: C/C++ for performance and control; MicroPython for quick development and debugging.
5. Cost
Raspberry Pi:
Board Cost: $10 (Pi Zero) to $35-$75 (Pi 3 or Pi 4 models).
Additional Costs: Power supplies, SD cards, cases, peripherals, potentially exceeding $100.
Value Proposition: Higher cost but extensive capabilities, suitable for complex applications.
ESP32:
Board Cost: Typically $5-$10.
Additional Costs: Fewer peripherals needed, reducing overall system cost.
Value Proposition: Highly cost-effective for simple, cost-sensitive projects.
6. Power Consumption
Raspberry Pi:
Consumption: Higher due to its full-featured OS and higher processing power.
Power Supply: Requires a stable 5V power supply, typically 2.5A or more.
Suitability: Best for applications with reliable power sources.
ESP32:
Consumption: Low-power design with deep sleep modes and efficient power management.
Power Supply: Can operate on battery power for extended periods.
Suitability: Ideal for battery-powered or solar-powered IoT applications.
7. Complexity of Setup
Raspberry Pi:
Setup: Involves OS installation, Wi-Fi configuration, and additional software setup.
Ease of Use: Broad ecosystem and community support for troubleshooting.
Learning Curve: Higher due to OS complexity.
ESP32:
Setup: Simpler, involving firmware flashing and code development.
Ease of Use: Growing community support, good documentation.
Learning Curve: Lower, especially with Arduino IDE or MicroPython.
8. Integration with Databases
Raspberry Pi:
Local Databases: Can run MySQL, PostgreSQL, SQLite for local storage and complex queries.
Remote Databases: Interfaces with cloud databases via APIs.
Use Cases: Suitable for extensive data storage, processing, and local analytics.
ESP32:
Remote Databases: Interfaces with databases through HTTP/HTTPS, MQTT, or APIs.
Local Storage: Limited, suitable for buffering data before transmission.
Use Cases: Best for periodic data transmission to a central server.
9. Scalability and Expandability
Raspberry Pi:
Scalability: High, with multiple USB ports, GPIO pins, and support for I2C, SPI, UART.
Expandability: Can connect multiple sensors, peripherals, and expansion boards (HATs).
Use Cases: Ideal for larger, complex projects needing scalability.
ESP32:
Scalability: Moderate, fewer GPIO pins but sufficient for many IoT projects.
Expandability: Supports I2C, SPI, UART; multiple sensors can be connected.
Use Cases: Suitable for compact, efficient IoT solutions.
10. Community and Support
Raspberry Pi:
Community: Extensive, with many tutorials, forums, and resources.
Support: Strong, especially for educational, hobbyist, and professional uses.
Documentation: Comprehensive, with official support from the Raspberry Pi Foundation.
ESP32:
Community: Growing, with many tutorials and forums.
Support: Adequate, focused on embedded systems and IoT.
Documentation: Good, provided by Espressif and third-party contributors.
11. Real-Time Operating Capabilities
Raspberry Pi:
RTOS: Not typically used, though real-time kernels (PREEMPT-RT) are available for specific applications.
Suitability: Best for applications where real-time performance is not critical.
ESP32:
RTOS: FreeRTOS support, ideal for real-time applications.
Suitability: Designed for real-time processing, making it suitable for time-sensitive tasks.
12. Security
Raspberry Pi:
Security Features: Depends on OS and software; can use advanced security protocols and encryption.
Use Cases: Suitable for applications requiring robust security measures.
ESP32:
Security Features: Built-in hardware security features (e.g., secure boot, flash encryption).
Use Cases: Adequate for secure IoT applications.
Summary
Sampling Speed: Both the Raspberry Pi and ESP32 can handle the sensors’ sampling rates, with the Pi having more processing power for higher-frequency data collection.
Overhead from OS: The Raspberry Pi has more overhead due to its full OS, while the ESP32 operates with minimal overhead, ideal for real-time applications.
Wireless Data Transfer: The Raspberry Pi achieves higher data transfer speeds, but the ESP32’s capabilities are sufficient for sensor data transmission.
Programming Language: The Raspberry Pi offers more flexibility, while the ESP32 focuses on C/C++ and MicroPython.
Cost: The ESP32 is significantly cheaper, making it attractive for cost-sensitive projects.
Power Consumption: The ESP32 is more power-efficient, suitable for battery-powered applications.
Complexity of Setup: The Raspberry Pi setup is more complex but has broader support, while the ESP32 setup is simpler and more focused.
Integration with Databases: The Raspberry Pi can run full database servers locally, while the ESP32 typically relies on remote databases.
Scalability and Expandability: The Raspberry Pi is more scalable and expandable, suitable for larger projects.
Community and Support: The Raspberry Pi has a larger community and more extensive support.
Real-Time Operating Capabilities: The ESP32 is better suited for real-time applications with its RTOS support.
Security: Both offer security features, but the ESP32 has built-in hardware security for IoT applications.
In conclusion, the Raspberry Pi offers more computational power and flexibility, making it suitable for complex applications requiring robust data processing and local storage. The ESP32, with its low cost, low power consumption, and efficient real-time processing capabilities, is ideal for simpler, cost-effective, and portable sensor applications. Additionally, the cost of failure for a raspberry pi is quite large while an ESP32 is negligible. I would like to have a real time setup along with a dynamic plot of temp, humidity, etc. for a plant monitoring project. To do this at scale(someone has 50+ plants….) I need to dramatically reduce cost. Additionally, I need to also begin budgeting for a Pi cluster project to learn distributed/parallel computing, which, funny enough, are two classes in my Master’s at UIUC that I would like to take(CS425 and CS484 respectively). Truth be told much of this IoT Raspberry Pi/Arduino type work recently stems from a desire to take the IoT class at UIUC, but that class’ priority relative to other courses such as Machine Learning, Statistical Learning, etc. is dead last and puts me over the credits I need. Additionally, implementing X/Y/Z with this hardware allows me to learn how things work at a fraction of tuition(1/10).
Ideally the cluster would serve as a way of actually learning parallel computing and also establishing some local web services to make my life easier. The tentative list of local web service projects include:
Plant Monitoring Dashboard
Project: A centralized dashboard to monitor temp, humidity, soil moisture, and other environmental factors for plants of concern.
Tech: I plan on using Flask/Django for the backend, React for the frontend, and a database like PostgreSQL for the DB.
Benefits: I also plan on implementing some sort of algorithm to determine when there’s a problem and sending push notifications or texts to the user.
Personal Cloud Storage
- Self hosted cloud storage solution using something like NextCloud to store and share files securely. I might keep this guy local though as the only thing I plan on exposing to the World Wide Web would be a website which uses Cloudflare for security.
Local Development Server(Local Git)
Project: Development environment for testing and deploying web applications and other software projects.
Tech: I’d likely use Docker for containerized environments, Jenkins for CI/CD pipelines, and Git for version control. I would also deeply expand my knowledge of what’s what with this tech.
Benefits: This should actually help standardize and streamline my development workflow, let me perform automated testing/deployment, and provide a stable/cheap platform for experimentation with new tech.
Media Server(Mostly PS4/Samsung TV)
Project: Local media server to host/stream media content like movies/music/photos
Tech: Plex or Jellyfin for media management and streaming.
Benefits: Centralized media library with near seamless access to my favorite(repeatedly watched) shows.
IoT Device Management
Project: Platform to manage/monitor/update IoT devices deployed throughout the house. Likely temp/humidity/air quality sensors and plant sensors. Possible intrusion detection sensors if I decide to revisit some of my previous projects.
Technology: Mosquitto for MQTT communication, InfluxDB for time-series data, and Grafana(if I’m not using JSON’s…) for visualization.
Benefits: Centralized control and visualization of all IoT devices, real time monitoring, and simplified firmware updates. I will have to write quite a bit of C++ code now that I think about it to ensure the ESP32 works and works fast.
Network Monitoring Tool
Project: Basic system to monitor the health/performance of my network.
Tech: Nagios or Zabbix.
Benefits: Real time alerts on network issues, detailed performance metrics, and insights into network traffic patterns.
PiHole
Project: Pi hold server to provide network wide ad blocking, improve browsing speed, and improve privacy by blocking unwanted ads/trackers. It can also act as a local DNS resolver boosting network performance.
Tech: Raspberry Pi for running the software.
Benefits: Ad blocking, boost privacy, reduce data usage, reduce DNS query times and overall help speed up my network.
HPC Pi
Project: Build a scalable Raspberry Pi cluster(at least 3) to learn and implement parallel computing, distributed computing, and high performance computing concepts.
Tech: Multiple raspberry pis, networking equipment, cluster management software, and parallel computing libraries(OpenMP/MPI). Plus Kubernetes.
Benefits: Hands on experience with distributed systems and parallel processing, cost effective way of exploring high performance computing, and gaining marketable skills in cluster setup, management, and scalability.
Tentatively something like this tutorialwill be followed and result in a three pi setup vs the 8 pi setup below(from the tutorial):
Hopefully I have some time before the summer ends and classes begin to experiment. Fingers crossed and my willpower willing, I will take the distributed systems course this fall alongside an intensive statistical learning course!