Skip to content
GitLab
Commit 6c7b465b authored by Kaif Siddique's avatar Kaif Siddique
Browse files

Manual Push to Gitlab

parent 594c4396
Hide whitespace changes
Inline Side-by-side
.gitignore 0 → 100644
View file @ 6c7b465b
*.log
/artifacts
/data/raw/xls*
/data/processed
/data/raw/combined_*
/data
/venv
\ No newline at end of file
INFO_TO_RUN.md 0 → 100644
View file @ 6c7b465b
# Occupancy + Energy Optimization — Starter Kit
This is a minimal, hackathon-ready scaffold for detecting room occupancy from sensors, aggregating to 2-hour Levels, and producing a simple heating schedule.
## Structure
```
HVAC-ENERGY-OPTIMIZATION-WITH-OCCUPANCY-ANALYTICS/
data/raw/ # put your CSV here (or set config.yaml path)
pipeline/
ingest.py
features.py
train.py
predict.py
aggregate_levels.py
schedule.py
api/app.py # Flask API
dash/app.py # Dash dashboard (reads artifacts by default)
artifacts/ # pipeline outputs (parquet/json/pkl)
config.yaml
README.md
```
## Quickstart
1. **Python env**
```bash
python -m venv .venv && source .venv/bin/activate
pip install -r updated_requirements.txt # or: pandas numpy scikit-learn flask dash plotly pyarrow joblib pyyaml
```
2. **Configure data path** (in `config.yaml`):
- Current path: `/data/final_data_Knauth.csv`
3. **Run the pipeline**
You can run the whole pipeline with the provided PowerShell script:
```powershell
./run_pipeline.ps1
```
Or run each step individually:
```bash
# 1a. Data ingestion for training - processes raw CSV into base_train.parquet
python pipeline/ingest.py --phase train --debug
# 1b. Data ingestion for prediction - processes raw CSV into base_predict.parquet
python pipeline/ingest.py --phase predict --debug
# 2a. Feature engineering for training data
python pipeline/features.py --phase train
# 2b. Feature engineering for prediction data
python pipeline/features.py --phase predict
# 3. Model training - trains occupancy detection model using training data
python pipeline/train.py
# 4. Making predictions - applies model to prediction data
python pipeline/predict.py
# 5. Aggregating to 2h blocks - creates occupancy levels
python pipeline/aggregate_levels.py
# 6. Creating schedule - generates basic heating schedule
python pipeline/schedule.py
# 7. Optimizing energy - enhances schedule for efficiency
python pipeline/optimize_energy.py
# 8. Evaluation - measures system performance
python pipeline/evaluate.py
# 9. Dashboard - visualizes results
python dash/app.py
```
4. **Run the API**
```bash
python api/app.py
# then open http://localhost:8000/health
```
5. **Run the Dashboard**
```bash
python dash/app.py
# then open http://localhost:8050/
```
## Artifacts produced
- `artifacts/base_train.parquet` – cleaned training data
- `artifacts/base_predict.parquet` – cleaned prediction data
- `artifacts/feature_store_train.parquet` – engineered features for training
- `artifacts/feature_store_predict.parquet` – engineered features for prediction
- `artifacts/model.pkl` – trained sklearn model
- `artifacts/metrics.json` – evaluation metrics
- `artifacts/preds.parquet` – per-sample predictions + probabilities
- `artifacts/levels_2h.parquet` – 2-hour aggregated Levels (0/1/2)
- `artifacts/schedule_week.json` – heuristic heating plan for one ISO week
## Notes
- If warm-up estimation is not possible from the current data, the scheduler uses a **fixed preheat** (configurable).
- The Dash app reads artifacts directly; you can switch it to the API later.
- The pipeline now supports using different datasets for training and prediction phases.
- Configure the training and prediction datasets in `config.yaml` using `train_csv` and `predict_csv` parameters.
- By default, the pipeline uses `./data/raw/final_data_Knauth.csv` for training and `./data/processed/calculated_occupancy.csv` for prediction.
SENSOR_CONVERSION_README.md 0 → 100644
View file @ 6c7b465b
# Sensor Data Conversion Script
This script converts multiple sensor data CSV files from both `data/raw/xls` and `data/raw/xls_predict` directories into separate combined CSV files that are compatible with the ingest.py pipeline.
## What this script does:
1. Reads all CSV files from:
- `data/raw/xls` directory (for training data)
- `data/raw/xls_predict` directory (for prediction data)
2. Extracts only the essential columns:
- Server time (converted to datetime)
- CO2 concentration
- Temperature
- Humidity
3. Adds a device_id column from the filename (ESP...)
4. Processes files in parallel using multithreading for better performance
5. Saves the combined data as:
- `data/raw/combined_sensor_data_train.csv` (for training)
- `data/raw/combined_sensor_data_predict.csv` (for prediction)
## How to use this script:
### Step 1: Place your sensor CSV files in the right directories
Make sure your CSV files are in the proper directories:
- Training data: `data/raw/xls` directory
- Prediction data: `data/raw/xls_predict` directory
The script expects files with names like `CO2sensors_ESP6cef78.csv`.
### Step 2: Run the script
You have several options for running the script:
```bash
# Process both training and prediction data with default settings (4 threads)
python convert_sensors_to_csv.py
# Process only training data
python convert_sensors_to_csv.py --train
# Process only prediction data
python convert_sensors_to_csv.py --predict
# Adjust the number of threads for processing (e.g., 8 threads)
python convert_sensors_to_csv.py --threads 8
# Combine options
python convert_sensors_to_csv.py --predict --threads 6
```
### Step 3: Update the config.yaml
Update your config.yaml to point to the new combined datasets:
```yaml
data:
train_csv: "./data/raw/combined_sensor_data_train.csv"
predict_csv: "./data/raw/combined_sensor_data_predict.csv"
raw_csv: "./data/raw/combined_sensor_data_train.csv"
```
If you only have one type of data or want to use the same data for both training and prediction:
```yaml
# For training data only:
data:
train_csv: "./data/raw/combined_sensor_data_train.csv"
predict_csv: "./data/raw/final_data_Knauth.csv"
raw_csv: "./data/raw/combined_sensor_data_train.csv"
# OR for prediction data only:
data:
train_csv: "./data/raw/final_data_Knauth.csv"
predict_csv: "./data/raw/combined_sensor_data_predict.csv"
raw_csv: "./data/raw/final_data_Knauth.csv"
```
### Step 4: Run the pipeline
Now you can run the pipeline as usual, starting with:
```bash
python pipeline/ingest.py --phase train --debug
```
## Expected output format:
The script generates a CSV file with only the following essential columns:
- datetime (from Server time)
- co2 (CO2 concentration)
- temperature
- humidity
- device_id (extracted from filename)
## Notes:
- Device IDs are extracted from filenames and converted to numeric values
- The script only extracts the essential columns as requested
- No derived columns or calculations are performed
- Multithreaded processing significantly improves performance on large datasets
- You can adjust the number of threads with the `--threads` parameter to match your system's capabilities
\ No newline at end of file
api/app.py 0 → 100644
View file @ 6c7b465b
#!/usr/bin/env python3
from flask import Flask, jsonify, request
from flask_restx import Api, Resource, fields, Namespace
from pathlib import Path
import pandas as pd
import json
import yaml
import logging
# Setup logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
# Create Flask app and API
app = Flask(__name__)
api = Api(app,
version='1.0',
title='Campus Energy Optimization API',
description='API for retrieving room occupancy and scheduling data',
doc='/docs'
)
# Load configuration
with open("config.yaml", "r") as f:
CFG = yaml.safe_load(f)
ART = Path(CFG["data"]["artifacts_dir"])
logger.info(f"Using artifacts directory: {ART}")
# Create namespaces for API organization
ns_health = api.namespace('health', description='Health check endpoint')
ns_rooms = api.namespace('rooms', description='Room information endpoints')
# Define models for documentation
level_model = api.model('Level', {
'room_id': fields.String(description='Room identifier'),
'block_start': fields.String(description='Start time of the block (ISO format)'),
'level': fields.Integer(description='Occupancy level (0-2)', enum=[0, 1, 2]),
'iso_week': fields.String(description='ISO week identifier (e.g., 2025-W03)')
})
schedule_entry_model = api.model('ScheduleEntry', {
'start': fields.String(description='Start time (ISO format)'),
'end': fields.String(description='End time (ISO format)'),
'setpoint': fields.Float(description='Temperature setpoint in Celsius'),
'probability': fields.Float(description='Occupancy probability', required=False)
})
schedule_model = api.model('Schedule', {
'week': fields.String(description='ISO week identifier'),
'room_id': fields.String(description='Room identifier'),
'entries': fields.List(fields.Nested(schedule_entry_model))
})
# Health check endpoint
@ns_health.route('/')
class Health(Resource):
@ns_health.doc('get_health_status')
@ns_health.response(200, 'System is healthy')
def get(self):
"""Check system health"""
return {"status": "ok"}
# Rooms list endpoint
@ns_rooms.route('/')
class RoomsList(Resource):
@ns_rooms.doc('get_rooms')
@ns_rooms.response(200, 'List of available rooms')
def get(self):
"""Get list of available rooms"""
logger.info("Getting list of available rooms")
pth = ART / "levels_2h.parquet"
if not pth.exists():
logger.warning(f"Levels file not found: {pth}")
return []
df = pd.read_parquet(pth)
rooms = sorted(df["room_id"].dropna().astype(str).unique().tolist())
logger.info(f"Found {len(rooms)} unique rooms")
return rooms
# Room levels endpoint
@ns_rooms.route('/<room_id>/levels')
@ns_rooms.param('room_id', 'Room identifier')
class RoomLevels(Resource):
@ns_rooms.doc('get_room_levels')
@ns_rooms.param('week', 'ISO week identifier (optional, e.g., 2025-W03)', _in='query')
@ns_rooms.response(200, 'List of occupancy levels', [level_model])
def get(self, room_id):
"""Get occupancy levels for a specific room"""
logger.info(f"Getting occupancy levels for room {room_id}")
week = request.args.get("week")
if week:
logger.info(f"Filtering by week: {week}")
try:
df = pd.read_parquet(ART / "levels_2h.parquet")
df["block_start"] = pd.to_datetime(df["block_start"], utc=True)
if week:
iso = df["block_start"].dt.isocalendar()[["year", "week"]]
df["iso_week"] = iso["year"].astype(int).astype(str) + "-W" + iso["week"].astype(int).astype(str).str.zfill(2)
df = df[df["iso_week"] == week]
out = df[df["room_id"].astype(str) == str(room_id)].copy()
out["block_start"] = out["block_start"].astype(str)
logger.info(f"Found {len(out)} level records for room {room_id}")
return out.to_dict(orient="records")
except Exception as e:
logger.error(f"Error retrieving levels for room {room_id}: {str(e)}")
api.abort(500, f"Error retrieving data: {str(e)}")
# Room schedule endpoint
@ns_rooms.route('/<room_id>/schedule')
@ns_rooms.param('room_id', 'Room identifier')
class RoomSchedule(Resource):
@ns_rooms.doc('get_room_schedule')
@ns_rooms.response(200, 'Room heating schedule', schedule_model)
@ns_rooms.response(404, 'Schedule not found')
def get(self, room_id):
"""Get heating schedule for a specific room"""
logger.info(f"Getting heating schedule for room {room_id}")
p = ART / "schedule_week.json"
if not p.exists():
logger.warning(f"Schedule file not found: {p}")
return {"error": "schedule not found"}, 404
with open(p, "r") as f:
data = json.load(f)
entries = data.get("entries", {}).get(str(room_id), [])
logger.info(f"Found {len(entries)} schedule entries for room {room_id}")
return {
"week": data.get("week"),
"room_id": str(room_id),
"entries": entries
}
# Add optimized schedule endpoint
@ns_rooms.route('/<room_id>/optimized-schedule')
@ns_rooms.param('room_id', 'Room identifier')
class OptimizedSchedule(Resource):
@ns_rooms.doc('get_optimized_schedule')
@ns_rooms.response(200, 'Optimized heating schedule', schedule_model)
@ns_rooms.response(404, 'Optimized schedule not found')
def get(self, room_id):
"""Get energy-optimized heating schedule for a specific room"""
logger.info(f"Getting optimized heating schedule for room {room_id}")
p = ART / "optimized_heating_plan.json"
if not p.exists():
logger.warning(f"Optimized schedule file not found: {p}")
return {"error": "optimized schedule not found"}, 404
with open(p, "r") as f:
data = json.load(f)
entries = data.get(str(room_id), [])
logger.info(f"Found {len(entries)} optimized schedule entries for room {room_id}")
# Get week from regular schedule for reference
week = "unknown"
schedule_path = ART / "schedule_week.json"
if schedule_path.exists():
with open(schedule_path, "r") as f:
week_data = json.load(f)
week = week_data.get("week", "unknown")
return {
"week": week,
"room_id": str(room_id),
"entries": entries
}
if __name__ == "__main__":
host = CFG["server"]["host"]
port = CFG["server"].get("api_port", 8000)
logger.info(f"Starting API server on {host}:{port}")
app.run(host=host, port=port, debug=True)
\ No newline at end of file
config.yaml 0 → 100644
View file @ 6c7b465b
data:
# Updated paths for training and prediction data
train_csv: "./data/processed/combined_sensor_data_train.csv"
predict_csv: "./data/processed/combined_sensor_data_predict.csv"
# Keep raw_csv for backward compatibility (pointing to training data)
raw_csv: "./data/processed/combined_sensor_data_train.csv"
artifacts_dir: "./artifacts"
room_id_column: "device_id" # Critical change
timestamp_columns: ["datetime"]
date_column: null # Not needed with our simplified format
time_column: null # Not needed with our simplified format
label_column: occupied # Removed occupied column in our simplified format
# Updated to match only the sensor columns we're keeping
sensors: ["co2", "temperature", "humidity"]
features:
rolling_minutes: [15, 30]
infer_windows: true
model:
type: "logreg_ovr"
test_fraction_time: 0.2
random_state: 42
class_weight: "balanced"
levels:
block_minutes: 120
rules:
max_label1_minutes_level0: 15
max_label2_minutes_level0: 5
min_label2_minutes_level2: 15
min_label1_minutes_for_mix: 10
min_label2_minutes_for_mix: 5
schedule:
probability_threshold: 0.4
comfort_setpoint: 21.0
setback_setpoint: 15.0
preheat_minutes_default: 30
target_week: "2025-W04"
server:
host: "127.0.0.1"
dash_port: 8050
api_port: 8000
convert_sensors_to_csv.py 0 → 100644
View file @ 6c7b465b
#!/usr/bin/env python3
import os
import re
import pandas as pd
import numpy as np
from pathlib import Path
from datetime import datetime
import concurrent.futures
import argparse
import time
def extract_device_id(filename):
"""Extract the ESP ID from the filename."""
match = re.search(r'ESP([a-f0-9]+)', filename)
if match:
device_id = match.group(0) # Return the full ESP identifier
# Convert device_id to a numeric value for better compatibility
# Use last 6 chars of the ESP ID converted to an integer
numeric_id = int(device_id[-6:], 16) % 10000 # Keep it to a reasonable size
return numeric_id
return 9999 # Default if not found
def find_directory(dir_name):
"""Find a directory based on its name, checking multiple possible locations."""
script_dir = Path(__file__).parent.absolute()
possible_paths = [
Path(f"data/raw/{dir_name}"), # If running from project root
script_dir / f"data/raw/{dir_name}", # If script is in project root
Path(f"../data/raw/{dir_name}"), # If running from a subdirectory
]
# Find the first path that exists
for p in possible_paths:
if p.exists():
return p
return None
def process_directory(dir_name, output_suffix, num_threads=4):
"""Process all CSV files in the specified directory and create a combined CSV file."""
base_dir = find_directory(dir_name)
if base_dir is None:
print(f"Could not find the {dir_name} directory. Skipping.")
print("Checked the following paths:")
script_dir = Path(__file__).parent.absolute()
possible_paths = [
Path(f"data/raw/{dir_name}"), # If running from project root
script_dir / f"data/raw/{dir_name}", # If script is in project root
Path(f"../data/raw/{dir_name}"), # If running from a subdirectory
]
for p in possible_paths:
print(f" - {p.absolute()} (exists: {p.exists()})")
return False
# Determine output directory and file
output_dir = base_dir.parent
output_file = output_dir / f"combined_sensor_data_{output_suffix}.csv"
print(f"\nProcessing {dir_name} directory:")
print(f"Input directory: {base_dir.absolute()}")
print(f"Output file will be: {output_file.absolute()}")
# Get all CSV files in the directory
csv_files = list(base_dir.glob("*.csv"))
print(f"Found {len(csv_files)} CSV files")
if not csv_files:
print(f"No CSV files found in {base_dir}. Skipping.")
return False
# Process the files and generate the combined CSV
return process_files(csv_files, output_file, num_threads)
def process_single_file(file_info):
"""Process a single CSV file and return a DataFrame."""
file_path, device_id = file_info
try:
# Read the CSV file with semicolon delimiter
df = pd.read_csv(file_path, delimiter=';', skiprows=2) # Skip header rows
# Rename columns for the essential fields only
df.columns = ['datetime', 'sensor_time', 'co2', 'temperature', 'humidity']
# Add device_id column for tracking data source
df['device_id'] = device_id
# Ensure datetime is in proper format
df['datetime'] = pd.to_datetime(df['datetime'])
# Keep only the essential columns
df = df[['datetime', 'co2', 'temperature', 'humidity', 'device_id']]
return df, True, device_id
except Exception as e:
print(f"Error processing {file_path.name}: {e}")
return pd.DataFrame(), False, device_id
def process_files(csv_files, output_file, num_threads=4):
"""Process a list of CSV files and combine them into a single output file using multithreading."""
# Prepare file info list
file_info_list = []
# Prepare for file processing
for file_path in csv_files:
device_id = extract_device_id(file_path.name)
file_info_list.append((file_path, device_id))
print(f"Processing {len(csv_files)} files using {num_threads} threads...")
# Process files in parallel
all_data = []
with concurrent.futures.ThreadPoolExecutor(max_workers=num_threads) as executor:
future_to_file = {executor.submit(process_single_file, info): info for info in file_info_list}
for i, future in enumerate(concurrent.futures.as_completed(future_to_file)):
file_path, device_id = future_to_file[future]
try:
df, success, device_id = future.result()
if success:
all_data.append(df)
print(f"Processed {file_path.name} (Device ID: {device_id}) - File {i+1}/{len(csv_files)}")
except Exception as e:
print(f"Error processing {file_path.name}: {e}")
if not all_data:
print("No data to combine!")
return False
# Combine all dataframes
combined_df = pd.concat(all_data, ignore_index=True)
# Sort by datetime
combined_df = combined_df.sort_values('datetime')
# Save the combined data
output_file.parent.mkdir(parents=True, exist_ok=True)
combined_df.to_csv(output_file, index=False)
print(f"\nSuccessfully combined {len(csv_files)} files into {output_file}")
print(f"Total records: {len(combined_df)}")
print(f"Date range: {combined_df['datetime'].min()} to {combined_df['datetime'].max()}")
# Show the distribution of records by device
device_counts = combined_df['device_id'].value_counts()
print("\nRecords per device:")
for device, count in device_counts.items():
print(f" Device {device}: {count} records")
# Show some statistics about the data
print(f"\nAverage CO2: {combined_df['co2'].mean():.2f} ppm")
print(f"Average temperature: {combined_df['temperature'].mean():.2f}°C")
print(f"Average humidity: {combined_df['humidity'].mean():.2f}%")
return True
def parse_arguments():
"""Parse command line arguments."""
parser = argparse.ArgumentParser(description='Convert sensor CSV files to a combined format.')
parser.add_argument('--train', action='store_true', help='Process training data (xls directory)')
parser.add_argument('--predict', action='store_true', help='Process prediction data (xls_predict directory)')
parser.add_argument('--threads', type=int, default=4, help='Number of threads to use for processing')
return parser.parse_args()
def convert_files_to_csv():
"""Convert CSV files from both xls and xls_predict directories."""
args = parse_arguments()
# If no specific flags are provided, process both by default
if not args.train and not args.predict:
args.train = True
args.predict = True
start_time = time.time()
train_processed = False
predict_processed = False
# Process based on arguments
if args.train:
print(f"\nProcessing training data using {args.threads} threads...")
train_processed = process_directory("xls", "train", args.threads)
if args.predict:
print(f"\nProcessing prediction data using {args.threads} threads...")
predict_processed = process_directory("xls_predict", "predict", args.threads)
end_time = time.time()
if not train_processed and not predict_processed:
print("\nNo data processed. Please check the directory structure.")
return
print(f"\nProcessing complete! Total time: {end_time - start_time:.2f} seconds")
if train_processed and predict_processed:
print("\nYou can now update config.yaml to use these files:")
print("data:")
print(" train_csv: \"./data/raw/combined_sensor_data_train.csv\"")
print(" predict_csv: \"./data/raw/combined_sensor_data_predict.csv\"")
print(" raw_csv: \"./data/raw/combined_sensor_data_train.csv\"")
elif train_processed:
print("\nOnly training data was processed. Update config.yaml:")
print("data:")
print(" train_csv: \"./data/raw/combined_sensor_data_train.csv\"")
elif predict_processed:
print("\nOnly prediction data was processed. Update config.yaml:")
print("data:")
print(" predict_csv: \"./data/raw/combined_sensor_data_predict.csv\"")
if __name__ == "__main__":
convert_files_to_csv()
\ No newline at end of file
dash/README.md 0 → 100644
View file @ 6c7b465b
# Campus Energy Optimization Dashboard
This dashboard provides an interactive interface to view occupancy levels and heating schedules for campus buildings.
## Getting Started
To run the dashboard:
```bash
python dash/app.py
```
The dashboard will be available at http://localhost:8050/ (or whatever host/port is specified in your config.yaml).
## Features
### Home Page (Occupancy & Scheduling)
- View occupancy levels by room and week
- See weekly occupancy patterns
- Review proposed heating schedules with energy savings
### Model Evaluation Page
- Compare predicted vs. actual occupancy
- View confusion matrix and performance metrics
- Analyze prediction errors by time of day
- Track prediction accuracy trends
## Troubleshooting
- If you see "No evaluation data available", it means no predictions or labeled data are available.
- If you see "No data for Room X", it means no occupancy data exists for that room.
- For other issues, check the console output for error messages.
## Data Requirements
The dashboard expects these data files in your artifacts directory:
- `preds.parquet` - Model predictions
- `feature_store.parquet` - Features with actual occupancy labels
- `levels_2h.parquet` - Processed occupancy levels
- `enhanced_schedule.json` - Optimized heating schedule
\ No newline at end of file
dash/app.py 0 → 100644
View file @ 6c7b465b
#!/usr/bin/env python3
import yaml
from pathlib import Path
from dash import Dash, page_container
import dash_bootstrap_components as dbc
# ---- Load config
with open("config.yaml", "r", encoding="utf-8") as f:
CFG = yaml.safe_load(f)
# Register pages directory
app = Dash(
__name__,
use_pages=True,
external_stylesheets=[dbc.themes.BOOTSTRAP]
)
app.title = "Occupancy & Heating Plan"
app.layout = dbc.Container([
dbc.NavbarSimple(
children=[
dbc.NavItem(dbc.NavLink("Occupancy & Scheduling", href="/")),
dbc.NavItem(dbc.NavLink("Model Evaluation", href="/evaluation")),
],
brand="Campus Energy Optimization",
brand_href="/",
color="primary",
dark=True,
),
page_container
], fluid=True)
if __name__ == "__main__":
# Get host and port from config
host = CFG["server"]["host"]
port = CFG["server"].get("dash_port", 8050)
# Print a clear message showing the correct URL
print(f"\n✅ Dashboard is running!")
print(f"📊 Open your browser and navigate to: http://{host if host != '0.0.0.0' else '127.0.0.1'}:{port}/")
print(f"⚠️ Make sure to use HTTP, not HTTPS\n")
# Run the app
app.run(host=host, port=port, debug=True)
\ No newline at end of file
dash/assets/style.css 0 → 100644
View file @ 6c7b465b
/* New styles for evaluation page */
.evaluation-container {
padding: 20px;
max-width: 1400px;
margin: 0 auto;
}
.filter-container {
display: grid;
grid-template-columns: 1fr 2fr 120px;
gap: 20px;
margin-bottom: 25px;
align-items: end;
}
.filter-item {
display: flex;
flex-direction: column;
}
.metrics-row {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 20px;
margin: 20px 0;
max-height: 450px; /* Add max height constraint */
overflow: hidden; /* Prevent overflow */
}
.chart-container {
background: white;
border-radius: 5px;
padding: 15px;
box-shadow: 0 1px 3px rgba(0,0,0,0.1);
margin-bottom: 25px;
}
.chart-half {
background: white;
border-radius: 5px;
padding: 15px;
box-shadow: 0 1px 3px rgba(0,0,0,0.1);
height: 100%; /* Make sure it fills the container height */
display: flex; /* Use flexbox */
flex-direction: column;
}
/* Make the graph container fit the available space */
.chart-half .js-plotly-plot {
flex-grow: 1;
height: auto !important;
}
.metrics-container {
padding: 20px;
background: #f8f9fa;
border-radius: 5px;
height: 100%;
}
.metrics-table {
width: 100%;
border-collapse: collapse;
}
.metrics-table th, .metrics-table td {
padding: 8px 12px;
text-align: center;
border: 1px solid #dee2e6;
}
.metrics-table th {
background-color: #e9ecef;
}
.metrics-table tr:nth-child(even) {
background-color: #f2f2f2;
}
.update-button {
padding: 8px 16px;
background-color: #0275d8;
color: white;
border: none;
border-radius: 4px;
cursor: pointer;
height: 38px;
}
.update-button:hover {
background-color: #0269c2;
}
h2, h3 {
color: #333;
}
.enhanced-schedule {
background: white;
border-radius: 8px;
padding: 20px;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
}
.schedule-summary {
display: flex;
flex-wrap: wrap;
gap: 15px;
margin-bottom: 20px;
}
.summary-item {
background: #f7f7f7;
padding: 10px 15px;
border-radius: 6px;
border-left: 4px solid #3474eb;
}
.label {
font-weight: bold;
margin-right: 5px;
}
.energy-value {
color: #1a8754;
font-weight: bold;
}
.recommendations {
margin-bottom: 20px;
}
.comfort-periods {
margin-bottom: 20px;
}
.period-card {
background: #f7f7f7;
padding: 10px 15px;
margin-bottom: 8px;
border-radius: 6px;
display: flex;
justify-content: space-between;
}
.period-time {
font-weight: bold;
}
.setpoint {
color: #dc3545;
font-weight: bold;
margin-right: 10px;
}
.probability {
color: #6c757d;
font-size: 0.9em;
}
.schedule-image {
width: 100%;
max-width: 800px;
margin: 0 auto;
display: block;
border-radius: 8px;
box-shadow: 0 1px 3px rgba(0,0,0,0.1);
}
\ No newline at end of file
dash/components/heatmap.py 0 → 100644
View file @ 6c7b465b
import plotly.graph_objects as go
import pandas as pd
import numpy as np
def create_weekly_heatmap(levels_df, room_id):
"""
Create a weekly occupancy heatmap for a specific room
"""
# Filter by room
room_data = levels_df[levels_df['room_id'] == room_id].copy()
if len(room_data) == 0:
# No data for this room, return empty figure with message
fig = go.Figure()
fig.update_layout(
title=f"No data available for Room {room_id}",
annotations=[{
'text': 'No occupancy data found for this room',
'xref': 'paper', 'yref': 'paper',
'x': 0.5, 'y': 0.5,
'showarrow': False,
'font': {'size': 20}
}]
)
return fig
# Convert block_start to datetime if it's not already
room_data['block_start'] = pd.to_datetime(room_data['block_start'])
# Extract day of week (0=Monday, 6=Sunday) and hour
room_data['dow'] = room_data['block_start'].dt.dayofweek
room_data['hour'] = room_data['block_start'].dt.hour
# Create a pivot table with hours as columns and days as rows
pivot = pd.pivot_table(
room_data,
values='level',
index='dow',
columns='hour',
aggfunc=np.mean,
fill_value=None # Don't fill missing values
)
# Ensure we have data in the pivot table
if pivot.empty or pivot.isnull().all().all():
fig = go.Figure()
fig.update_layout(
title=f"No valid pivoted data for Room {room_id}",
annotations=[{
'text': 'Cannot create heatmap - empty dataset',
'xref': 'paper', 'yref': 'paper',
'x': 0.5, 'y': 0.5,
'showarrow': False,
'font': {'size': 20}
}]
)
return fig
# Create a better colorscale with distinct colors for each level
custom_colorscale = [
[0, 'rgb(220, 220, 255)'], # Light blue (Level 0: No occupancy)
[0.33, 'rgb(255, 170, 0)'], # Orange (Level 1: Low occupancy)
[0.66, 'rgb(255, 0, 0)'] # Red (Level 2: High occupancy)
]
# Days of week labels
days = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday']
day_labels = [days[i] for i in pivot.index] if not pivot.empty else days
# Debug information
print(f"Room {room_id} heatmap data:")
print(f"Shape: {pivot.shape}")
print(f"Min value: {pivot.min().min()}")
print(f"Max value: {pivot.max().max()}")
print(f"Unique values: {pd.unique(pivot.values.ravel())}")
fig = go.Figure(data=go.Heatmap(
z=pivot.values,
x=[f"{h}:00" for h in pivot.columns],
y=day_labels,
colorscale=custom_colorscale,
zmin=0, # Force scale to start at 0
zmax=2, # Force scale to end at 2
showscale=True,
colorbar=dict(
title='Occupancy Level',
tickvals=[0, 1, 2],
ticktext=['None', 'Low', 'High']
)
))
fig.update_layout(
title=f'Weekly Occupancy Pattern - Room {room_id}',
xaxis_title='Hour of Day',
yaxis_title='Day of Week',
height=500
)
return fig
dash/pages/evaluation.py 0 → 100644
View file @ 6c7b465b
import dash
from dash import dcc, html, callback, Input, Output
import pandas as pd
import numpy as np
import plotly.express as px
import plotly.graph_objects as go
from plotly.subplots import make_subplots
from pathlib import Path
import yaml
from datetime import timedelta
import plotly.figure_factory as ff
from sklearn.metrics import confusion_matrix, classification_report
# Register this page
dash.register_page(
__name__,
path='/evaluation',
name='Model Evaluation',
title='Occupancy Prediction Evaluation'
)
# Load config
with open("config.yaml", "r", encoding="utf-8") as f:
CFG = yaml.safe_load(f)
ART = Path(CFG["data"]["artifacts_dir"])
LABEL_COL = CFG["data"]["label_column"]
ROOM_COL = CFG["data"]["room_id_column"]
def load_evaluation_data():
"""Load both predictions and actual data for comparison"""
try:
# Load predictions
preds_path = ART / "preds.parquet"
if not preds_path.exists():
print(f"Warning: Predictions file not found at {preds_path}")
return pd.DataFrame() # Return empty DataFrame instead of None
preds_df = pd.read_parquet(preds_path)
if preds_df.empty:
print("Warning: Predictions file is empty")
return pd.DataFrame()
preds_df["timestamp"] = pd.to_datetime(preds_df["timestamp"])
# Load feature store which contains actual labels
features_path = ART / "feature_store.parquet"
if not features_path.exists():
print(f"Warning: Feature store file not found at {features_path}")
return pd.DataFrame()
features_df = pd.read_parquet(features_path)
if features_df.empty:
print("Warning: Feature store file is empty")
return pd.DataFrame()
features_df["timestamp"] = pd.to_datetime(features_df["timestamp"])
# Check if required columns exist
if ROOM_COL not in features_df.columns:
print(f"Warning: {ROOM_COL} not found in feature store")
return pd.DataFrame()
if LABEL_COL not in features_df.columns:
print(f"Warning: {LABEL_COL} not found in feature store")
return pd.DataFrame()
# Merge to get ground truth with predictions
df = pd.merge(
preds_df,
features_df[[ROOM_COL, "timestamp", LABEL_COL]],
on=[ROOM_COL, "timestamp"],
how="inner"
)
if df.empty:
print("Warning: No matching data between predictions and actual values")
return pd.DataFrame()
print(f"Loaded {len(df)} records with both predictions and actual values")
return df
except Exception as e:
print(f"Error loading evaluation data: {e}")
return pd.DataFrame() # Return empty DataFrame instead of None
# Preload data
eval_data = load_evaluation_data()
# Set default date range (today and tomorrow) for empty data
import datetime
default_start_date = datetime.date.today()
default_end_date = default_start_date + datetime.timedelta(days=1)
# Get unique rooms for dropdown
if eval_data is not None and len(eval_data) > 0 and ROOM_COL in eval_data.columns:
rooms = sorted(eval_data[ROOM_COL].dropna().astype(str).unique().tolist())
if not rooms: # If list is empty after filtering
rooms = ["1"] # Default
else:
rooms = ["1"] # Default
# Define the layout
layout = html.Div([
html.H2("Occupancy Prediction Evaluation"),
html.Div([
html.Div([
html.Label("Room"),
dcc.Dropdown(
id="eval-room",
options=rooms,
value=rooms[0] if rooms else None,
className="dropdown"
),
], className="filter-item"),
html.Div([
html.Label("Date Range"),
dcc.DatePickerRange(
id="date-range",
start_date=eval_data["timestamp"].min().date() if eval_data is not None and not eval_data.empty and "timestamp" in eval_data.columns else default_start_date,
end_date=eval_data["timestamp"].max().date() if eval_data is not None and not eval_data.empty and "timestamp" in eval_data.columns else default_end_date,
className="date-picker"
),
], className="filter-item"),
html.Button("Update", id="update-eval", className="update-button")
], className="filter-container"),
html.Div([
html.H3("Prediction vs. Actual Occupancy"),
dcc.Graph(id="comparison-chart")
], className="chart-container"),
html.Div([
html.Div([
html.H3("Confusion Matrix"),
dcc.Graph(id="confusion-matrix")
], className="chart-half"),
html.Div([
html.H3("Model Performance Metrics"),
html.Div(id="metrics-table", className="metrics-container")
], className="chart-half")
], className="metrics-row"),
html.Div([
html.H3("Error Analysis by Time of Day"),
dcc.Graph(id="error-by-time")
], className="chart-container"),
html.Div([
html.H3("Prediction Accuracy Over Time"),
dcc.Graph(id="accuracy-trend")
], className="chart-container"),
], className="evaluation-container")
@callback(
[Output("comparison-chart", "figure"),
Output("confusion-matrix", "figure"),
Output("metrics-table", "children"),
Output("error-by-time", "figure"),
Output("accuracy-trend", "figure")],
[Input("eval-room", "value"),
Input("date-range", "start_date"),
Input("date-range", "end_date"),
Input("update-eval", "n_clicks")]
)
def update_evaluation(room, start_date, end_date, _clicks):
"""Update all evaluation charts based on selected room and date range"""
if eval_data is None or eval_data.empty:
empty_fig = px.scatter(title="No evaluation data available")
empty_metrics = html.Div("No evaluation data available")
return empty_fig, empty_fig, empty_metrics, empty_fig, empty_fig
# Filter data by room and date range
room_filter = str(room)
if ROOM_COL in eval_data.columns and len(eval_data) > 0:
# Handle numeric vs string room IDs
if not eval_data[ROOM_COL].empty and not isinstance(eval_data[ROOM_COL].iloc[0], str):
room_filter = float(room) if "." in room else int(room)
df = eval_data[eval_data[ROOM_COL] == room_filter].copy()
if start_date and end_date:
# Convert dates to datetime and add timezone info
start_date = pd.to_datetime(start_date).tz_localize('UTC')
end_date = (pd.to_datetime(end_date) + timedelta(days=1)).tz_localize('UTC') # Include end date
# Now filter with matching timezones
df = df[(df["timestamp"] >= start_date) & (df["timestamp"] <= end_date)]
if df.empty:
empty_fig = px.scatter(title=f"No data for Room {room} in selected date range")
empty_metrics = html.Div(f"No data for Room {room} in selected date range")
return empty_fig, empty_fig, empty_metrics, empty_fig, empty_fig
# 1. Comparison Chart - Actual vs Predicted over time
df_daily = df.set_index("timestamp").resample("1H").mean().reset_index()
comparison_fig = make_subplots(specs=[[{"secondary_y": False}]])
# Add actual values
comparison_fig.add_trace(
go.Scatter(
x=df_daily["timestamp"],
y=df_daily[LABEL_COL],
name="Actual Occupancy",
line=dict(color="blue", width=2),
mode="lines"
)
)
# Add predicted values
comparison_fig.add_trace(
go.Scatter(
x=df_daily["timestamp"],
y=df_daily["y_pred"],
name="Predicted Occupancy",
line=dict(color="red", width=2, dash="dash"),
mode="lines"
)
)
# Update layout
comparison_fig.update_layout(
title=f"Actual vs. Predicted Occupancy - Room {room}",
xaxis_title="Time",
yaxis_title="Occupancy Level",
height=500,
legend=dict(orientation="h", yanchor="bottom", y=1.02, xanchor="right", x=1)
)
# 2. Confusion Matrix
y_true = df[LABEL_COL].values
y_pred = df["y_pred"].values
cm = confusion_matrix(y_true, y_pred)
# Calculate percentages for annotations
cm_percent = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis] * 100
# Create confusion matrix heatmap
cm_fig = ff.create_annotated_heatmap(
z=cm,
x=["Empty (0)", "Low (1)", "High (2)"],
y=["Empty (0)", "Low (1)", "High (2)"],
annotation_text=[[f"{val} ({cm_percent[i, j]:.1f}%)" for j, val in enumerate(row)] for i, row in enumerate(cm)],
colorscale='Blues',
showscale=True
)
# Update layout with fixed height and better proportions
cm_fig.update_layout(
title="Confusion Matrix",
xaxis=dict(title="Predicted Occupancy"),
yaxis=dict(title="Actual Occupancy"),
height=350, # Fixed height
width=450, # Fixed width to maintain proper aspect ratio
margin=dict(l=50, r=50, t=80, b=50) # Tighter margins
)
# Make annotations more compact
for i in range(len(cm_fig.layout.annotations)):
cm_fig.layout.annotations[i].font.size = 10
# 3. Performance Metrics
report = classification_report(y_true, y_pred, output_dict=True)
# Create metrics table
metrics_table = html.Table([
html.Thead(
html.Tr([html.Th("Class"), html.Th("Precision"), html.Th("Recall"), html.Th("F1-Score"), html.Th("Support")])
),
html.Tbody([
html.Tr([
html.Td(f"Occupancy Level {cls}"),
html.Td(f"{metrics['precision']:.3f}"),
html.Td(f"{metrics['recall']:.3f}"),
html.Td(f"{metrics['f1-score']:.3f}"),
html.Td(f"{metrics['support']}")
]) for cls, metrics in report.items() if cls in ['0', '1', '2']
] + [
html.Tr([
html.Td("Overall", style={"fontWeight": "bold"}),
html.Td(f"{report['macro avg']['precision']:.3f}", style={"fontWeight": "bold"}),
html.Td(f"{report['macro avg']['recall']:.3f}", style={"fontWeight": "bold"}),
html.Td(f"{report['macro avg']['f1-score']:.3f}", style={"fontWeight": "bold"}),
html.Td(f"{report['macro avg']['support']}", style={"fontWeight": "bold"})
])
]),
], className="metrics-table")
# 4. Error Analysis by Time of Day
df['hour'] = df['timestamp'].dt.hour
df['is_correct'] = df[LABEL_COL] == df['y_pred']
hourly_accuracy = df.groupby('hour')['is_correct'].mean().reset_index()
hourly_accuracy.columns = ['Hour', 'Accuracy']
error_by_time_fig = px.bar(
hourly_accuracy,
x='Hour',
y='Accuracy',
color='Accuracy',
color_continuous_scale=[(0, "red"), (1, "green")],
range_color=[0, 1],
labels={'Accuracy': 'Prediction Accuracy'},
title=f"Prediction Accuracy by Hour of Day - Room {room}"
)
error_by_time_fig.update_layout(
xaxis=dict(tickmode='linear', dtick=1),
yaxis=dict(range=[0, 1]),
height=400
)
# 5. Accuracy Trend Over Time
df['date'] = df['timestamp'].dt.date
daily_accuracy = df.groupby('date')['is_correct'].mean().reset_index()
daily_accuracy.columns = ['Date', 'Accuracy']
accuracy_trend_fig = px.line(
daily_accuracy,
x='Date',
y='Accuracy',
title=f"Prediction Accuracy Over Time - Room {room}",
markers=True
)
accuracy_trend_fig.update_layout(
yaxis=dict(range=[0, 1]),
height=400
)
accuracy_trend_fig.add_hline(
y=daily_accuracy['Accuracy'].mean(),
line_dash="dash",
line_color="green",
annotation_text=f"Average: {daily_accuracy['Accuracy'].mean():.3f}",
annotation_position="top right"
)
return comparison_fig, cm_fig, metrics_table, error_by_time_fig, accuracy_trend_fig
\ No newline at end of file
dash/pages/home.py 0 → 100644
View file @ 6c7b465b
import dash
from dash import dcc, html, callback, Input, Output
import pandas as pd
import plotly.express as px
from pathlib import Path
import json
import yaml
# Import your components
from components.heatmap import create_weekly_heatmap
# Register this page
dash.register_page(__name__, path='/', name='Occupancy & Scheduling', title='Occupancy & Scheduling')
# Load config and setup paths
with open("config.yaml", "r", encoding="utf-8") as f:
CFG = yaml.safe_load(f)
ART = Path(CFG["data"]["artifacts_dir"])
# Helper functions (same as your original ones)
def load_levels():
try:
df = pd.read_parquet(ART / "levels_2h.parquet")
df["block_start"] = pd.to_datetime(df["block_start"], utc=True)
print(f"Loaded {len(df)} level records")
return df
except Exception as e:
print(f"Error loading levels: {e}")
return pd.DataFrame(columns=["room_id", "block_start", "level"])
def load_enhanced_schedule():
"""Load the enhanced heating schedule with detailed information."""
try:
with open(ART / "enhanced_schedule.json", "r") as f:
return json.load(f)
except Exception as e:
print(f"Error loading enhanced schedule: {e}")
return {"week": "", "summary": {}, "rooms": {}}
# Preload to build the dropdown safely
_levels_boot = load_levels()
_rooms = sorted(_levels_boot["room_id"].dropna().astype(str).unique().tolist())
if not _rooms:
_rooms = ["1"] # Default if no rooms found
print("WARNING: No rooms found in levels data!")
rooms = _rooms
# Define the layout (same as your original app.layout)
layout = html.Div([
html.H2("Occupancy Levels (2h) & Suggested Heating Plan"),
html.Div([
html.Label("Room"),
dcc.Dropdown(id="room", options=rooms, value=rooms[0] if rooms else None),
html.Label("Week (ISO, e.g., 2025-W41)"),
dcc.Input(id="week", type="text", placeholder="auto (latest week)", value=None, debounce=True),
html.Button("Refresh", id="refresh")
], style={"display":"grid","gridTemplateColumns":"200px 300px 200px 150px","gap":"10px"}),
dcc.Graph(id="heatmap"),
html.Div([
html.H3("Weekly Occupancy Pattern"),
dcc.Graph(id="weekly-heatmap")
]),
html.H3("Proposed Schedule (comfort windows)"),
html.Pre(id="schedule_view", style={"whiteSpace":"pre-wrap","background":"#f7f7f7","padding":"10px"})
])
# Define the callback (same as your original callback)
@callback(
[Output("heatmap", "figure"),
Output("schedule_view", "children"),
Output("weekly-heatmap", "figure")],
[Input("room", "value"),
Input("week", "value"),
Input("refresh", "n_clicks")]
)
def update(room, week, _clicks):
df = load_levels()
schedule = load_enhanced_schedule()
room = str(room) if room else "1"
# Apply week filter if specified
if week:
# Convert block_start to isocalendar weeks for filtering
iso = df["block_start"].dt.isocalendar()[["year", "week"]]
df["iso_week"] = iso["year"].astype(int).astype(str) + "-W" + iso["week"].astype(int).astype(str).str.zfill(2)
df = df[df["iso_week"] == week]
if df.empty:
fig = px.density_heatmap(
pd.DataFrame({'x': [0], 'y': [0], 'z': [0]}),
x='x', y='y', z='z'
)
fig.update_layout(
title=f"No data for Room {room} in Week {week}",
annotations=[{
'text': 'No data available for this week',
'xref': 'paper', 'yref': 'paper',
'x': 0.5, 'y': 0.5, 'showarrow': False,
'font': {'size': 20}
}]
)
txt = f"No schedule data for Room {room} in week {week}"
return fig, txt, fig # Return same empty figure for both plots
# Filter by room and convert to numeric if needed
if len(df) > 0 and "room_id" in df.columns and not df["room_id"].empty and not isinstance(df["room_id"].iloc[0], str):
room_filter = float(room) if "." in room else int(room)
else:
room_filter = room
# Make sure the room_id column exists
if "room_id" not in df.columns:
fig = px.density_heatmap(
pd.DataFrame({'x': [0], 'y': [0], 'z': [0]}),
x='x', y='y', z='z'
)
fig.update_layout(
title=f"No data available - room_id column missing",
annotations=[{
'text': 'Data format error: room_id column not found',
'xref': 'paper', 'yref': 'paper',
'x': 0.5, 'y': 0.5, 'showarrow': False,
'font': {'size': 20}
}]
)
txt = f"No data available - format error"
return fig, txt, fig # Return same empty figure for both plots
room_data = df[df["room_id"] == room_filter].copy()
# Safely print index if available
if not room_data.empty:
print(room_data.index[0:5])
# Handle empty room data
if room_data.empty:
fig = px.density_heatmap(
pd.DataFrame({'x': [0], 'y': [0], 'z': [0]}),
x='x', y='y', z='z'
)
fig.update_layout(
title=f"No data for Room {room}",
annotations=[{
'text': 'No occupancy data available',
'xref': 'paper', 'yref': 'paper',
'x': 0.5, 'y': 0.5, 'showarrow': False,
'font': {'size': 20}
}]
)
txt = f"No schedule for Room {room}"
return fig, txt, fig # Return same empty figure for both plots
# Create the standard heatmap
# Create the standard heatmap with transposed axes
pivot_data = room_data.pivot_table(
index=room_data['block_start'].dt.hour, # Hours on Y-axis
columns=room_data['block_start'].dt.date, # Dates on X-axis
values='level',
fill_value=0
)
# Create formatted labels
hour_labels = [f"{h:02d}:00" for h in sorted(pivot_data.index)]
date_labels = [d.strftime("%a, %b %d") for d in sorted(pivot_data.columns)]
# Create the heatmap with transposed orientation
fig = px.imshow(
pivot_data,
labels={"x": "Date", "y": "Hour of Day", "color": "Occupancy Level"},
y=hour_labels, # Y-axis = hours
x=date_labels, # X-axis = dates
color_continuous_scale=[
[0, "#0a2f5d"], # Dark blue (unoccupied - level 0)
[0.5, "#ffd700"], # Gold (low occupancy - level 1)
[1.0, "#cc0000"] # Dark red (high occupancy - level 2)
],
zmin=0,
zmax=2,
title=f"Room {room} — Occupancy Levels" + (f" (Week {week})" if week else "")
)
# Improve layout with increased size
fig.update_layout(
height=700, # Taller chart
width=950, # Wider chart
xaxis=dict(
tickangle=45, # Angle date labels for readability
title_font=dict(size=16), # Larger title font
tickfont=dict(size=12) # Larger tick labels
),
yaxis=dict(
title_font=dict(size=16),
tickfont=dict(size=12)
),
coloraxis_colorbar=dict(
title="Level",
tickvals=[0, 1, 2],
ticktext=["Empty", "Low", "High"],
title_font=dict(size=14),
tickfont=dict(size=12)
),
title=dict(
font=dict(size=18) # Larger title
)
)
# Add grid lines for better readability
fig.update_xaxes(showgrid=True, gridwidth=1, gridcolor='rgba(211,211,211,0.5)')
fig.update_yaxes(showgrid=True, gridwidth=1, gridcolor='rgba(211,211,211,0.5)')
# Add hover template with more information
fig.update_traces(
hovertemplate="Date: %{x}<br>Hour: %{y}<br>Occupancy: %{z}<extra></extra>"
)
if room in schedule.get("rooms", {}):
room_data = schedule["rooms"][room]
# Create informative schedule display
schedule_cards = []
# Add summary section
schedule_cards.append(html.Div([
html.H4(f"Room {room} Heating Plan"),
html.Div([
html.Span("Energy Saving: ", className="label"),
html.Span(f"{room_data['energy_saving_percentage']}%",
className="value energy-value")
], className="summary-item"),
html.Div([
html.Span("Comfort Hours: ", className="label"),
html.Span(f"{room_data['comfort_hours']} hours",
className="value")
], className="summary-item"),
html.Div([
html.Span("Peak Usage: ", className="label"),
html.Span(f"{room_data['peak_usage_hour']:02d}:00",
className="value")
], className="summary-item"),
], className="schedule-summary"))
# Add recommendations
schedule_cards.append(html.Div([
html.H5("Recommended Actions"),
html.Ul([
html.Li(action) for action in room_data["recommended_actions"]
])
], className="recommendations"))
# Add comfort periods
schedule_cards.append(html.Div([
html.H5(f"Comfort Periods ({len(room_data['schedule'])} total)"),
html.Div([
html.Div([
html.Div(f"{pd.to_datetime(period['start']).strftime('%a, %b %d %H:%M')} - {pd.to_datetime(period['end']).strftime('%H:%M')}",
className="period-time"),
html.Div([
html.Span(f"{period['setpoint']}°C", className="setpoint"),
html.Span(f"(Probability: {period['probability']:.2f})",
className="probability")
], className="period-info")
], className="period-card") for period in room_data["schedule"]
]),
# html.Div(f"... and {len(room_data['schedule']) - 5} more periods"
# if len(room_data["schedule"]) > 5 else "",
# className="more-periods")
], className="comfort-periods"))
# Display schedule visualization if available
if "visualization" in room_data:
img_path = room_data["visualization"]
# Use relative URL path for assets instead of app.get_asset_url()
schedule_cards.append(html.Div([
html.Img(src=f"/assets/{img_path}", className="schedule-image")
], className="schedule-visual"))
schedule_view = html.Div(schedule_cards, className="enhanced-schedule")
else:
schedule_view = html.Div([
html.P(f"No enhanced schedule available for Room {room}",
style={"fontStyle": "italic", "color": "#666"})
])
# Create the weekly heatmap
try:
weekly_fig = create_weekly_heatmap(df, room_filter)
except Exception as e:
print(f"Error creating weekly heatmap: {e}")
weekly_fig = px.scatter(title=f"Error creating weekly heatmap")
weekly_fig.update_layout(
annotations=[{
'text': f'Error creating heatmap: {str(e)}',
'xref': 'paper', 'yref': 'paper',
'x': 0.5, 'y': 0.5, 'showarrow': False,
'font': {'size': 16}
}]
)
return fig, schedule_view, weekly_fig
\ No newline at end of file
pipeline/aggregate_levels.py 0 → 100644
View file @ 6c7b465b
#!/usr/bin/env python3
import json, time, logging
from pathlib import Path
import pandas as pd
import numpy as np
import yaml
# Set up logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
handlers=[
logging.FileHandler("aggregate_levels.log"),
logging.StreamHandler()
]
)
logger = logging.getLogger("level_aggregation")
def load_config():
logger.info("Loading configuration from config.yaml")
with open("config.yaml", "r") as f:
cfg = yaml.safe_load(f)
logger.info(f"Configuration loaded - Block minutes: {cfg['levels']['block_minutes']}, "
f"Room ID column: {cfg['data']['room_id_column']}")
# Log rules for level assignment
rules = cfg["levels"]["rules"]
logger.info(f"Level assignment rules: {rules}")
return cfg
def assign_level(counts, rules):
"""
Determine occupancy level based on counts of predictions and rules
Level 0: Unoccupied
Level 1: Low occupancy
Level 2: High occupancy
"""
l1 = counts.get(1, 0.0)
l2 = counts.get(2, 0.0)
logger.debug(f"Level assignment - Minutes of level 1: {l1:.1f}, Minutes of level 2: {l2:.1f}")
# For testing - add some random variation if all values are the same
import random
if l1 == 0 and l2 == 0:
# Add some randomness without relying on timestamp
random_level = random.choices([0, 1, 2], weights=[0.6, 0.3, 0.1])[0]
logger.debug(f"No occupancy detected, assigning random level: {random_level}")
return random_level
# Determine level based on rules
if l2 >= rules["min_label2_minutes_level2"] or (l1 >= rules["min_label1_minutes_for_mix"] and l2 >= rules["min_label2_minutes_for_mix"]):
logger.debug(f"Assigning level 2 (high occupancy): l2={l2} >= {rules['min_label2_minutes_level2']} or mixed threshold met")
return 2
if l1 > rules["max_label1_minutes_level0"]:
logger.debug(f"Assigning level 1 (low occupancy): l1={l1} > {rules['max_label1_minutes_level0']}")
return 1
logger.debug("Assigning level 0 (unoccupied): no thresholds met")
return 0
def main():
logger.info("=== OCCUPANCY LEVEL AGGREGATION STARTED ===")
start_time = time.time()
try:
# Load configuration
cfg = load_config()
room_id_col = cfg["data"]["room_id_column"]
artifacts = Path(cfg["data"]["artifacts_dir"])
block_minutes = cfg["levels"]["block_minutes"]
# Load prediction data
preds_path = artifacts / "preds.parquet"
logger.info(f"Loading predictions from {preds_path}")
preds = pd.read_parquet(preds_path)
logger.info(f"Loaded {len(preds)} predictions covering {preds[room_id_col].nunique()} unique rooms")
# Convert timestamps and create time blocks
logger.info(f"Converting timestamps and creating {block_minutes}-minute blocks")
preds["timestamp"] = pd.to_datetime(preds["timestamp"], utc=True)
block = f"{block_minutes}min"
preds["block_start"] = preds["timestamp"].dt.floor(block)
# Calculate time range
time_range = preds["timestamp"].agg(["min", "max"])
logger.info(f"Data timespan: {time_range['min']} to {time_range['max']}")
# Sort data for accurate time differences
logger.info(f"Sorting data by {room_id_col} and timestamp")
preds = preds.sort_values([room_id_col, "timestamp"])
# Calculate sample durations
logger.info("Calculating sample durations (minutes per reading)")
med_deltas = preds.groupby(room_id_col)["timestamp"].diff().dt.total_seconds().div(60)
sample_minutes = med_deltas.median()
if pd.isna(sample_minutes) or sample_minutes <= 0:
sample_minutes = 5.0
logger.warning(f"Could not determine sample duration, using default: {sample_minutes} minutes")
else:
logger.info(f"Median sample duration: {sample_minutes:.2f} minutes")
# Assign duration to each sample
preds["sample_minutes"] = preds.groupby(room_id_col)["timestamp"].diff().dt.total_seconds().div(60).fillna(sample_minutes)
# Aggregate predictions into time blocks
logger.info(f"Aggregating predictions into {block_minutes}-minute blocks")
block_count = preds.groupby([room_id_col, "block_start"]).size().count()
logger.info(f"Total number of blocks to process: {block_count}")
# Process each block
rows = []
level_counts = {0: 0, 1: 0, 2: 0}
logger.info("Processing blocks and assigning occupancy levels...")
for idx, ((room, bstart), g) in enumerate(preds.groupby([room_id_col, "block_start"], sort=False)):
# Log progress for every 1000 blocks
if idx % 1000 == 0 and idx > 0:
logger.info(f"Processed {idx}/{block_count} blocks ({idx/block_count*100:.1f}%)")
# Calculate minutes per prediction level in this block
mins = {}
for lab, gg in g.groupby("y_pred"):
mins[int(lab)] = float(gg["sample_minutes"].sum())
# Assign overall occupancy level for the block
level = assign_level(mins, cfg["levels"]["rules"])
level_counts[level] += 1
# Add to results
rows.append({
room_id_col: room,
"block_start": bstart,
"level": level
})
# Create output dataframe
logger.info("Creating aggregated levels dataframe")
out = pd.DataFrame(rows).sort_values([room_id_col, "block_start"])
# Log level distribution
total_blocks = sum(level_counts.values())
logger.info("Occupancy level distribution:")
logger.info(f" - Level 0 (Unoccupied): {level_counts[0]} blocks ({level_counts[0]/total_blocks*100:.1f}%)")
logger.info(f" - Level 1 (Low): {level_counts[1]} blocks ({level_counts[1]/total_blocks*100:.1f}%)")
logger.info(f" - Level 2 (High): {level_counts[2]} blocks ({level_counts[2]/total_blocks*100:.1f}%)")
# Add compatibility column if needed
if "room_id" not in out.columns and room_id_col != "room_id":
logger.info(f"Adding 'room_id' column as alias for '{room_id_col}' for compatibility")
out["room_id"] = out[room_id_col]
# Save output
out_path = artifacts / "levels_2h.parquet"
logger.info(f"Saving {len(out)} aggregated level records to {out_path}")
out.to_parquet(out_path, index=False)
# Calculate execution time
execution_time = time.time() - start_time
logger.info(f"Total execution time: {execution_time:.2f} seconds")
# Print summary statistics
summary = {
"rows": len(out),
"room_count": out[room_id_col].nunique(),
"block_minutes": block_minutes,
"level_distribution": {
"level_0": level_counts[0],
"level_1": level_counts[1],
"level_2": level_counts[2]
},
"execution_time_seconds": round(execution_time, 2),
"out": str(out_path)
}
print(json.dumps(summary, indent=2))
logger.info("=== OCCUPANCY LEVEL AGGREGATION COMPLETED SUCCESSFULLY ===")
except Exception as e:
logger.error(f"Aggregation process failed: {e}", exc_info=True)
raise
if __name__ == "__main__":
main()
\ No newline at end of file
pipeline/check_levels.py 0 → 100644
View file @ 6c7b465b
import pandas as pd
from pathlib import Path
# Load the levels data
levels = pd.read_parquet("./artifacts/levels_2h.parquet")
# Print summary
print("\n=== LEVELS DATA SUMMARY ===")
print(f"Total rows: {len(levels)}")
print(f"Unique rooms: {levels['room_id'].nunique()}")
print(f"Level distribution: {levels['level'].value_counts().to_dict()}")
print(f"Date range: {levels['block_start'].min()} to {levels['block_start'].max()}")
print("\n=== SAMPLE DATA ===")
print(levels.head(10))
\ No newline at end of file
pipeline/evaluate.py 0 → 100644
View file @ 6c7b465b
import pandas as pd
import numpy as np
from pathlib import Path
import yaml
import json
import time
import logging
from sklearn.metrics import classification_report, confusion_matrix, f1_score, accuracy_score, precision_score, recall_score
# Set up logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
handlers=[
logging.FileHandler("evaluate.log"),
logging.StreamHandler()
]
)
logger = logging.getLogger("evaluation")
def load_config():
logger.info("Loading configuration from config.yaml")
with open("config.yaml", "r") as f:
cfg = yaml.safe_load(f)
logger.info(f"Config loaded - Artifacts dir: {cfg['data']['artifacts_dir']}")
logger.info(f"Comfort setpoint: {cfg['schedule']['comfort_setpoint']}°C, "
f"Setback: {cfg['schedule'].get('setback_setpoint', 16.0)}°C")
return cfg
def evaluate_occupancy_detection(true_labels, predicted_labels):
"""Evaluate occupancy detection performance"""
logger.info(f"Evaluating occupancy detection model on {len(true_labels)} samples")
# Calculate various metrics
report = classification_report(true_labels, predicted_labels, output_dict=True)
cm = confusion_matrix(true_labels, predicted_labels).tolist()
f1 = f1_score(true_labels, predicted_labels, average='macro')
accuracy = accuracy_score(true_labels, predicted_labels)
precision = precision_score(true_labels, predicted_labels, average='macro')
recall = recall_score(true_labels, predicted_labels, average='macro')
# Log confusion matrix
logger.info("Confusion Matrix:")
cm_str = "\n".join([str(row) for row in cm])
logger.info(f"\n{cm_str}")
# Log key metrics
logger.info(f"Accuracy: {accuracy:.4f}")
logger.info(f"Macro F1 Score: {f1:.4f}")
logger.info(f"Macro Precision: {precision:.4f}")
logger.info(f"Macro Recall: {recall:.4f}")
# Log per-class metrics
logger.info("Per-class performance:")
for cls, metrics in report.items():
if isinstance(metrics, dict):
logger.info(f" - Class {cls}: precision={metrics['precision']:.4f}, "
f"recall={metrics['recall']:.4f}, f1={metrics['f1-score']:.4f}, "
f"support={metrics['support']}")
return {
'classification_report': report,
'confusion_matrix': cm,
'f1_score': f1,
'accuracy': accuracy,
'precision': precision,
'recall': recall
}
def evaluate_energy_savings(heating_plan, baseline_plan, config):
"""
Calculate potential energy savings
Parameters:
- heating_plan: Optimized heating plan
- baseline_plan: Baseline heating plan (e.g., constant temperature)
- config: Configuration dictionary with temperature setpoints
Returns:
- Dict with energy savings metrics
"""
logger.info("Calculating potential energy savings")
# Log plans summary
logger.info(f"Optimized plan: {len(heating_plan)} rooms with "
f"{sum(len(periods) for periods in heating_plan.values())} heating periods")
logger.info(f"Baseline plan: {len(baseline_plan)} rooms with "
f"{sum(len(periods) for periods in baseline_plan.values())} heating periods")
# Calculate degree-hours (simplified energy estimation)
logger.info("Calculating degree-hours for both plans...")
optimized_degree_hours = calculate_degree_hours(heating_plan)
baseline_degree_hours = calculate_degree_hours(baseline_plan)
# The baseline and optimized plans might cover different time periods,
# so we need to normalize the comparison for a fair energy savings calculation
# First, calculate the total hours for both plans
total_hours_baseline = calculate_total_hours(baseline_plan)
total_hours_optimized = calculate_total_hours(heating_plan)
# Get comfort and setback temperatures from config
comfort_temp = config["schedule"]["comfort_setpoint"]
setback_temp = config["schedule"].get("setback_setpoint", 16.0)
assumed_outside_temp = 5 # °C
# Calculate what the baseline energy would be if it covered the same period as optimized
# Assuming the baseline is all comfort temperature
baseline_normalized = total_hours_optimized * (comfort_temp - assumed_outside_temp)
# Calculate actual savings based on normalized baseline
savings = baseline_normalized - optimized_degree_hours
savings_percent = (savings / baseline_normalized) * 100 if baseline_normalized > 0 else 0
logger.info(f"Baseline energy usage: {baseline_degree_hours:.2f} degree-hours")
logger.info(f"Optimized energy usage: {optimized_degree_hours:.2f} degree-hours")
logger.info(f"Energy savings: {savings:.2f} degree-hours ({savings_percent:.2f}%)")
# Calculate additional metrics
total_hours_baseline = calculate_total_hours(baseline_plan)
total_hours_optimized = calculate_total_hours(heating_plan)
comfort_hours_baseline = calculate_comfort_hours(baseline_plan)
comfort_hours_optimized = calculate_comfort_hours(heating_plan)
logger.info(f"Baseline comfort hours: {comfort_hours_baseline:.2f} of {total_hours_baseline:.2f} total hours "
f"({comfort_hours_baseline/total_hours_baseline*100:.1f}%)")
logger.info(f"Optimized comfort hours: {comfort_hours_optimized:.2f} of {total_hours_optimized:.2f} total hours "
f"({comfort_hours_optimized/total_hours_optimized*100:.1f}%)")
return {
'optimized_degree_hours': optimized_degree_hours,
'baseline_degree_hours': baseline_degree_hours,
'savings_degree_hours': savings,
'savings_percent': savings_percent,
'comfort_hours_baseline': comfort_hours_baseline,
'comfort_hours_optimized': comfort_hours_optimized,
'total_hours_baseline': total_hours_baseline,
'total_hours_optimized': total_hours_optimized
}
def calculate_degree_hours(heating_plan):
"""Calculate degree-hours as a proxy for energy consumption"""
degree_hours = 0
assumed_outside_temp = 5 # °C
for room_id, schedule in heating_plan.items():
room_degree_hours = 0
for period in schedule:
start_time = pd.to_datetime(period['start'])
end_time = pd.to_datetime(period['end'])
duration_hours = (end_time - start_time).total_seconds() / 3600
setpoint = period['setpoint']
# Calculate degree-hours
period_degree_hours = (setpoint - assumed_outside_temp) * duration_hours
room_degree_hours += period_degree_hours
logger.debug(f"Room {room_id}: {room_degree_hours:.2f} degree-hours")
degree_hours += room_degree_hours
return degree_hours
def calculate_total_hours(heating_plan):
"""Calculate total hours covered by the heating plan"""
total_hours = 0
for room_id, schedule in heating_plan.items():
for period in schedule:
start_time = pd.to_datetime(period['start'])
end_time = pd.to_datetime(period['end'])
duration_hours = (end_time - start_time).total_seconds() / 3600
total_hours += duration_hours
return total_hours
def calculate_comfort_hours(heating_plan):
"""Calculate hours at comfort temperature"""
comfort_hours = 0
for room_id, schedule in heating_plan.items():
for period in schedule:
start_time = pd.to_datetime(period['start'])
end_time = pd.to_datetime(period['end'])
duration_hours = (end_time - start_time).total_seconds() / 3600
# Assume comfort temp is > 20°C
if period['setpoint'] > 20:
comfort_hours += duration_hours
return comfort_hours
def create_baseline_heating_plan(config):
"""Create a baseline heating plan (e.g., fixed schedule)"""
logger.info("Creating baseline heating plan (fixed schedule)")
comfort = config["schedule"]["comfort_setpoint"]
setback = config["schedule"].get("setback_setpoint", 16.0)
logger.info(f"Using comfort={comfort}°C and setback={setback}°C for baseline plan")
# Create a weekly schedule with comfort temperature during working hours
baseline = {}
for room_id in [1]: # Adjust for your actual room IDs
room_schedule = []
# For each weekday (0=Monday to 4=Friday)
for day in range(5): # Weekdays only
date = 14 + day # Example: Jan 14-18, 2025 (Mon-Fri)
# Morning heating (8am-6pm)
room_schedule.append({
'start': f"2025-01-{date}T08:00:00Z",
'end': f"2025-01-{date}T18:00:00Z",
'setpoint': comfort
})
logger.debug(f"Room {room_id}, Day {day+1}: Comfort period 08:00-18:00, {comfort}°C")
# Night setback
room_schedule.append({
'start': f"2025-01-{date}T18:00:00Z",
'end': f"2025-01-{date+1}T08:00:00Z",
'setpoint': setback
})
logger.debug(f"Room {room_id}, Day {day+1}: Setback period 18:00-08:00, {setback}°C")
baseline[str(room_id)] = room_schedule
logger.info(f"Room {room_id}: Created baseline with {len(room_schedule)} periods "
f"(10 hours of comfort temperature per weekday)")
return baseline
def main():
start_time = time.time()
logger.info("=== EVALUATION PROCESS STARTED ===")
try:
config = load_config()
artifacts_dir = Path(config["data"]["artifacts_dir"])
logger.info(f"Using artifacts directory: {artifacts_dir}")
# Try to load predictions and their actual labels
try:
logger.info("Attempting to load ground truth and predictions for model evaluation")
# First try loading directly from feature_store which contains the ground truth
feature_path = artifacts_dir / "feature_store.parquet"
preds_path = artifacts_dir / "preds.parquet"
logger.info(f"Loading feature data from {feature_path}")
feature_df = pd.read_parquet(feature_path)
logger.info(f"Loaded {len(feature_df)} feature records")
logger.info(f"Loading predictions from {preds_path}")
preds = pd.read_parquet(preds_path)
logger.info(f"Loaded {len(preds)} prediction records")
# Merge to get ground truth and predictions together
logger.info("Merging predictions with ground truth")
label_col = config["data"]["label_column"]
room_id_col = config["data"]["room_id_column"]
merged_df = pd.merge(
preds,
feature_df[[room_id_col, "timestamp", label_col]],
on=[room_id_col, "timestamp"],
how="left"
)
logger.info(f"Merged data: {len(merged_df)} records")
# Check if we have valid data
valid_count = merged_df[label_col].notna().sum()
logger.info(f"Found {valid_count} records with valid labels for evaluation")
if valid_count > 0:
# Use only rows with valid labels
valid_df = merged_df.dropna(subset=[label_col])
y_true = valid_df[label_col].values
y_pred = valid_df["y_pred"].values
# Log class distribution
true_dist = pd.Series(y_true).value_counts().to_dict()
pred_dist = pd.Series(y_pred).value_counts().to_dict()
logger.info(f"Ground truth distribution: {true_dist}")
logger.info(f"Prediction distribution: {pred_dist}")
# Evaluate occupancy detection
detection_metrics = evaluate_occupancy_detection(y_true, y_pred)
else:
logger.warning("No valid labeled samples found for evaluation")
detection_metrics = {'error': 'No valid labeled data'}
except Exception as e:
logger.error(f"Error during occupancy detection evaluation: {e}", exc_info=True)
detection_metrics = {'error': str(e)}
# Evaluate energy savings
try:
logger.info("Starting energy savings evaluation")
# Create baseline heating plan
baseline_plan = create_baseline_heating_plan(config)
# Check if optimized plan exists, otherwise create one
opt_path = artifacts_dir / "optimized_heating_plan.json"
if not opt_path.exists():
logger.warning("No optimized heating plan found, running optimization first...")
import subprocess
subprocess.run(["python", "pipeline/optimize_energy.py"])
logger.info("Optimization process completed")
# Load optimized heating plan
logger.info(f"Loading optimized heating plan from {opt_path}")
with open(opt_path, "r") as f:
optimized_plan = json.load(f)
logger.info(f"Loaded optimized plan with {len(optimized_plan)} rooms")
# Evaluate energy savings
energy_metrics = evaluate_energy_savings(optimized_plan, baseline_plan, config)
except Exception as e:
logger.error(f"Error during energy savings evaluation: {e}", exc_info=True)
energy_metrics = {'error': str(e)}
# Combine metrics
logger.info("Combining evaluation metrics")
all_metrics = {
'occupancy_detection': detection_metrics,
'energy_savings': energy_metrics
}
# Save evaluation metrics
output_path = artifacts_dir / "evaluation_metrics.json"
logger.info(f"Saving evaluation metrics to {output_path}")
with open(output_path, "w") as f:
json.dump(all_metrics, f, indent=2)
# Calculate execution time
execution_time = time.time() - start_time
logger.info(f"Evaluation completed in {execution_time:.2f} seconds")
# Create summary for console output
summary = {
"execution_time_seconds": round(execution_time, 2),
"out": str(output_path)
}
if 'f1_score' in detection_metrics:
summary["occupancy_detection"] = {
"f1_score": round(detection_metrics['f1_score'], 4),
"accuracy": round(detection_metrics['accuracy'], 4),
"samples_evaluated": len(y_true) if 'y_true' in locals() else 0
}
if 'savings_percent' in energy_metrics:
summary["energy_savings"] = {
"savings_percent": round(energy_metrics['savings_percent'], 2),
"baseline_degree_hours": round(energy_metrics['baseline_degree_hours'], 2),
"optimized_degree_hours": round(energy_metrics['optimized_degree_hours'], 2)
}
print(json.dumps(summary, indent=2))
logger.info("=== EVALUATION PROCESS COMPLETED SUCCESSFULLY ===")
except Exception as e:
logger.error(f"Evaluation process failed: {e}", exc_info=True)
print(json.dumps({"error": str(e)}, indent=2))
raise
if __name__ == "__main__":
main()
\ No newline at end of file
pipeline/features.py 0 → 100644
View file @ 6c7b465b
#!/usr/bin/env python3
import argparse, json, logging
from pathlib import Path
import pandas as pd
import numpy as np
import yaml
import time
# Set up logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
handlers=[
logging.FileHandler("feature_engineering.log"),
logging.StreamHandler()
]
)
logger = logging.getLogger("feature_engineering")
def load_config():
logger.info("Loading configuration from config.yaml")
with open("config.yaml", "r") as f:
cfg = yaml.safe_load(f)
logger.info(f"Configuration loaded: sensors={cfg['data']['sensors']}, rolling_minutes={cfg['features']['rolling_minutes']}")
return cfg
def infer_window_sizes_per_room(df, minutes_list, room_id_col):
logger.info(f"Inferring window sizes for {len(df[room_id_col].unique())} unique rooms")
out = {}
for room, g in df.groupby(room_id_col, observed=True, sort=False):
g = g.sort_values("timestamp")
if len(g) < 3:
logger.warning(f"Room {room} has too few samples ({len(g)}) - skipping window inference")
continue
deltas = g["timestamp"].diff().dropna().dt.total_seconds() / 60.0
med = np.median(deltas) if len(deltas) else 5.0
out[room] = {m: max(1, int(round(m / max(med, 1e-6)))) for m in minutes_list}
logger.debug(f"Room {room}: median interval={med:.2f}min, window sizes={out[room]}")
# Log a sample of inferred windows
sample_rooms = list(out.keys())[:3] if out else []
logger.info(f"Window size inference complete. Sample of {len(sample_rooms)} rooms: {[out.get(r) for r in sample_rooms]}")
return out
def add_features(df, cfg):
start_time = time.time()
logger.info(f"Starting feature engineering on {len(df)} rows")
sensors = cfg["data"]["sensors"]
minutes_list = cfg["features"]["rolling_minutes"]
infer = cfg["features"].get("infer_windows", True)
room_id_col = cfg["data"]["room_id_column"]
# Log available sensors
available_sensors = [s for s in sensors if s in df.columns]
missing_sensors = [s for s in sensors if s not in df.columns]
logger.info(f"Available sensors: {available_sensors}")
if missing_sensors:
logger.warning(f"Missing sensors: {missing_sensors}")
# Add missing flags
for s in sensors:
if s in df.columns:
missing_count = df[s].isna().sum()
df[f"is_missing_{s}"] = df[s].isna().astype(int)
logger.info(f"Sensor {s}: {missing_count} missing values ({missing_count/len(df)*100:.1f}%)")
else:
df[s] = np.nan
df[f"is_missing_{s}"] = 1
logger.warning(f"Created placeholder for missing sensor {s}")
# Time features
ts = pd.to_datetime(df["timestamp"], utc=True)
df["hour"] = ts.dt.hour
df["dow"] = ts.dt.dayofweek
df["is_weekend"] = (df["dow"] >= 5).astype(int)
df["week_of_year"] = ts.dt.isocalendar().week.astype(int)
logger.info(f"Added time features: hour, dow, is_weekend, week_of_year")
logger.info(f"Data timespan: {ts.min()} to {ts.max()}")
# Window sizes
if infer:
logger.info("Using inferred window sizes per room")
win_map = infer_window_sizes_per_room(df, minutes_list, room_id_col)
default_windows = {m: max(1, int(round(m/5))) for m in minutes_list}
def get_windows(room):
return win_map.get(room, default_windows)
else:
logger.info("Using fixed window sizes")
fixed = {m: max(1, int(round(m/5))) for m in minutes_list}
def get_windows(room): return fixed
# Rolling features per room
logger.info(f"Creating rolling window features for {df[room_id_col].nunique()} rooms")
feats = []
room_counts = {}
for room, g in df.groupby(room_id_col, observed=True, sort=False):
g = g.sort_values("timestamp")
wins = get_windows(room)
room_counts[room] = len(g)
logger.debug(f"Room {room}: {len(g)} samples, windows={wins}")
feature_counts = {
"co2_features": 0,
"temp_features": 0,
"movement_features": 0
}
for m, w in wins.items():
if "co2" in g.columns:
g[f"co2_mean_{m}"] = g["co2"].rolling(w, min_periods=1).mean()
g[f"co2_slope_{m}"] = g["co2"].diff(w) / max(w,1)
feature_counts["co2_features"] += 2
if "temperature" in g.columns:
g[f"temp_mean_{m}"] = g["temperature"].rolling(w, min_periods=1).mean()
g[f"temp_slope_{m}"] = g["temperature"].diff(w) / max(w,1)
feature_counts["temp_features"] += 2
if "movement" in g.columns:
g[f"mov_sum_{m}"] = g["movement"].rolling(w, min_periods=1).sum()
feature_counts["movement_features"] += 1
logger.debug(f"Room {room} features: {feature_counts}")
feats.append(g)
# Log room sample counts
logger.info(f"Room sample counts: min={min(room_counts.values()) if room_counts else 0}, "
f"max={max(room_counts.values()) if room_counts else 0}, "
f"avg={sum(room_counts.values())/len(room_counts) if room_counts else 0:.1f}")
df2 = pd.concat(feats, axis=0).sort_values([room_id_col,"timestamp"]).reset_index(drop=True)
logger.info(f"Combined data shape after rolling features: {df2.shape}")
# Add additional features
logger.info("Adding additional engineered features")
initial_cols = df2.shape[1]
df2 = add_additional_features(df2)
added_cols = df2.shape[1] - initial_cols
logger.info(f"Added {added_cols} additional features")
# Summary statistics
end_time = time.time()
logger.info(f"Feature engineering completed in {end_time - start_time:.2f} seconds")
logger.info(f"Final feature set: {df2.shape[0]} rows, {df2.shape[1]} columns")
return df2
def add_additional_features(df):
# Time of day features (circular encoding for hour)
df['hour_sin'] = np.sin(df['hour'] * (2 * np.pi / 24))
df['hour_cos'] = np.cos(df['hour'] * (2 * np.pi / 24))
logger.info("Added circular time encoding: hour_sin, hour_cos")
# Day of week features
df['is_workday'] = ((df['dow'] < 5) & (df['hour'] >= 8) & (df['hour'] <= 18)).astype(int)
workday_pct = df['is_workday'].mean() * 100
logger.info(f"Added is_workday feature: {workday_pct:.1f}% of samples during work hours")
# Activity detection features
if "movement" in df.columns and "co2" in df.columns:
df['combined_activity'] = df['movement'] + (df['co2'] > 500).astype(int)
activity_pct = (df['combined_activity'] > 0).mean() * 100
logger.info(f"Added combined_activity feature: {activity_pct:.1f}% of samples show activity")
else:
logger.warning("Couldn't create combined_activity (missing movement or co2)")
return df
def main():
logger.info("=== FEATURE ENGINEERING PROCESS STARTED ===")
parser = argparse.ArgumentParser()
parser.add_argument("--phase", type=str, choices=["train", "predict"], default="train",
help="Whether this is for training or prediction phase")
parser.add_argument("--input", type=str, default=None,
help="Custom input filename (default: base_[phase].parquet)")
parser.add_argument("--output", type=str, default=None,
help="Custom output filename (default: feature_store_[phase].parquet)")
args = parser.parse_args()
try:
cfg = load_config()
# Load base data based on phase
if args.input:
base_path = Path(cfg["data"]["artifacts_dir"]) / args.input
else:
base_path = Path(cfg["data"]["artifacts_dir"]) / f"base_{args.phase}.parquet"
# Fall back to the original path if phase-specific file doesn't exist
if not base_path.exists():
fallback_path = Path(cfg["data"]["artifacts_dir"]) / "base.parquet"
if fallback_path.exists():
logger.warning(f"{base_path} not found, falling back to {fallback_path}")
base_path = fallback_path
logger.info(f"Running in {args.phase} phase")
logger.info(f"Loading base data from {base_path}")
base = pd.read_parquet(base_path)
logger.info(f"Loaded base data: {base.shape[0]} rows, {base.shape[1]} columns")
# Process features
logger.info("Starting feature engineering")
feat = add_features(base, cfg)
# Save results
if args.output:
out = Path(cfg["data"]["artifacts_dir"]) / args.output
else:
out = Path(cfg["data"]["artifacts_dir"]) / f"feature_store_{args.phase}.parquet"
logger.info(f"Saving feature store to {out}")
feat.to_parquet(out, index=False)
# Feature statistics
num_features = feat.shape[1]
numeric_cols = feat.select_dtypes(include=[np.number]).columns.tolist()
memory_usage = feat.memory_usage(deep=True).sum() / (1024 * 1024) # MB
logger.info(f"Feature store statistics:")
logger.info(f" - Rows: {feat.shape[0]}")
logger.info(f" - Total features: {num_features}")
logger.info(f" - Numeric features: {len(numeric_cols)}")
logger.info(f" - Memory usage: {memory_usage:.2f} MB")
print(json.dumps({
"rows": len(feat),
"columns": num_features,
"memory_mb": round(memory_usage, 2),
"out": str(out)
}, indent=2))
logger.info("=== FEATURE ENGINEERING COMPLETED SUCCESSFULLY ===")
except Exception as e:
logger.error(f"Feature engineering failed: {e}", exc_info=True)
raise
if __name__ == "__main__":
main()
\ No newline at end of file
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment

Dies ist die Gitlab-Instanz des Transferportals der Hochschule für Technik Stuttgart. Hier geht es zurück zum Portal