Commit fba97647 authored by Abbassy's avatar Abbassy
Browse files

Update docs and finalize DTA feedback display logic

parent 9df5705b
# Moodle Assign Feedback Plugin - DTA (Dockerized Test Agent) Dockerized Test Agent Feedback (assignfeedback_dta)
==================================================
## Overview
Purpose
The assignfeedback_dta plugin is part of the MoDTA (Moodle Dockerized Test Agent) system. It automatically calculates and displays grades for student submissions tested using the Dockerized Test Agent (DTA) system. When students submit code through the DTA submission plugin, the DTA backend runs automated tests and sends results back to Moodle. This feedback plugin takes those test results and automatically calculates a grade, displaying it in a dedicated feedback column. -------
- Provides an informational column “Dockerized Test Agent Feedback” in the grading table.
The MoDTA system comprises two key components: - Shows only the suggested grade percentage from the DTA backend; feedback block stays hidden.
- Optional auto-grading: when “Grade automatically” is enabled, the plugin applies grades; otherwise it only displays the suggestion.
• The MoDTA Moodle submission plugin (`assignsubmission_dta`), designed as an assignment submission tool.
• The MoDTA Moodle feedback plugin (`assignfeedback_dta`), designed to automatically calculate and display grades based on test results. Installation
• The DTA backend web service, which interfaces with the Moodle plugins through REST and employs the JSON file format for communication. ------------
- Unzip into `{moodleroot}/mod/assign/feedback/dta`.
## Goals - Visit Site administration → Notifications to complete installation.
- Ensure the companion submission plugin `assignsubmission_dta` is installed and the DTA backend is reachable (HTTP allowlist/ports configured in Moodle security settings).
MoDTA empowers teachers to provide programming Moodle assignments for their students, which are automatically corrected upon hand-in using teacher-provided tests. The feedback plugin extends this functionality by automatically calculating grades based on test results, providing immediate feedback to both students and teachers.
Configuration
The feedback plugin optionally supports automatic grade calculation, allowing teachers to: -------------
- Automatically calculate grades based on test results - Settings → “Grade automatically”: default is **disabled**.
- Display grades in a dedicated feedback column - Disabled: only the column shows the suggestion percent (info-only); no grades are applied.
- Optionally apply grades to the "Final grade" column for automatic grading - Enabled: the column still shows the suggestion; the plugin also writes the calculated grade.
- Or keep "Final grade" empty for manual grade entry - Feedback block is suppressed (`is_empty` returns true); all user-facing text is English and sourced from `lang/en/assignfeedback_dta.php`.
## What is this plugin? Data
----
The assignfeedback_dta plugin automatically calculates and displays grades for student submissions tested using the Dockerized Test Agent (DTA) system. When students submit code, the DTA backend runs automated tests and sends results back to Moodle. This plugin takes those test results and automatically calculates a grade, displaying it in a dedicated feedback column. - No own database tables. Reads DTA results from `assignsubmission_dta` tables (`assignsubmission_dta_summary` / `assignsubmission_dta_result`).
## How does it work? Behavior notes (current)
------------------------
1. **Student submits code** - Uploads Java project files through the DTA submission plugin (`assignsubmission_dta`) - Suggested grade is calculated with 50/50 weighting (tests/competencies) as provided by the backend; “hard 0” for compilation errors removed.
2. **Backend tests the code** - DTA backend automatically runs test cases - Column renders percent when DTA data exists; nothing else is shown elsewhere.
3. **Results come back** - Backend sends test results to Moodle, stored in `assignsubmission_dta_summary` and `assignsubmission_dta_result` tables
4. **Plugin calculates grade** - This feedback plugin reads the test results and calculates a grade based on: Packaging
- Test success rate (passed tests / total tests) ---------
- Compilation errors (more than 1 error results in 0%) - Release ZIP excludes VCS and assets: `.git`, `.assets`.
- Optional competency scores (if available)
5. **Grade appears in feedback column** - Shows "80.00 / 100.00" in "Dockerized Test Agent Feedback" column
## The "Grade automatically" Button
In the assignment settings, you'll find a checkbox labeled **"Grade automatically"** in the Feedback types section. This button controls how the plugin handles grade calculation.
**When checked (Automatic Grading ON):**
- The plugin automatically calculates and displays grades based on test results
- Grades appear immediately in the DTA Feedback column after backend processing
- Grades are also automatically applied to the "Final grade" column
- You can still manually adjust grades in the "Final grade" column if needed
**When unchecked (Automatic Grading OFF):**
- The plugin shows test results and calculates a suggested grade
- The grade is displayed only in the DTA Feedback column
- No grade is automatically applied to "Final grade"
- You review each submission and manually enter grades in the "Final grade" column
## Scenarios
### Scenario 1: Automatic Grading Enabled
**Setup:** You enable the plugin and check "Grade automatically" in assignment settings.
**Student submits:** Code passes 8 out of 10 tests, no compilation errors.
**Result:** Plugin automatically calculates 80% and displays "80.00 / 100.00" in DTA Feedback column. The grade is also automatically applied to the "Final grade" column.
**Use case:** Perfect for large classes where you want quick automated grading with the option to manually adjust exceptional cases.
### Scenario 2: Automatic Grading Disabled
**Setup:** You enable the plugin but leave "Grade automatically" unchecked.
**Student submits:** Code passes 7 out of 10 tests.
**Result:** Plugin calculates suggested grade of 70% and displays it in DTA Feedback column. "Final grade" column remains empty. You review the code, notice good documentation and effort, and manually enter 85% in Final grade column.
**Use case:** Ideal when code quality, documentation, and subjective factors matter alongside test results.
### Scenario 3: Compilation Errors
**Setup:** Automatic grading enabled.
**Student submits:** Code has 2 compilation errors.
**Result:** Plugin automatically sets grade to 0% and displays "0.00 / 100.00". This ensures students fix critical errors first.
**Use case:** Helps identify submissions that need immediate attention before test cases can even run.
### Scenario 4: Perfect Submission
**Setup:** Automatic grading enabled.
**Student submits:** All 10 tests pass, no errors.
**Result:** Plugin calculates 100% and displays "100.00 / 100.00" immediately. Grade is automatically applied to "Final grade".
**Use case:** Instant feedback for students who submit correct code, encouraging early submissions.
### Scenario 5: Switching Modes Mid-Assignment
**Initial:** Automatic grading enabled, students submitting and getting grades automatically.
**Change:** You uncheck "Grade automatically" after reviewing initial submissions.
**Result:** New submissions show suggested grades only in DTA Feedback column. "Final grade" remains empty. You manually review and enter final grades. Existing automatic grades remain visible for reference.
**Use case:** Flexibility to adapt grading strategy based on assignment progress and student performance patterns.
## Key features
-**Automatic grade calculation** based on test results
-**Grade display in DTA Feedback column** - Always shows calculated grade
-**Optional automatic grade application** - "Final grade" can be automatically set or left empty
-**Checkbox saving works** - "Grade automatically" setting is properly saved
-**Batch testing support** - Works with batch testing functionality from submission plugin
-**Competency-based grading** - Supports competency scores if available from backend
## Installation & Configuration
### Prerequisites
The `assignsubmission_dta` plugin must be installed first, as the feedback plugin depends on it for test results. The feedback plugin reads from the database tables created by the submission plugin:
- `mdl_assignsubmission_dta_summary` - Summary of test results
- `mdl_assignsubmission_dta_result` - Individual test case results
### Quick installation
1. Download `assignfeedback_dta.zip`
2. Go to **Site administration → Plugins → Install plugins**
3. Upload the ZIP file as shown in Fig. 1. ![Fig. 1: Plugin Installer](.assets/install_plugin.png)
4. Follow the installation wizard
### Manual installation
1. Extract ZIP and copy `dta` folder to `/mod/assign/feedback/dta/`
2. Set permissions: `chown -R www-data:www-data dta/`
3. Go to **Site administration → Notifications** to upgrade
4. Purge caches: **Site administration → Development → Purge all caches**
### Configuration
After installation, the plugin requires no additional configuration. It automatically reads test results from the submission plugin's database tables. However, ensure that:
- The DTA backend is properly configured (see submission plugin documentation)
- The submission plugin is installed and working
- HTTP Security settings allow communication with the backend (if needed)
## Usage
With the MoDTA feedback plugin installed and the submission plugin configured:
### Teacher Setup
When creating or editing an assignment, a teacher can enable the DTA Feedback plugin in the **Feedback types** section as shown in Fig. 2. ![Fig. 2: Assignment Settings - Feedback Types](.assets/assignment_settings.png)
1. **Enable in assignment:**
- Edit assignment → **Feedback types** section
- Check **"Dockerized Test Agent Feedback"**
- Check **"Grade automatically"** (optional - controls automatic vs manual grading)
2. **Students submit code** via DTA submission plugin
3. **View grades:**
- Open assignment grading page as shown in Fig. 3. ![Fig. 3: Submissions Table with DTA Feedback](.assets/submissions_table.png)
- See grades in **"Dockerized Test Agent Feedback"** column
- Format: "85.50 / 100.00"
- If "Grade automatically" is enabled, grades also appear in **"Final grade"** column
- If disabled, **"Final grade"** column remains empty for manual entry
### Student View
Students see the calculated grade in the DTA Feedback column when viewing their submission status as shown in Fig. 4. ![Fig. 4: Student Submission View](.assets/student_submission_view.png) The grade is displayed in the format "XX.XX / 100.00" showing both the calculated grade and maximum possible grade.
## Grade calculation
The plugin uses a sophisticated calculation method:
- **Test Score:** Passed tests / Total tests (e.g., 8/10 = 80%)
- **With competencies:** (Test Score × 0.8) + (Competency Score × 0.2)
- **Without competencies:** Test Score × 1.0
- **Compilation errors (>1):** Grade = 0% (regardless of test results)
### Detailed Calculation Process
1. **Read test results** from `assignsubmission_dta_summary` and `assignsubmission_dta_result` tables
2. **Count successful tests** vs total tests
3. **Check compilation errors** - if more than 1, set grade to 0%
4. **Calculate test score** as percentage of passed tests
5. **Apply competency weighting** if competency data is available (80% test score, 20% competency score)
6. **Display grade** in DTA Feedback column
7. **Optionally apply grade** to "Final grade" if "Grade automatically" is enabled
## Technical Details
### Plugin Structure
The plugin follows Moodle's standard feedback plugin structure:
```
assignfeedback_dta/
├── version.php # Version information
├── locallib.php # Main plugin logic
├── lang/
│ └── en/
│ └── assignfeedback_dta.php # Language strings
├── classes/
│ └── local/
│ └── grading_service.php # Grade calculation engine
├── batchtestform.php # Batch test form
├── testuntestedform.php # Test untested form
└── styles.css # Stylesheet
```
### Database Tables Used
The plugin reads from tables created by the submission plugin:
- `mdl_assignsubmission_dta_summary` - Summary of test results (1:1 relation to submission)
- `mdl_assignsubmission_dta_result` - Individual test case results (1:n relation to summary)
### Configuration Storage
Plugin settings are stored in `mdl_assign_plugin_config`:
- `assignfeedback_dta_enabled` - Whether the plugin is enabled (0 or 1)
- `assignfeedback_dta_gradeautomatically` - Whether automatic grading is enabled (0 or 1)
### Key Methods
- `view_summary()` - Displays grade in feedback column, optionally applies grade
- `is_grade_automatically_enabled()` - Checks if automatic grading is enabled
- `get_settings()` / `save_settings()` - Manages plugin settings
- `grading_service->calculate_summary()` - Calculates grade from test results
- `grading_service->apply_grade()` - Applies calculated grade to Moodle gradebook
## Requirements
- Moodle 3.1 or higher
- PHP 7.4 or higher
- **`assignsubmission_dta` plugin must be installed first**
- DTA backend web service (configured via submission plugin)
## Troubleshooting
**Checkbox doesn't stay checked:** Fixed in version 1.0.0 - ensure you're using the latest version.
**No grade appears:**
- Check if "Grade automatically" is enabled (if you want automatic application)
- Wait for backend to process submission
- Ensure submission plugin is installed and working
- Verify test results exist in database tables
**Grade in wrong column:**
- If "Grade automatically" is enabled: Grade appears in both DTA Feedback and "Final grade"
- If disabled: Grade appears only in DTA Feedback column
**Grade not calculated:**
- Ensure submission plugin has processed the submission
- Check that test results exist in `assignsubmission_dta_summary` table
- Verify backend is working correctly
## Version
- **Version:** 1.0.0
- **Status:** Stable
- **License:** GNU GPL v3 or later
- **Moodle Compatibility:** 3.1+
## Related Documentation
For more information about the MoDTA system:
- Submission Plugin: See `assignsubmission_dta` plugin documentation
- Backend: https://transfer.hft-stuttgart.de/gitlab/HFTSoftwareProject/dtabackend
- Test Runner: https://transfer.hft-stuttgart.de/gitlab/HFTSoftwareProject/dtatestrunner
...@@ -91,15 +91,6 @@ class grading_service { ...@@ -91,15 +91,6 @@ class grading_service {
$summary['compilationerrors'] = (int)$DB->count_records_select(assign_feedback_dta::TABLE_DTA_RESULTS, $summary['compilationerrors'] = (int)$DB->count_records_select(assign_feedback_dta::TABLE_DTA_RESULTS,
'assignment_id = :assignment_id AND submission_id = :submission_id AND state = 3', $params); 'assignment_id = :assignment_id AND submission_id = :submission_id AND state = 3', $params);
// Only set grade to 0 if there are more than 1 compilation errors.
if ($summary['compilationerrors'] > 1) {
$summary['ratio'] = 0.0;
$summary['test_score'] = 0.0;
$summary['finalgrade'] = 0.0;
$summary['gradable'] = true;
return $summary;
}
// Calculate test score (0-100%). // Calculate test score (0-100%).
if ($summary['totaltests'] > 0) { if ($summary['totaltests'] > 0) {
$summary['test_score'] = ($summary['successcount'] / $summary['totaltests']) * 100.0; $summary['test_score'] = ($summary['successcount'] / $summary['totaltests']) * 100.0;
...@@ -113,10 +104,10 @@ class grading_service { ...@@ -113,10 +104,10 @@ class grading_service {
$summary['competency_percent'] = $this->calculate_competency_percent($dtasubmission); $summary['competency_percent'] = $this->calculate_competency_percent($dtasubmission);
$summary['has_competencies'] = ($summary['competency_percent'] > 0); $summary['has_competencies'] = ($summary['competency_percent'] > 0);
// Calculate combined final score: 80% test score + 20% competency score. // Calculate combined final score: give competencies 50% weight when present.
// If no competencies available, use test score only (weight 1.0). // If no competencies available, use test score only (weight 1.0).
$test_weight = $summary['has_competencies'] ? 0.8 : 1.0; $test_weight = $summary['has_competencies'] ? 0.5 : 1.0;
$competency_weight = $summary['has_competencies'] ? 0.2 : 0.0; $competency_weight = $summary['has_competencies'] ? 0.5 : 0.0;
$final_percent = ($test_weight * $summary['test_score']) + ($competency_weight * $summary['competency_percent']); $final_percent = ($test_weight * $summary['test_score']) + ($competency_weight * $summary['competency_percent']);
......
...@@ -40,5 +40,6 @@ $string["dta_tables_missing"] = "DTA submission data tables are not available."; ...@@ -40,5 +40,6 @@ $string["dta_tables_missing"] = "DTA submission data tables are not available.";
$string["grade_automatically"] = "Grade automatically"; $string["grade_automatically"] = "Grade automatically";
$string["grade_automatically_help"] = "If enabled, the plugin will automatically calculate and apply grades based on test results. If disabled, the plugin will only display test results as feedback, and the teacher must manually assign grades."; $string["grade_automatically_help"] = "If enabled, the plugin will automatically calculate and apply grades based on test results. If disabled, the plugin will only display test results as feedback, and the teacher must manually assign grades.";
$string["suggested_grade"] = "Suggested grade"; $string["suggested_grade"] = "Suggested grade";
$string["gradingsuggestion"] = "Grading suggestion";
$string["status_pass"] = "PASS"; $string["status_pass"] = "PASS";
$string["status_fail"] = "FAIL"; $string["status_fail"] = "FAIL";
...@@ -82,7 +82,7 @@ class assign_feedback_dta extends assign_feedback_plugin { ...@@ -82,7 +82,7 @@ class assign_feedback_dta extends assign_feedback_plugin {
* @return bool * @return bool
*/ */
public function is_enabled() { public function is_enabled() {
return false; // Hide the feedback column completely return true; // Allow feedback column to be shown
} }
/** /**
...@@ -97,7 +97,7 @@ class assign_feedback_dta extends assign_feedback_plugin { ...@@ -97,7 +97,7 @@ class assign_feedback_dta extends assign_feedback_plugin {
* @return bool True if feedback is empty, false if feedback exists * @return bool True if feedback is empty, false if feedback exists
*/ */
public function is_empty(stdClass $grade) { public function is_empty(stdClass $grade) {
// Always return true to hide the feedback column completely // Hide the feedback block entirely; column output is handled via view_summary().
return true; return true;
} }
...@@ -131,7 +131,8 @@ class assign_feedback_dta extends assign_feedback_plugin { ...@@ -131,7 +131,8 @@ class assign_feedback_dta extends assign_feedback_plugin {
null, array(0, 1)); null, array(0, 1));
$mform->addHelpButton('assignfeedback_dta_gradeautomatically', 'grade_automatically', self::COMPONENT_NAME); $mform->addHelpButton('assignfeedback_dta_gradeautomatically', 'grade_automatically', self::COMPONENT_NAME);
// Set default value from saved config, or 1 if not set yet. // Set default value from saved config, or 1 if not set yet.
$defaultvalue = $this->get_config('gradeautomatically', 1); // Default to disabled (0) if not set.
$defaultvalue = $this->get_config('gradeautomatically', 0);
$mform->setDefault('assignfeedback_dta_gradeautomatically', $defaultvalue); $mform->setDefault('assignfeedback_dta_gradeautomatically', $defaultvalue);
$mform->setType('assignfeedback_dta_gradeautomatically', PARAM_BOOL); $mform->setType('assignfeedback_dta_gradeautomatically', PARAM_BOOL);
...@@ -148,26 +149,26 @@ class assign_feedback_dta extends assign_feedback_plugin { ...@@ -148,26 +149,26 @@ class assign_feedback_dta extends assign_feedback_plugin {
* @return bool * @return bool
*/ */
public function save_settings(stdClass $data) { public function save_settings(stdClass $data) {
// Check if the checkbox value exists in the form data. // Persist checkbox robustly:
// advcheckbox returns the value directly (0 or 1), not as an array. // - If plugin is enabled but the field is missing (e.g. hideIf), default to 0 (disabled).
if (isset($data->assignfeedback_dta_gradeautomatically)) { // - If plugin is disabled, force 0.
// Get the old value before saving. $oldvalue = $this->get_config('gradeautomatically', 0);
$oldvalue = $this->get_config('gradeautomatically', 1); $enabled = !empty($data->assignfeedback_dta_enabled);
// Value is set - save it (0 or 1) if (!$enabled) {
$value = 0;
} else if (property_exists($data, 'assignfeedback_dta_gradeautomatically')) {
$value = !empty($data->assignfeedback_dta_gradeautomatically) ? 1 : 0; $value = !empty($data->assignfeedback_dta_gradeautomatically) ? 1 : 0;
} else {
$value = 0; // enabled but no field posted -> keep disabled by default
}
$this->set_config('gradeautomatically', $value); $this->set_config('gradeautomatically', $value);
// If the setting was changed from enabled (1) to disabled (0), // If the setting was changed from enabled (1) to disabled (0), clear auto grades.
// remove all automatically set grades for this assignment.
if ($oldvalue == 1 && $value == 0) { if ($oldvalue == 1 && $value == 0) {
$this->clear_automatic_grades(); $this->clear_automatic_grades();
} }
} else {
// Checkbox was hidden (because DTA was not enabled), so it won't be in $data.
// In this case, we don't change the existing value - it stays as is.
// This is correct behavior: if DTA is disabled, we can't change this setting anyway.
}
return true; return true;
} }
...@@ -250,7 +251,8 @@ class assign_feedback_dta extends assign_feedback_plugin { ...@@ -250,7 +251,8 @@ class assign_feedback_dta extends assign_feedback_plugin {
* @return bool True if automatic grading is enabled * @return bool True if automatic grading is enabled
*/ */
public function is_grade_automatically_enabled() { public function is_grade_automatically_enabled() {
return (bool)$this->get_config('gradeautomatically', 1); // Default to enabled // Default to disabled unless explicitly enabled in settings.
return (bool)$this->get_config('gradeautomatically', 0);
} }
/** /**
...@@ -618,7 +620,7 @@ class assign_feedback_dta extends assign_feedback_plugin { ...@@ -618,7 +620,7 @@ class assign_feedback_dta extends assign_feedback_plugin {
* @return string HTML output * @return string HTML output
*/ */
public function view(stdClass $grade) { public function view(stdClass $grade) {
// Return empty string - no feedback button needed // Render nothing in the feedback block; column output is handled in view_summary().
return ''; return '';
} }
...@@ -630,10 +632,10 @@ class assign_feedback_dta extends assign_feedback_plugin { ...@@ -630,10 +632,10 @@ class assign_feedback_dta extends assign_feedback_plugin {
* @return string * @return string
*/ */
public function view_summary(stdClass $grade, &$showviewlink) { public function view_summary(stdClass $grade, &$showviewlink) {
debugging('MOJEC view_summary() wurde ausgeführt', DEBUG_DEVELOPER);
$showviewlink = false; $showviewlink = false;
$gradesummary = null;
// Get submission for grading calculations // Get submission for grading calculations
$submission = null; $submission = null;
$dtasubmission = null; $dtasubmission = null;
...@@ -658,7 +660,14 @@ class assign_feedback_dta extends assign_feedback_plugin { ...@@ -658,7 +660,14 @@ class assign_feedback_dta extends assign_feedback_plugin {
} }
} }
// Return empty string - no feedback button needed // Summary column: info-only percent shown regardless of auto-grading setting.
// (Auto-apply still gated above; this column is informational.)
if (!empty($gradesummary) && !empty($gradesummary['finalgrade']) && !empty($gradesummary['maxgrade']) && $gradesummary['maxgrade'] > 0) {
$percent = ($gradesummary['finalgrade'] / $gradesummary['maxgrade'] * 100.0);
return format_float($percent, 1) . '%';
}
// No data to show
return ''; return '';
} }
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment