Update snapshot

README.md

+# The code accompaning `Application of pre-trained transformers to estimate the difficulty of closed questions using question and choice text`
+
+
+This contains the code that was used to produce the results in the paper.
+It contains a number of subdirectories that are somewhat separate, somewhat dependent on each other.
+
+Most of the subdirectories contain their own `README.md`.
+
+One thing that is common to almost all directories is their general Python toolchain. 
+
+## Common Python toolchain
+
+The project uses Python 3.10+ (I used my system python, which is at `3.11.3`, i did not knowingly use any python 3.11 features and much of the code was also tested with `3.10`) and `python-poetry` (`1.5.1`) was used for other python package dependencies and tools.
+
+If your system has an older python version and cant easily be updated (Ubuntu/Debian, looking at you) perhaps look at [`pyenv`](https://github.com/pyenv/pyenv).
+
+To install poetry just follow [the guide](https://python-poetry.org/docs/#installation).
+
+### Poetry
+
+Poetry is generally similar to `npm`.
+
+Generally all you should need to know, if you dont want to change anything with the dependencies, is to run `poetry install` in the directories of the subprojects (they contain a `pyproject.toml` file) and then execute python files with `poetry run python <whateverfile.py>`.
+
+Poetry manages dependency versions in 2 files, one being dev-facing: `pyproject.toml` contains the versions as defined(ish) by the developer. Its following the rules for [semantic versioning](https://semver.org/) and their specifiers.
+You may add a dependency (also to that file) by being in the project directory and executing `poetry add <pypi.org name>`.
+
+There is another file. The `poetry.lock` file should be version controlled too, though not be edited by you.
+It contains the concrete versions, that were actually installed by poetry.
+This is intended to solve the "but it worked on my pc" issue plagueing tools like `pip`.
+
+Poetry installs dependencies in a [python virtual environment](https://docs.python.org/3/library/venv.html#module-venv), that it manages for you.
+If you want access to that virtual environment, you can see information about it with `poetry env info`.
+This may be helpful with tools such as `jupyter`, or to get IDE autocomplete.
+
+E.g. with my Visual Studio Code (vscode) that has the Python extension installed, all I have to do to get completions, documentation on mouse hover and other niceties for my dependencies (numpy, tensorflow etc) is to open the project in a vscode instance, then click on the number in the lower right corner of the editor (after "Python"), then click `+ Enter interpreter path`, and then copy paste the `Executable` path from `poetry env info` into there.
+
+These instructions are all untested, and don't contain any pictures, so good luck to these trying to do expecially the last part here without some preexisting knowledge, but frankly I think I've done much more here already than other people would.
+
+Don't get intimidated by all the big words thrown around here.
+You dont need to understand most of them to begin with, and eventually you'll develop an understanding for them.
+ChatGPT can probably help your understanding for these, haven't tried if for something like that!
+
+### Other tools & choices for Python
+
+The python code contains a lot of typing.
+This is mostly informal in nature (for both me as well as the IDE), and often doesnt pass typechecking.
+Sadly there doesnt really seem to be any standard for python typechecking anyways.
+
+That said, some of the larger projects were occasionally checked with `mypy` (a typechecker) and ruff (a linter), and some select offenders were solved.
+
+What wasnt solved on purpose is a lot of errors about values potentially being None (which is also offered in a somewhat unhelpful way by mypy btw).
+Python doesnt have any syntactic sugar to makes these easier to handle (such as [Optional chaining](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Optional_chaining) or chaining non-null assertion).
+
+Also usually the "wrong" code will lead to the same results a non-null assertion would lead to in other languages, an exception saying "that thing is None!.
+Sadly I'm not aware of a way to tell mypy that this is fine, so I'll ignore these kinds of errors.
+
+## Overview over the subdirectories
+
+* [common_py](./common_py/) contains some utilities that were used across the different projects
+* [raw_data](./raw_data/) contains a bunch of gitignores ingnoring non pubpic data and READMEs explaining the data that should be there. This way other subdirectories can reference these by path.
+* [enem_aggregate](./enem_aggregate/) contains the code that converted the raw data from enem, alongside the infos from Marinho, into a much simpler unified CSV containing the aggregated interactions.
+* [moodle_extract](./moodle_extract/) contains the code that parsed the moodle question xml and bundled it together with the experimental data, in form of a CSV.
+* [qde_model_code](./qde_model_code/) contains the heart of this thing, the code that trained the ml model on the data generated by the other projects.
+* [data_explore](./data_explore) (may not be added) contains some (not all) of my python scripts that were used to get insights into data and often to plot the data.
+
+### Configuration
+
+None of the projects here has a defined CLI.
+Configuration is done in the source files themselves.
+I did not want to add a dependency or the complexity of having a CLI (that doesnt suck), and frankly with how fast the underlying code and with that the API changed, it wouldve been a lot of work that would just have hindered experimentation.
+
+An effort was made to have the regularly changed configuration of each potential entrypoint in few places.
+
+### Why a Monorepo, why separate projects
+
+This was created as a Monorepo from the individual projects.
+These were usually already (local) git repos.
+Having these separate was a good thing, I had no global structure and a lot of copy paste in the beginning while I figured out the structure.
+And the projects history may also contain data that was not meant to be released.
+Doing tasks with a lot of data wrangling doesn't easily convert to the things established in software repositories.
+
+I considered implementing these as git submodules, but this was not done mostly because of complexity.
+
+Git Subrepositories don't come without gotchas, the [official documentation](https://git-scm.com/book/en/v2/Git-Tools-Submodules) as an entire subsection about these, and theres some others that may be more problematic.
+
+This project will already be complex to execute.
+I dont have to make them learn how to handle subrepositories too.
+
+Add 
+* the mentioned potential leaking of data, which is workintensive to solve
+* while this probably already contains a lot of not so glamarous code, other stuff is in the past and I'd prefer it not being available to the worldwide web
+* submodules make multiple remote repositories difficult from my understanding (as they refer to the submodules per remote url), and I consider cloning this in a private repo in case my HFT gitlab repo gets deleted
\ No newline at end of file

common_py/.gitignore

+__pycache__
\ No newline at end of file

common_py/README.md

+# Common Python tools
+
+I've used multiple Python packages as part of this thesis.
+They have been separated as they had different purposes and different dependencies.
+E.g. raw data exploration (including diagrams used in the thesis) was mostly done in one package, having all these scripts in the same package that defines and fits the model would make that core reporitory larger, while probably not being helpful to most people unless they work on exactly the same data.
+Similarly the extraction of data from the moodle exports relies on some libraries that are also not required in the remainder of this project, and unlikely to be helpful to people that dont work on moodle questions.
+
+But I wanted to reuse some code across these, and Copy-Paste got out of hand with special versions that were hard to differentiate for me. 
+This package now holds a single true version of these, and is imported in the other packages (using poetry with a relative path, and a marker that this library may change).
+
+I won't necessarily publicise all project data (certainly not my tex files), so perhaps not all things in here will be reused in the public project files.
\ No newline at end of file

common_py/common_py/__init__.py

+# import plotting2
+# import .utils
\ No newline at end of file

common_py/common_py/plotting.py

+"""
+I find that pyplot is a pain to use and more importantly one often needs to interact with it
+inbetween other code.
+
+This intends to provide flexible enough wrappers around pyplots API,
+that can be used in a more declarative way.
+This makes it easier to differentiate between whats plotting code,
+and what is unrelated, and also from the plotting code,
+what makes it in the resulting pdf. 
+"""
+
+import matplotlib.pyplot as plt
+from typing import Tuple, Literal, Iterable, Callable, NamedTuple, Callable
+from abc import ABC, abstractmethod
+from dataclasses import dataclass
+import dataclasses
+from .utils import UNREACHABLE
+import numpy.typing as npt
+import numpy as np
+
+@dataclass
+class AxData:
+    label: str | None = None
+    bottom: float | None = None
+    top: float | None = None
+    scale: Literal["log"] | Literal["linear"] = "linear"
+
+class Plottable(ABC):
+    @abstractmethod
+    def apply(self, ax: plt.Axes):
+        UNREACHABLE("please implement")
+
+@dataclass
+class Plot(Plottable):
+    yData: Iterable
+    xData: Iterable | None = None
+    kwargs: dict = dataclasses.field(default_factory=lambda: {})
+    label: str | None = None
+    kind: Literal["line"] | Literal["scatter"] = "line"
+
+    def apply(self, ax: plt.Axes):
+        xData = self.xData if self.xData else range(len(self.yData))
+        match self.kind:
+            case "line": pass
+            case "scatter": self.kwargs = {"marker": "x", "linestyle": "None", **self.kwargs}
+            case _: UNREACHABLE("unrecognized plot kind")
+        ax.plot(xData, self.yData, label = self.label, **self.kwargs)
+
+@dataclass
+class Bar(Plottable):
+    heights: Iterable
+    xPositions: Iterable | None = None
+    width: float | None = None
+    kwargs: dict | None = None
+    label: str | None = None
+
+    def apply(self, ax: plt.Axes):
+        xPositions = self.xPositions if self.xPositions is not None else range(len(self.heights))
+        ax.bar(xPositions, self.heights, label = self.label, width=self.width, **(self.kwargs if self.kwargs else {}))
+
+@dataclass
+class Function(Plottable):
+    function: Callable[[float], float]
+    "The function to plot, takes x value, returns y value"
+    range_from: Iterable[npt.NDArray[float]]
+    samples: int = 2
+    kwargs: dict | None = None
+    label: str | None = None
+
+    def apply(self, ax: plt.Axes):
+        x_max = max(map(lambda range_series: np.max(range_series), self.range_from))
+        x_min = min(map(lambda range_series: np.min(range_series), self.range_from))
+        x_vals = np.linspace(x_min, x_max, self.samples)
+        ax.plot(x_vals, self.function(x_vals), label = self.label , **(self.kwargs if self.kwargs else {}))
+
+@dataclass
+class VertLine(Plottable):
+    position: float
+    linestyle: str
+    color: str = "green"
+    kwargs: dict | None = None
+    label: str | None = None
+
+    def apply(self, ax: plt.Axes):
+        ax.axvline(self.position, label = self.label, linestyle = self.linestyle , **(self.kwargs if self.kwargs else {}))
+
+@dataclass
+class InMemoryPlot:
+    fig: plt.Figure
+    ax: plt.Axes
+    save_path: str | None = None
+
+    def draw_and_save(self):
+        assert self.save_path != None
+        self.fig.savefig(self.save_path)
+        plt.close(self.fig)
+
+@dataclass(frozen=False)
+class PlotData:
+    # xdata: 
+    graphs: list[Plottable]
+    x_ax: AxData
+    y_ax: AxData
+    title: str | None = None
+    save_path: str | None = None
+
+    def create(self, add_legend: bool = True) -> InMemoryPlot:
+        fig: plt.Figure = plt.figure()
+        ax: plt.Axes = fig.add_subplot(111)
+        if self.title:
+            ax.set_title(self.title)
+        for graph in self.graphs:
+            graph.apply(ax)
+        ax.set_yscale(self.y_ax.scale)
+        ax.set_xscale(self.x_ax.scale)
+        if self.y_ax.label:
+            ax.set_ylabel(self.y_ax.label)
+        if self.x_ax.label:
+            ax.set_xlabel(self.x_ax.label)
+        ax.set_ylim(self.y_ax.bottom, self.y_ax.top)
+        ax.set_xlim(self.x_ax.bottom, self.x_ax.top)
+        if add_legend:
+            ax.legend()
+        ax.grid(True)
+        fig.tight_layout()
+        return InMemoryPlot(fig, ax, self.save_path)

common_py/common_py/string_processing.py

+from bs4 import BeautifulSoup
+from functools import reduce
+from typing import Iterable
+
+def cleanHtml(input: str):
+    return BeautifulSoup(input, features='lxml').get_text(separator=" ", strip=True)
+
+def combineStrIter(input: Iterable[str], separator: str = " "):
+    return reduce(lambda acc, itm: acc + separator + itm, input, "")
+
+def stripLines(input:str):
+    return combineStrIter(map(str.strip, input.splitlines())).strip()

common_py/common_py/utils.py

+"A few unrelated tools that may be reused across the different projects"
+
+from typing import NamedTuple
+from numpy import typing as npt
+import numpy as np
+
+def UNREACHABLE(reason: str = "Not specified"):
+    "Used to assert that the current branch of execution is unreachable under intended circumstances."
+    raise Exception(f"Should be unreachable because: {reason}")
+
+import warnings
+import functools
+
+def identity(arg):
+    "just returns the argument."
+    return arg
+
+class Statistics(NamedTuple):
+    min: float
+    max: float
+    mean: float
+    std_dev: float
+
+    def from_numpy(source: npt.ArrayLike):
+        return Statistics(
+            np.min(source),
+            np.max(source),
+            np.mean(source),
+            np.std(source),
+        )
+
+def deprecated(reason: str = "No reason given"):
+    """
+    This is a decorator which can be used to mark functions
+    as deprecated. It will result in a warning being emitted
+    when the function is used.
+
+    Because I'm to lazy to look up how to do it right, you'll always need to call this decorator:
+
+    ```py
+    # correct
+    @deprecated()
+    def some_function():
+        pass
+
+    # not correct
+    @deprecated
+    def some_other_function():
+        pass
+    ```
+
+    You can optionally provide a reason in that call (that is the optional parameter of this function)
+    """
+    def inner_deprecated(func):
+        @functools.wraps(func)
+        def wrapper(*args, **kwargs):
+            warnings.simplefilter('always', DeprecationWarning)  # turn off filter
+            warnings.warn(
+                f"Call to deprecated function {func.__name__}.\nReason for deprecation: {reason}",
+                category=DeprecationWarning,
+                stacklevel=2,
+            )
+            warnings.simplefilter('default', DeprecationWarning)  # reset filter
+            return func(*args, **kwargs)
+        return wrapper
+    return inner_deprecated

common_py/pyproject.toml

+[tool.poetry]
+name = "common-py"
+version = "0.1.0"
+description = ""
+authors = ["Samuel Maier <samuel.maier2@hotmail.de>"]
+readme = "README.md"
+packages = [
+    {include = "common_py"}, 
+    {include = "common_py/py.typed"},
+]
+
+[tool.poetry.dependencies]
+python = ">=3.10"
+matplotlib = "^3.7.1"
+# thanks tensorflow...
+numpy = ">=1.22,<1.24"
+beautifulsoup4 = "^4.12.2"
+lxml = "^4.9.2"
+
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"

data_explore/.gitignore

+*.pdf
+*.json
+*.csv
\ No newline at end of file

data_explore/enem_interactions_histogram.py

+from common_py.plotting import *
+import polars as pl
+import os
+
+if __name__ == "__main__":
+    df = (pl
+        .scan_csv("../enem_aggregate/GENERATED/localGen_result.csv")
+        .collect()
+    )
+
+    print(df)
+    PlotData(
+        title="Interactions with ENEM questions",
+        save_path=f"./{os.path.basename(__file__).split('.')[0]}.pdf",
+        x_ax=AxData("#student interactions"),
+        y_ax=AxData("#questions"),
+        graphs=[
+            Plot(xData=df["#total_answers"].to_list(), yData=df["qname"].to_list(), kwargs={"marker": "x", "linestyle": "None"}),
+        ]
+    ).create().draw_and_save()
\ No newline at end of file

data_explore/inep_interactions_head_select_rows.py

+from common_py.plotting import *
+import polars as pl
+import numpy as np
+import os
+import json
+
+if __name__ == "__main__":
+    df: pl.DataFrame = (
+        pl.scan_csv("../raw_data/brasilian/official_data/2016/DADOS/MICRODADOS_ENEM_*.csv", separator=";", encoding="utf8-lossy")
+        .head(n=8)
+        .select([
+            pl.col("NU_INSCRICAO"), 
+            pl.lit("...").alias("[more rows]"),
+            pl.col("^TX_RESPOSTAS_..$"),
+            pl.lit("...").alias("[yet more rows]"),
+        ])
+        .collect()
+    )
+    df.write_csv(f"{os.path.basename(__file__).split('.')[0]}.csv")
+    print(df)
\ No newline at end of file

data_explore/irt_inep_vs_correctness.py

+from common_py.plotting import *
+import polars as pl
+import os
+
+if __name__ == "__main__":
+    df = pl.scan_csv("../enem_aggregate/GENERATED/localGen_result.csv").collect()
+    # print(df.filter(pl.col("irt_b") > 10))
+    plot = PlotData(
+        title="IRT difficulty (INEP) vs correctness",
+        save_path=f"./{os.path.basename(__file__).split('.')[0]}.pdf",
+        x_ax=AxData("INEP IRT Difficulty", top=8),
+        y_ax=AxData("Correctness (all participants)"),
+        graphs=[
+            Plot(xData=df["irt_b"].to_list(), yData=df["correctness"].to_list(), kwargs={"marker": "x", "linestyle": "None"}),
+        ]
+    ).create()
+    plot.draw_and_save()
\ No newline at end of file

data_explore/irt_marinho_vs_correctness.py

+from common_py.plotting import *
+import polars as pl
+import os
+import numpy as np
+import json
+
+if __name__ == "__main__":
+    df = pl.scan_csv("../enem_aggregate/GENERATED/localGen_result.csv").collect()
+    # print(df.filter(pl.col("irt_b") > 10))
+    irt_marinho = df["irt_b_marinho"]
+    correctness = df["correctness"]
+    filter_for_nulls = (irt_marinho.is_not_null() & correctness.is_not_null())
+    df = df.filter(filter_for_nulls)
+    irt_marinho = df["irt_b_marinho"]
+    correctness = df["correctness"]
+
+    filename = os.path.basename(__file__).split('.')[0]
+
+    plot = PlotData(
+        title="IRT difficulty (Marinho) vs correctness",
+        save_path=f"./{filename}.pdf",
+        x_ax=AxData("Marinho IRT Difficulty"),
+        y_ax=AxData("Correctness (all participants)"),
+        graphs=[
+            Plot(xData=df["irt_b_marinho"].to_list(), yData=df["correctness"].to_list(), kwargs={"marker": "x", "linestyle": "None"}),
+        ]
+    ).create(add_legend=False)
+    plot.draw_and_save()
+    linear_corrcoef = np.corrcoef(irt_marinho.to_numpy(), correctness.to_numpy())[0,1]
+
+    with open(f"./{filename}.json", mode="w") as fh:
+        json.dump({
+            "correlations_coefficient": linear_corrcoef,
+        }, fh, indent="\t")
+    for topic in ["MT", "LC", "CH", "CN"]:
+        df_topic = df.lazy().filter(pl.col("topic") == topic).collect()
+
+        irt_marinho = df_topic["irt_b_marinho"]
+        correctness = df_topic["correctness"]
+        # print(df.filter(pl.col("irt_b") > 10))
+        plot = PlotData(
+            title="IRT difficulty (Marinho) vs correctness",
+            save_path=f"./{filename}_{topic}.pdf",
+            x_ax=AxData("Marinho IRT Difficulty"),
+            y_ax=AxData("Correctness (all participants)"),
+            graphs=[
+                Plot(xData=irt_marinho.to_list(), yData=correctness.to_list(), kwargs={"marker": "x", "linestyle": "None"}),
+            ]
+        ).create(add_legend=False)
+        plot.draw_and_save()
\ No newline at end of file

data_explore/irt_marinho_vs_inep.py

+from common_py.plotting import *
+import polars as pl
+import numpy as np
+import os
+import json
+
+if __name__ == "__main__":
+    df = pl.scan_csv("../dataextraction/enem_aggregate/GENERATED/localGen_result.csv").collect()
+    # df = df.filter(pl.col("qname").str.contains("2017").is_not())
+    print(df.filter(pl.col("irt_b") > 10))
+    print(df.select(pl.col("^.*irt.*$")).describe())
+    inep_irt = df["irt_b"]
+    marinho_irt = df["irt_b_marinho"]
+    filter_for_nulls = (inep_irt.is_not_null() & marinho_irt.is_not_null())
+    inep_irt: pl.Series = inep_irt.filter(filter_for_nulls)
+    marinho_irt: pl.Series = marinho_irt.filter(filter_for_nulls)
+
+    # fitting and correlation REALLY dont like the outliers in INEP data,
+    # so we limit the range using quantiles.
+    # I got here thinking of confidence intervals,
+    # however this does not assume a normal distribution,
+    # unlike most default models of confidence intervals
+    border_exclude = 0.01
+    "has to be in [0,0.5)"
+
+    correlation_range = [inep_irt.quantile(border_exclude), inep_irt.quantile(1 - border_exclude)]
+    filter_for_range = inep_irt.is_between(*correlation_range)
+    inep_irt_rangefiltered: pl.Series = inep_irt.filter(filter_for_range)
+    marinho_irt_rangefiltered: pl.Series = marinho_irt.filter(filter_for_range)
+
+    linear_corrcoef = np.corrcoef(inep_irt_rangefiltered, marinho_irt_rangefiltered)[0,1]
+    # see https://numpy.org/doc/stable/reference/routines.polynomials.classes#fitting
+    [b, m] = np.polynomial.Polynomial.fit(inep_irt_rangefiltered.to_numpy(), marinho_irt_rangefiltered.to_numpy(), deg = 1).convert()
+    filename = os.path.basename(__file__).split('.')[0]
+    plot = PlotData(
+        title="IRT difficulty by Marinho vs INEP",
+        save_path=f"./{filename}.pdf",
+        x_ax=AxData("INEP", bottom=inep_irt.quantile(0.001), top=inep_irt.quantile(0.999)),
+        y_ax=AxData("Marinho"),
+        graphs=[
+            Plot(xData=inep_irt.to_list(), yData=marinho_irt.to_list(), kwargs={"marker": "x", "linestyle": "None"}),
+            Function(function=lambda x: m*x + b, range_from=[correlation_range], label=f"regression over correlation range"),
+        ]
+    ).create().draw_and_save()
+    with open(f"./{filename}.json", mode="w") as fh:
+        json.dump({
+            "correlations_coefficient": linear_corrcoef,
+            "filter_range": correlation_range,
+        }, fh, indent="\t")
\ No newline at end of file

data_explore/participants_hft.py

+import polars as pl
+
+if __name__ == "__main__":
+    df = pl.scan_csv("../moodle_extract/GENERATED/result.csv").collect()
+
+    print(df.select(pl.col("#quiz_participants")).describe())
\ No newline at end of file

data_explore/pyproject.toml

+[tool.poetry]
+name = "data-explore"
+version = "0.1.0"
+description = ""
+authors = ["Samuel Maier <samuel.maier2@hotmail.de>"]
+readme = "README.md"
+
+[tool.poetry.dependencies]
+python = ">=3.10"
+common-py = {path = "../common_py", develop = true}
+polars = "^0.18.2"
+numpy = "^1.25.0"
+
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"