Skip to content
GitLab
Commit 5697095e authored by Samuel Maier's avatar Samuel Maier
Browse files

Update snapshot

parent b6dd9af8
Show whitespace changes
Inline Side-by-side
LICENSE 0 → 100644
View file @ 5697095e
MIT License
Copyright (c) 2023 Samuel Maier
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
README.md
View file @ 5697095e
# The code accompaning `Application of pre-trained transformers to estimate the difficulty of closed questions using question and choice text` # The code accompaning `Application of pre-trained transformers to estimate the difficulty of closed questions using question and choice text`
This contains the code that was used to produce the results in the paper. This contains the code used to produce the results in the paper.
It contains a number of subdirectories that are somewhat separate, somewhat dependent on each other. It contains a number of subdirectories that are somewhat separate and somewhat interdependent.
Most of the subdirectories contain their own `README.md`. Most of the subdirectories have their own `README.md`.
One thing that is common to almost all directories is their general Python toolchain. One thing that almost all directories have in common is their general Python toolchain.
## Common Python toolchain ## Common Python toolchain
The project uses Python 3.10+ (I used my system python, which is at `3.11.3`, i did not knowingly use any python 3.11 features and much of the code was also tested with `3.10`) and `python-poetry` (`1.5.1`) was used for other python package dependencies and tools. The project uses Python 3.10+, and `python-poetry` (`1.5.1`) has been used for other Python package dependencies and tools.
If your system has an older python version and cant easily be updated (Ubuntu/Debian, looking at you) perhaps look at [`pyenv`](https://github.com/pyenv/pyenv). If your system has an older version of python and cannot be easily upgraded (Ubuntu/Debian, looking at you), you may want to look at [`pyenv`](https://github.com/pyenv/pyenv).
To install poetry just follow [the guide](https://python-poetry.org/docs/#installation). To install poetry, just follow [the guide](https://python-poetry.org/docs/#installation).
### Poetry ### Poetry
Poetry is generally similar to `npm`. Poetry is generally similar to `npm`.
Generally all you should need to know, if you dont want to change anything with the dependencies, is to run `poetry install` in the directories of the subprojects (they contain a `pyproject.toml` file) and then execute python files with `poetry run python <whateverfile.py>`. In general, unless you want to change dependencies, all you need to do is run `poetry install` in the subproject directories (they contain a `pyproject.toml` file) and then run python files with `poetry run python <whateverfile.py>`.
Poetry manages dependency versions in 2 files, one being dev-facing: `pyproject.toml` contains the versions as defined(ish) by the developer. Its following the rules for [semantic versioning](https://semver.org/) and their specifiers. Poetry manages dependency versions in 2 files, one of which is dev-facing:
You may add a dependency (also to that file) by being in the project directory and executing `poetry add <pypi.org name>`. `pyproject.toml` contains the versions as defined(ish) by the developer. It follows the rules of [semantic versioning] (https://semver.org/).
You can add a dependency (also to this file) by being in the project directory and running `poetry add <pypi.org name>`.
There is another file. The `poetry.lock` file should be version controlled too, though not be edited by you. There is another file. The `poetry.lock` file should also be version controlled, but not edited by you.
It contains the concrete versions, that were actually installed by poetry. It contains the actual versions of the dependencies that poetry installs.
This is intended to solve the "but it worked on my pc" issue plagueing tools like `pip`. This should solve the "but it worked on my pc" problem that plagues tools like `pip`.
Poetry installs dependencies in a [python virtual environment](https://docs.python.org/3/library/venv.html#module-venv), that it manages for you. Poetry installs dependencies in a [python virtual environment](https://docs.python.org/3/library/venv.html#module-venv) which it manages for you.
If you want access to that virtual environment, you can see information about it with `poetry env info`. If you want to access this virtual environment, you can see information about it with `poetry env info`.
This may be helpful with tools such as `jupyter`, or to get IDE autocomplete. This can be useful with tools like `jupyter`, or to get the IDE to autocomplete.
E.g. with my Visual Studio Code (vscode) that has the Python extension installed, all I have to do to get completions, documentation on mouse hover and other niceties for my dependencies (numpy, tensorflow etc) is to open the project in a vscode instance, then click on the number in the lower right corner of the editor (after "Python"), then click `+ Enter interpreter path`, and then copy paste the `Executable` path from `poetry env info` into there. Example output:
```sh
Virtualenv
Python: 3.11.3
Implementation: CPython
Path: /home/<user>/.cache/pypoetry/virtualenvs/qde-model-code-9-Jx55oZ-py3.11
Executable: /home/<user>/.cache/pypoetry/virtualenvs/qde-model-code-9-Jx55oZ-py3.11/bin/python
Valid: True
These instructions are all untested, and don't contain any pictures, so good luck to these trying to do expecially the last part here without some preexisting knowledge, but frankly I think I've done much more here already than other people would. System
Platform: linux
OS: posix
Python: 3.11.3
Path: /usr
Executable: /usr/bin/python3.11
```
Don't get intimidated by all the big words thrown around here. For example, with my Visual Studio code (vscode) that has the Python extension installed, "all I have to do" to get completions, documentation on mouse hover, and other niceties for my dependencies (numpy, tensorflow, etc) is to open the project in a vscode instance, then click on the number in the lower right corner of the editor (after "Python"), then click `+ enter interpreter path`, and then copy and paste the (first) `Executable` path from `poetry env info` into there (so `/home/<user>/.cache/pypoetry/virtualenvs/qde-model-code-9-Jx55oZ-py3.11/bin/python`).
You dont need to understand most of them to begin with, and eventually you'll develop an understanding for them.
ChatGPT can probably help your understanding for these, haven't tried if for something like that! These instructions are all untested and do not include pictures, so good luck to those trying to do expecially the last part here without some prior knowledge, but frankly I think I've already done a lot more here than other people would.
Don't be intimidated by all the big words being thrown around here.
You don't need to understand most of them at first, and eventually you'll develop an understanding for them.
ChatGPT can probably help you understand them, I haven't tried it for that!
### Other tools & choices for Python ### Other tools & choices for Python
The python code contains a lot of typing. My Python code contains a lot of typing.
This is mostly informal in nature (for both me as well as the IDE), and often doesnt pass typechecking. This is mostly informal in nature (for me as well as the IDE), and often does not pass type checking.
Sadly there doesnt really seem to be any standard for python typechecking anyways. Unfortunately, there seems to be no real standard for Python type checking anyway.
That said, some of the larger projects were occasionally checked with `mypy` (a typechecker) and ruff (a linter), and some select offenders were solved. However, some of the larger projects have occasionally been checked with `mypy` (a typechecker) and ruff (a linter), and some select offenders have been fixed.
What wasnt solved on purpose is a lot of errors about values potentially being None (which is also offered in a somewhat unhelpful way by mypy btw). What has not been intentionally fixed is a lot of errors about values potentially being None (which is also offered in a somewhat unhelpful way by mypy, btw).
Python doesnt have any syntactic sugar to makes these easier to handle (such as [Optional chaining](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Optional_chaining) or chaining non-null assertion). Python does not have any syntactic sugar to make these easier to handle (such as [optional chaining](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Optional_chaining) or chaining non-null assertions).
Also usually the "wrong" code will lead to the same results a non-null assertion would lead to in other languages, an exception saying "that thing is None!. Also usually the "wrong" code will lead to the same results a non-null assertion would lead to in other languages, an exception saying "that thing is None!.
Sadly I'm not aware of a way to tell mypy that this is fine, so I'll ignore these kinds of errors. Sadly I'm not aware of a way to tell mypy that this is fine, so I'll ignore these kinds of errors.
## Overview over the subdirectories ## Overview over the subdirectories
* [common_py](./common_py/) contains some utilities that were used across the different projects * [common_py](./common_py/) contains some utilities that have been used across projects.
* [raw_data](./raw_data/) contains a bunch of gitignores ingnoring non pubpic data and READMEs explaining the data that should be there. This way other subdirectories can reference these by path. * [raw_data](./raw_data/) contains a bunch of gitignores ignoring non-public data and READMEs explaining what data should be there. This way, other subdirectories can reference it by path.
* [enem_aggregate](./enem_aggregate/) contains the code that converted the raw data from enem, alongside the infos from Marinho, into a much simpler unified CSV containing the aggregated interactions. * [enem_aggregate](./enem_aggregate/) contains the code that converts the raw data from enem, along with the info from Marinho, into a much simpler unified CSV containing the aggregated interactions.
* [moodle_extract](./moodle_extract/) contains the code that parsed the moodle question xml and bundled it together with the experimental data, in form of a CSV. * [moodle_extract](./moodle_extract/) contains the code that parsed the moodle question xml and bundled it together with the experimental data in the form of a CSV.
* [qde_model_code](./qde_model_code/) contains the heart of this thing, the code that trained the ml model on the data generated by the other projects. * [qde_model_code](./qde_model_code/) contains the heart of the thing, the code that trained the ml model on the data generated by the other projects.
* [data_explore](./data_explore) (may not be added) contains some (not all) of my python scripts that were used to get insights into data and often to plot the data. * [data_explore](./data_explore) (may not be added) contains some (not all) of my Python scripts that were used to gain insight into the data and often to plot the data.
### Configuration ### Configuration
None of the projects here has a defined CLI. None of the projects here have a defined CLI.
Configuration is done in the source files themselves. Configuration is done in the source files themselves.
I did not want to add a dependency or the complexity of having a CLI (that doesnt suck), and frankly with how fast the underlying code and with that the API changed, it wouldve been a lot of work that would just have hindered experimentation. I did not want to add a dependency or the complexity of having a CLI (which does not suck), and frankly with how fast the underlying code and thus the API changes, it would have been a lot of work that would have just hindered experimentation.
An effort was made to have the regularly changed configuration of each potential entrypoint in few places. An effort was made to have the regularly changing configuration of each potential entry point in few places, and not spread over the repository.
### Why a Monorepo, why separate projects ### Why a Monorepo, why separate projects
This was created as a Monorepo from the individual projects. This was created as a monorepo from the individual projects.
These were usually already (local) git repos. These were usually already (local) git repos.
Having these separate was a good thing, I had no global structure and a lot of copy paste in the beginning while I figured out the structure. Having them separate was a good thing, I had no global structure and a lot of copying and pasting in the beginning while figuring out the structure.
And the projects history may also contain data that was not meant to be released. And the project history may also contain data that was not intended to be shared.
Doing tasks with a lot of data wrangling doesn't easily convert to the things established in software repositories. Doing tasks with a lot of data wrangling doesn't easily translate to the things that are established in software repositories.
I considered implementing these as git submodules, but this was not done mostly because of complexity. I considered implementing these as git submodules, but this was not done mainly because of the complexity.
Git Subrepositories don't come without gotchas, the [official documentation](https://git-scm.com/book/en/v2/Git-Tools-Submodules) as an entire subsection about these, and theres some others that may be more problematic. Git subrepositories don't come without gotchas, the [official documentation](https://git-scm.com/book/en/v2/Git-Tools-Submodules) has an entire subsection about these, and theres some others that may be more problematic.
This project will already be complex to execute. This project will already be complex to run.
I dont have to make them learn how to handle subrepositories too. I do not need to make them learn how to handle subrepositories as well.
Add Adding
* the mentioned potential leaking of data, which is workintensive to solve * the aforementioned potential data leakage, which is labor-intensive to fix
* while this probably already contains a lot of not so glamarous code, other stuff is in the past and I'd prefer it not being available to the worldwide web * While this probably already contains a lot of not-so-glamorous code, other stuff is in the past, and I'd prefer it not to be available to the worldwide web.
* submodules make multiple remote repositories difficult from my understanding (as they refer to the submodules per remote url), and I consider cloning this in a private repo in case my HFT gitlab repo gets deleted * Submodules make multiple remote repositories difficult from my understanding (as they point to the submodules per remote url), and I am considering cloning this into a private repo in case my HFT gitlab repo gets deleted.
\ No newline at end of file
common_py/README.md
View file @ 5697095e
# Common Python tools # Common Python tools
I've used multiple Python packages as part of this thesis. I've used several Python packages as part of this thesis.
They have been separated as they had different purposes and different dependencies. They were separated because they had different purposes and different dependencies.
E.g. raw data exploration (including diagrams used in the thesis) was mostly done in one package, having all these scripts in the same package that defines and fits the model would make that core reporitory larger, while probably not being helpful to most people unless they work on exactly the same data. For example, the exploration of the raw data (including the graphs used in the thesis) was mostly done in one package, having all those scripts in the same package that defines and adjusts the model would make this core reporter larger, while probably not being helpful to most people unless they are working on exactly the same data.
Similarly the extraction of data from the moodle exports relies on some libraries that are also not required in the remainder of this project, and unlikely to be helpful to people that dont work on moodle questions. Similarly, extracting data from the moodle exports relies on some libraries that are also not needed in the rest of this project, and are unlikely to be helpful to people who are not working on moodle questions.
But I wanted to reuse some code across these, and Copy-Paste got out of hand with special versions that were hard to differentiate for me. But I wanted to reuse some code across these, and copy-paste got out of hand with special versions that were hard for me to distinguish.
This package now holds a single true version of these, and is imported in the other packages (using poetry with a relative path, and a marker that this library may change). This package now contains a single true version of these, and is imported into the other packages (using poetry with a relative path, and a marker that this library may change).
I won't necessarily publicise all project data (certainly not my tex files), so perhaps not all things in here will be reused in the public project files. I won't necessarily release all project data (certainly not my tex files), so not everything in here may be reused in the public project files.
\ No newline at end of file
# Installation
See Top-Level [README](../README.md)
\ No newline at end of file
data_explore/README.md 0 → 100644
View file @ 5697095e
# Tbis directory
This directory contains a number of standalone files used to generate plots, tables, etc.
# Installation
See Top-Level [README](../README.md)
\ No newline at end of file
enem_aggregate/README.md
View file @ 5697095e
# ENEM aggregation code # ENEM aggregation code
This code does most of the grunt work of aggregating the massive interaction files from INEP for ENEM, as well as aggregating accross metafiles also provided by INEP for each year (which contain their IRT values for the questions) and also combining these with the question and answer text provided by Marinho. This code does most of the grunt work of aggregating the massive interaction files from INEP for ENEM, as well as aggregating the metafiles also provided by INEP for each year (which contain their IRT values for the questions), and also combining these with the question and answer text provided by Marinho.
It does *not* necessarily bring the results into the concrete format used by the models. That is done in python to allow for more model specific adaptions in in the model repository itself. It does *not* necessarily put the results into the concrete format used by the models. That is done in Python to allow for more model-specific customization in the model repository itself.
Originally this code was intended to be run in a streaming fashion on just any PC. Originally, this code was intended to run in a streaming fashion on any PC.
However that did not work out, so a part is executed on the Laptop now. This was done in Rust mainly for 3 reasons, unfortunately 2 of them didn't really work out:
Because there was already a lot of code, leading to at the very least a lot of manual labor I cant spare to port to Python, we do retain a split of the code. * I wanted to avoid having to learn and use yet another domain-specific language to do efficient data wrangling as required in Python. Unfortunately, Polars makes it difficult to use standard Rust tools, so there is still a lot of that here as well.
* We noticed that the Python program became difficult to abort when we ran out of system resources. The hope was that this was due to some inter-process communication or something going on with the Python polars API to core polars (which is implemented in rust). If there was indeed an improvement in this regard, it was not noticeable to me.
* Debugging can be easier in Rust, if you're comfortable with using panics. This is because it embeds the location of each potential panic, and when it happens, displays that location. Polars make debugging difficult due to their lazy and declarative nature. This is the one thing where it worked. Unfortunately, there are still situations where Polars panics internally, which are hard to trace.
However, this did not work out, so part of it is now done on the cluster.
Since there was already a lot of code when we gave up we kept it around, asI can not spare the time to port it to Python.
Because we're working with a lot of data, that may or may not be what we expected, this code is written in a very script like fashion. Because we're working with a lot of data that may or may not be what we expected, this code is written in a very script-like fashion.
## Installation ## Installation
I suggest installing rust and its connected tools via `rustup`. I suggest installing rust and its related tools via `rustup`.
Install `rustup` either via your package manager (`homebrew`, `winget`, `apt` etc probably have it), or from https://rustup.rs/. Install `rustup` either through your package manager (`homebrew`, `winget`, `apt`, etc. probably have it) or from https://rustup.rs/.
Rustup should install all the tools required to run the local code on your computer. Rustup should install all the tools needed to run the local code on your machine.
If you want to run the code on the cluster ensure that you can compile code that the cluster can run by executing `rustup toolchain install stable-x86_64-unknown-linux-gnu` (this is fine anywhere, and required on Apple Silicon or Windows). If you want to run the code on the cluster, make sure you can compile code that the cluster can run by running `rustup toolchain install stable-x86_64-unknown-linux-gnu` (this is fine everywhere, and required on Apple Silicon or Windows).
The results of both the local and the cluster code are mostly already present in `./GENERATED/`, so that you dont have to install the tools, and dont have to execute the lengthy cluster code. The results of both the local and cluster code are mostly already in `./GENERATED/`, so there is no need to install the tools and run the lengthy cluster code.
Update: I wont release the ENEM texts, as they were provided by Marinho and they didnt open them up themselfes. Theres a [file without that text](./GENERATED/localGen_result_no_secrets.csv) but the Identifiers used by Marinho when he send the data over to us, if you get the same format we got it should be easy to unify. Update: I wont release the ENEM texts, as they were provided by Marinho and they didnt open them themselves. Theres a [file without this text](./GENERATED/localGen_result_no_secrets.csv) but the identifiers used by Marinho when he sent the data over to us, if you get the same format we got it should be easy to unify.
## A few words about Rust
Rust doesnt have exceptions, instead relying on a `Result` type as return value, and `panic`s.
`panic`s are not intended to be catched and handled during program execution, instead they will terminate the program.
`Result` is an Enum, but Rust enums differ from eg. Java Enums, in that they may contain values per instance (this is know by terms such as tagged union, discriminated union, product type).
In handling, because it is a value that contains the "good path" value, it behaves similar to a checked exception in Java, though it is closer to the `Either` class in Haskell - with some additional meaning, the variants are called `Ok` and `Err`.
A `Result` may be promoted to a panic with eg. `unwrap()`, which, because of the script-like nature of this code is done frequently.
Rust also has the `Option` type, which is similar, but replaces the concept of (type-checked) nulls in Rust - it is equivalent to `Maybe` in Haskell. That type also has an `unwrap()` which behaves similar.
One last thing I've frequently done in this code that may be unknown to people is using `let-else` bindings.
I will assume familiarity with pattern matching, there is a lot of resources on it, and I've also done it in Python.
Rust has pattern matching, but in many spots, eg a `let` pattern-binding it requires that the bound pattern does always apply, so matching an vector that is known by the programmer but not the compiler to contain 2 elements is still impossible.
`let-else` bindings were adopted in Rust, from Swift, to allow to use these `fallible` bindings.
It takes the form
```rs
let [element1, element2] = arraylike_object else {
// some diverging code!
}
```
The `diverging` path must exit the scope that the pattern is expanded into.
That may be done via `return`, or also a `panic`, as that exits the program and Rust is aware of that.
Because of that I've mostly placed `unreachable!()` in there, which causes a `panic`.
`unreachable!()` and also `dbg!()` are Macros.
Rust Macros are relatively easier to understand and debug in comparison to e.g. C macros, and they allow some nice things. For example `dbg!()` does println debugging on stereoides, as it doesnt just print the inserted value, but also the code-snippet it contained, and its concrete source location.
Rust allows to assign a lot of things to other thing, eg if-else, they are all `expressions`.
In particular code blocks delimited with `{}` can be placed anywhere (similar to C), but they can also be assigned as expressions.
In this case, and in general in functions, an element thats not followed with an `;` will be returned.
In functions this is something that is not required.
But `return` always returns from the currenct function, so in `{}`-delimited blocks return will not work, so this "implicit return" has to be used.
I use these blocks in some assignments, where I want to ensure that the stuff I wite in this block isnt by accident picked up elsewhere, but in particular I use these blocks extensively without assigning them.
If I do the latter chances are that these are tests performed for debugging.
This code is not strictly required to understand the data flow, though perhaps youll find them useful (I sure did) to know how the data actually looks like at this point.
For an full intro into Rust I recommend [the excellent and free Rust book](https://doc.rust-lang.org/stable/book/) and/or [rustlings excercises](https://github.com/rust-lang/rustlings) for practice.
## Polars
The only (direkt and with a broad API) dependency of this project is `polars`, a `pandas` alternative written in Rust, with a python API (used in the related projects).
It has a few features that `pandas` is lacking, such as support for streaming.
Sadly these specific features are not very mature yet.
Generally the API of `polars` seems not very stable at this point, and also very much oriented around the python API, which means it mostly doesn't utilize the niceties that Rusts strong type system can provide, while still requiring strong typing in other situations, which can make some things awkward.
I'm almost certain that I've used a number of APIs in ways that are not expected by the authors, but dont immediately fail, which makes it very hard to debug.
The result can be seen with some awkward code, and some very annoyed comments, sorry about that.
## Running the local code ## Running the local code
The local code is defined in `./src/local.rs` and also utilizes stuff in `./src/lib.rs` The local code is defined in `./src/local.rs' and also uses stuff in `./src/lib.rs'.
Ensure that all paths are correct (you'll get a runtime error otherwise) and execute Make sure all paths are correct (otherwise you will get a runtime error) and execute:
```sh ```sh
cargo run --release --bin local cargo run --release --bin local
``` ```
`cargo` is Rusts official build system (aka CMake) and package manager (aka npm). Cargo is Rust's official build system (aka CMake) and package manager (aka npm).
`--release` enables code optimizations. `--release` enables code optimizations.
...@@ -92,25 +46,74 @@ name="local" ...@@ -92,25 +46,74 @@ name="local"
path="src/local.rs" path="src/local.rs"
``` ```
On first run this will download the dependencies and compile them. Because of the enabled performance optimizations (mostly for the cluster aggregation) this will take a long time, subsequent runs should have these cached. On the first run, this will download and compile the dependencies. Due to enabled performance optimizations (mostly for cluster aggregation) this will take a long time, subsequent runs should have them cached.
## Running the Cluster code ## Running the Cluster code
Please consult the section for [the local code](#running-the-local-code) first for more information on whats happening. Please read the [the local code](#running-the-local-code) section first for more information on what happens.
The bwUniCluster cluster uses x86_64 CPUs and a Linux OS, thus we have to compile for that target:
The bwUniCluster cluster uses x86_64 CPUs and a Linux OS, so we need to compile for that target:
```sh ```sh
cargo build --target x86_64-unknown-linux-gnu --release --bin cluster cargo build --target x86_64-unknown-linux-gnu --release --bin cluster
``` ```
Rust doesnt usually dynamically link against many libraries, but it does by default link against gnu C libraries (`libc`, `libgcc_s`, `libm`), as do most binaries. Rust does not dynamically link against many libraries by default, but it does link against gnu C libraries (`libc`, `libgcc_s`, `libm`), as do most binaries.
Avoiding this with `musl` was investigated but caused a lot of other issues. Getting around this with `musl` was investigated, but caused a lot of other problems.
The bwUniCluster only has ancient versions of these around (from 4 years ago for `libc`), that are no longer supported by Rust, and I did not find an easy way to add/use a newer versions of these particular libraries on the cluster. The bwUniCluster only has old versions of these (from 4 years ago for `libc`) that are no longer supported by Rust, and I could not find an easy way to add/use newer versions of these particular libraries on the cluster.
Because of that I ran the binary in a container that provides modern libraries. So I ran the binary in a container that provides modern libraries.
The only container runtime provided on bwUniCluster is `enroot`, so this is what I've used. The only container runtime provided on bwUniCluster is `enroot`, so that's what I used.
Alternatively, you could just build the code on the cluster, but it doesn't have Rust tools - so it would need a container anyway - and compiler feedback would be annoying this way, youd have to build on both devices anyway to do debugging.
All steps to execute the bwUniCluster code are in [`execute_cluster.sh`](execute_cluster.sh) (see the [script in model code](../../reimplement/cluster_push_and_queue.sh) for some comments).
(TODO: this is untested as written)
## A few words about Rust
Rust doesnt have exceptions, instead relying on a `Result` type as return value, and `panic`s.
`panic`s are not intended to be cought and handled during program execution, instead they will terminate the program.
`Result` is an enum, but Rust enums differ from eg. Java enums, in that they may contain values per instance (this is know as tagged union, discriminated union, product type).
Since it is a value containing the "good path" value, it behaves similarly to a checked exception in Java, though it is closer to the `Either` class in Haskell - with some additional meaning, the variants are called `Ok` and `Err`.
A `Result` can be panicked with e.g. `unwrap()`, which is often done due to the script-like nature of this code.
Rust also has the `Option` type, which is similar but replaces the concept of (type-checked) nulls in Rust - it is equivalent to `Maybe` in Haskell. This type also has an `unwrap()` which behaves similarly.
One last thing I've done a lot in this code that people may not be aware of is the use of `let-else' bindings.
I will assume familiarity with pattern matching, there are a lot of resources on it, and I've also done it in Python.
Rust has pattern matching, but in many places, e.g. a `let` pattern binding, it requires that the bound pattern is always true, so matching a vector that is known to the programmer but not to the compiler to contain 2 elements is still impossible.
The `let-else' bindings have been adopted in Rust from Swift to allow the use of these `fallible' bindings.
It takes the form:
```rs
let [element1, element2] = arraylike_object else {
// some diverging code!
}
```
The `diverging` path must leave the scope into which the pattern is being expanded.
This can be done with a `return', or with a `panic', as this terminates the program and Rust knows about it.
That's why I've mostly put `unreachable!()` in there, which causes a `panic`.
`unreachable!()` and also `dbg!()` are macros.
Rust macros are relatively easier to understand and debug than, say, C macros, and they allow some nice things. For example, `dbg!()` does println debugging on steroids, as it prints not only the inserted value, but also the snippet of code it contains, and its concrete source location.
Rust allows you to assign many things to other things, e.g. if-else, they are all `expressions'.
In particular, blocks of code delimited by `{}` can be placed anywhere (similar to C), but they can also be assigned as expressions.
In this case, and in general in functions, an element that is not followed by a `;` will be returned.
In functions, this is something that is not required.
But `return` always returns from the current function, so in `{}`-delimited blocks return will not work, so this "implicit return" has to be used.
I use these blocks in some assignments where I want to make sure that the stuff I write in this block is not accidentally picked up elsewhere, but mostly I use these blocks extensively without assigning them.
When I do the latter, chances are that these are tests being run for debugging purposes.
This code is not strictly necessary to understand the data flow, although you may find it useful (I certainly did) to know what the data actually looks like at this point.
For an full intro into Rust I recommend [the excellent and free Rust book](https://doc.rust-lang.org/stable/book/) and/or [rustlings excercises](https://github.com/rust-lang/rustlings) for practice.
## Polars
Alternatively one could simply build the code on the cluster, but it doesn't have Rust tooling - so it would need a container anyways -, and compiler feedback would be annoying this way, youd have to build on both devices anyways do debug. The only (direct and broad API) dependency of this project is `polars`, a `pandas` alternative written in Rust, with a Python API (used in the related projects).
It has some features that `pandas` lacks, such as support for streaming.
Unfortunately, these specific features are not very mature yet.
In general, the `polars` API does not seem to be very stable at this point, and is also very much oriented towards the Python API, which means that it mostly doesn't take advantage of the niceties that Rusts strong type system can provide, while still requiring strong typing in other situations, which can make some things awkward.
All the steps to execute the bwUniCluster Code are in [`execute_cluster.sh`](execute_cluster.sh) (refer to the [script in the model code](../../reimplement/cluster_push_and_queue.sh) for some comments). I'm almost certain that I've used a number of APIs in ways that the authors didn't expect, but didn't fail immediately, making it very hard to debug.
(TODO: this is untested as written down, test before final submit) The result is some awkward code and some very annoyed comments, sorry about that.
\ No newline at end of file \ No newline at end of file
generate_repos_snapshot.sh
View file @ 5697095e
...@@ -13,14 +13,15 @@ if [ "$(pwd)" != "$target_directory" ]; then ...@@ -13,14 +13,15 @@ if [ "$(pwd)" != "$target_directory" ]; then
exit 1 exit 1
fi fi
# old code that recreated the repository each time
# "remove" old repo # "remove" old repo
rm -rf .git # rm -rf .git
cd ../.. cd ../..
# my filesystem is btrfs. Its a copy on write filesystem. # my filesystem is btrfs. Its a copy on write filesystem.
# Because of that it copies the 50GB of enem raw data just like that, glorious thing. # Because of that it copies the 50GB of enem raw data just like that, glorious thing.
cp -r common_py data_explore enem_aggregate moodle_extract qde_model_code raw_data README.md meta/monorepo cp -r common_py data_explore enem_aggregate moodle_extract qde_model_code raw_data README.md LICENSE meta/monorepo
cd meta/monorepo || exit cd meta/monorepo || exit
...@@ -38,4 +39,6 @@ set +o xtrace ...@@ -38,4 +39,6 @@ set +o xtrace
printf "\n\nNow push up with:\n" printf "\n\nNow push up with:\n"
echo 'git push --force --set-upstream origin main' echo 'git push'
\ No newline at end of file # old code that recreated the repository each time
# echo 'git push --force --set-upstream origin main'
\ No newline at end of file
moodle_extract/README.md
View file @ 5697095e
# Parsing HFT Questions and mapping them to a more generic format # Parsing HFT Questions and mapping them to a more generic format
This contains code that parses the generated moodle Question XMLs, then maps them to Python dataclasses ([dataclasses with parts of the xml parsing](./moodle_questions_dataclasses.py), [the code that uses these dataclasses and parses a file](./extract_moodle_xml.py)). This contains code that parses the generated Moodle question XMLs and then maps them to Python data classes ([data classes with parts of the xml parsing](./moodle_questions_dataclasses.py), [the code that uses these data classes and parses a file](./extract_moodle_xml.py)).
Then other code in here takes these Dataclasses and maps them to a more generic format. Then other code in here takes those dataclasses and maps them to a more generic format.
Dataloss is inevitable in that step, thus its separated to be able to iterate. Data loss is inevitable in this step, so it is separated to be able to iterate.
[This is that code](./moodle_to_generic_questions.py) [This is the code](./moodle_to_generic_questions.py)
Then, because I dont want to execute that stuff on the cluster (that would be a nightmare), these questions end up in a csv file together with the experimental information. Then, because I do not want to run this stuff on the cluster (that would be a nightmare), these questions end up in a csv file along with the experimental information.
The mapping of questions to strings (csv doesnt support nested data, questions have answers -> nesting) is done in [this file](./map_generic_question_to_str.py). The mapping of questions to strings (csv does not support nested data, questions have answers -> nesting) is done in [this file](./map_generic_question_to_str.py).
Note that during the simplification when mapping to the generic format, answers for some questions are already mapped into strings eariler as part of the simplification (eg "select the correct word here" type of quesrions). Note that during the simplification when mapping to the generic format, answers to some questions are already mapped to strings as part of the simplification (e.g. "choose the right word here" type questions).
The reading of the experimental data and the writing of the csv [happens here](./combine_hft_data.py). Reading the experimental data and writing the csv [happens here](./combine_hft_data.py).
\ No newline at end of file
# Installation
See Top-Level [README](../README.md)
\ No newline at end of file
qde_model_code/README.md
View file @ 5697095e
# The model creation and Tests # The model creation and Tests
This repo contains the code that trains on the data created in the other projects, end most of the evaluation of the model too. This repo contains the code that trains on the data created in the other projects and does most of the evaluation of the model.
As such it it has multiple entrypoints. As such it has multiple entry points.
The model creation and the creation of the csv files is done on the cluster, drawing plots etc should be done locally. Model creation and creation of csv files is done on the cluster, plotting etc should be done locally.
## Cluster execution ## Cluster execution
Code that executes on the cluster should always start in [its single entrypoint](./cluster_entry.py), and also consider the docuumentation at the start of the file. Code that runs on the cluster should always start in [its single entry point](./cluster_entry.py), and also see the documentation at the top of the file.
To execute the code onto the cluster, either use or inspect the [corresponding shell script](./cluster_push_and_queue.sh). Please refer to the corresponding file in [enem aggregation](../enem_aggregate/execute_cluster.sh) and its [README](../enem_aggregate/README.md) for more detail on how this file works. To execute the code on the cluster, either use or examine the [appropriate shell script](./cluster_push_and_queue.sh). See the corresponding file in [enem aggregation](../enem_aggregate/execute_cluster.sh) and its [README](../enem_aggregate/README.md) for more details on how this file works.
You probably want to change `EXEC_DIR` in that shell script, if you want to iterate, so that the results are added to a new directory. You will probably want to change `EXEC_DIR` in this shell script if you want to iterate, so that the results are added to a new directory.
The execution directory will contain the code that generated the results as a nice side effect, that way you can just refer to the code alongside the results if youre no longer certain how these results were generated (if you were clean and always made a new `EXEC_DIR`). The execution directory will contain the code that generated the results as a nice side effect, so you can just refer to the code next to the results if you are no longer sure how those results were generated (if you were clean and always made a new `EXEC_DIR`, otherwise they were overwritten).
\ No newline at end of file
# Installation
See Top-Level [README](../README.md)
\ No newline at end of file
qde_model_code/training_enem.py
View file @ 5697095e
...@@ -86,19 +86,6 @@ def enem_training(seed: int): ...@@ -86,19 +86,6 @@ def enem_training(seed: int):
train_steps = len(train_df) // config.training.batch_size train_steps = len(train_df) // config.training.batch_size
# print(
# train[input_encoding_key].to_list(),
# train[config.target].to_list(),)
# dataset = tf.data.Dataset.from_tensor_slices((
# train[input_encoding_key].to_list(),
# train[config.target].to_list(),
# ))
# print(dataset)
# DEV: Exit early during local dev.
# Exit with a nonzero exit code (=> failure)
# to hopefully get a Failure mail if I push that to the cluster by accident
# exit(1)
history = model.fit( history = model.fit(
(train_tf_dataset (train_tf_dataset
.repeat(tf.data.experimental.INFINITE_CARDINALITY) .repeat(tf.data.experimental.INFINITE_CARDINALITY)
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment

Dies ist die Gitlab-Instanz des Transferportals der Hochschule für Technik Stuttgart. Hier geht es zurück zum Portal