This program downloads all files of a Stud.IP users current semester.

Go to file

myphs 0a5cada92d FIX: Skip dirs without id or name instead of aborting There are directories now that contain neither a name or an id delivered by the API for some reason, looking like this: { 'is_visible': True, 'is_readable': False, 'is_writable': True } which makes no sense at all. If such a directory is given the processing is stopped here and all further directories are ignored. This should not be the case. This commit fixes this behavior, so that all directories are processed as expected.		3 years ago
src	FIX: Skip dirs without id or name instead of aborting	3 years ago
.gitignore	database: files id and chdates are stored	5 years ago
Dockerfile	bugfixes: included error message for large files - skipping them	4 years ago
LICENSE	adding license	5 years ago
README.md	logo: resize logo	4 years ago
docker-compose.yml	database: switching from mysql to sqlite	4 years ago
docker-entry.sh	database: switching from mysql to sqlite	4 years ago
studip_crawler_alt.png	readme: updated logo	4 years ago

README.md

Stud.IP Crawler

This is a program that downloads all files available for a given Stud.IP user. It only downloads and searches through the courses in the current semester. If you run the program again it only downloads files that have changed since the last time running it.

Features/To-Dos

Downloads files of given users active semester via commandline
- Keeping file structure of Stud.IP
- Specify username
- Specify password
- Specify Stud.IP-URL
- Specify output directory
- Specify chunk size to download big files
- Specify all important database variables
Only download files after given date
- Save and read download date
- Possible reset of download date
Incremental file download
- Store id and chdate of downloaded files
Logging
- Console log
- Log file
- Specify log level
Docker
- DockerHub image
- Docker-compose with db

Errors and workarounds

If filename is too long it currently gets skipped
Filenames get sometimes displayed wrong (probably utf8 problems)

Installation

create an instance of a mysql database
git clone https://github.com/tiyn/studip-crawler
cd studip-crawler/src/
pip3install -r requirements - install dependencies

Usage

Just run the file via python3 run.py [options]. Alternatively to python3 run.py you can give yourself permissions using chmod +x run.py [options] and run it with ./run.py [options]. There are several options required to work. Run python3 run.py -h for a help menu and see which ones are important for you.

Docker

Environment-variables

Set the following variables with the -e tag.

Name	Usage	Default
`USER`	username on the studip server	`admin`
`PSWD`	password on the studip server	`admin`
`URL`	url of the studip server	`admin`
`INTERVAl`	update interval in seconds	`86400`

Volumes

Set the following volumes with the -v tag.

Volume-Name	Container mount	Description
`studip_data`	`/studip/src/data`	directory for studip files to be saved to

docker-compose.yml

Copy docker-compose.yml and change it to your needs. Then run docker-compose up.

Tested StudIP instances

Carl von Ossietzky Universität Oldenburg