tiyn
/
stud.ip-crawler
mirror of https://github.com/tiyn/stud.ip-crawler
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
This program downloads all files of a Stud.IP users current semester.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
31 Commits
2 Branches
0 Tags
276 KiB
 
 
 
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '85490cb616'
${ noResults }
Go to file
TiynGER 85490cb616
logo: resize logo 		4 years ago
src database: switching from mysql to sqlite 4 years ago
.gitignore database: files id and chdates are stored 5 years ago
Dockerfile bugfixes: included error message for large files - skipping them 4 years ago
LICENSE adding license 5 years ago
README.md logo: resize logo 4 years ago
docker-compose.yml database: switching from mysql to sqlite 4 years ago
docker-entry.sh database: switching from mysql to sqlite 4 years ago
studip_crawler.png logo: resize logo 4 years ago
studip_crawler_alt.png logo: resize logo 4 years ago

README.md

Stud.IP Crawler

stud.ip-crawler-logo

This is a program that downloads all files available for a given Stud.IP user. It only downloads and searches through the courses in the current semester. If you run the program again it only downloads files that have changed since the last time running it.

Features/To-Dos

  • Downloads files of given users active semester via commandline
    • Keeping file structure of Stud.IP
    • Specify username
    • Specify password
    • Specify Stud.IP-URL
    • Specify output directory
    • Specify chunk size to download big files
    • Specify all important database variables
  • Only download files after given date
    • Save and read download date
    • Possible reset of download date
  • Incremental file download
    • Store id and chdate of downloaded files
  • Logging
    • Console log
    • Log file
    • Specify log level
  • Docker
    • DockerHub image
    • Docker-compose with db

Errors and workarounds

  • If filename is too long it currently gets skipped
  • Filenames get sometimes displayed wrong (probably utf8 problems)

Installation

  • create an instance of a mysql database
  • git clone https://github.com/tiyn/studip-crawler
  • cd studip-crawler/src/
  • pip3install -r requirements - install dependencies

Usage

Just run the file via python3 run.py [options]. Alternatively to python3 run.py you can give yourself permissions using chmod +x run.py [options] and run it with ./run.py [options]. There are several options required to work. Run python3 run.py -h for a help menu and see which ones are important for you.

Docker

Environment-variables

Set the following variables with the -e tag.

Name Usage Default
USER username on the studip server admin
PSWD password on the studip server admin
URL url of the studip server admin
INTERVAl update interval in seconds 86400

Volumes

Set the following volumes with the -v tag.

Volume-Name Container mount Description
studip_data /studip/src/data directory for studip files to be saved to

docker-compose.yml

Copy docker-compose.yml and change it to your needs. Then run docker-compose up.

Tested StudIP instances

  • Carl von Ossietzky Universität Oldenburg