1
0
mirror of https://github.com/tiyn/stud.ip-crawler.git synced 2025-04-02 16:07:47 +02:00

docker: added build and docker-compose

This commit is contained in:
TiynGER 2020-11-09 22:46:21 +01:00
parent 564a9ba977
commit 4eb313eb9a
4 changed files with 114 additions and 18 deletions

View File

@ -2,30 +2,31 @@
This is a program that downloads all files available for a given Stud.IP user.
It only downloads and searches through the courses in the current semester.
If you run the program again it only downloads files that have changed since the last time running it.
If you run the program again it only downloads files that have changed since
the last time running it.
## Features/To-Dos
- [x] Downloads files of given users active semester via commandline
- [x] Keeping file structure of Stud.IP
- [x] Specify username
- [x] Specify password
- [x] Specify Stud.IP-URL
- [x] Specify output directory
- [x] Specify chunk size to download big files
- [x] Specify all important database variables
- [x] Keeping file structure of Stud.IP
- [x] Specify username
- [x] Specify password
- [x] Specify Stud.IP-URL
- [x] Specify output directory
- [x] Specify chunk size to download big files
- [x] Specify all important database variables
- [x] Only download files after given date
- [x] Save and read download date
- [x] Possible reset of download date
- [x] Save and read download date
- [x] Possible reset of download date
- [x] Incremental file download
- [x] Store id and chdate of downloaded files
- [x] Store id and chdate of downloaded files
- [x] Logging
- [x] Console log
- [x] Log file
- [x] Specify log level
- [ ] Docker
- DockerHub image
- Docker-compose with db
- [x] Console log
- [x] Log file
- [x] Specify log level
- [x] Docker
[x] DockerHub image
[x] Docker-compose with db
## Installation
@ -37,11 +38,39 @@ If you run the program again it only downloads files that have changed since the
## Usage
Just run the file via `python3 run.py [options]`.
Alternatively to `python3 run.py` you can give yourself permissions using `chmod +x run.py [options]` and
Alternatively to `python3 run.py` you can give yourself permissions using
`chmod +x run.py [options]` and
run it with `./run.py [options]`.
There are several options required to work.
Run `python3 run.py -h` for a help menu and see which ones are important for you.
### Docker
#### Environment-variables
Set the following variables with the -e tag.
| Name | Usage | Default |
| ---- |------ | ------- |
| `USER` | username on the studip server | `admin` |
| `PSWD` | password on the studip server | `admin` |
| `URL` | url of the studip server | `admin` |
| `HOST` | ip of the mysql instance to connect | `mysql` |
| `DB_USER` | username of the mysql instance to connect | `root` |
| `DB_PSWD` | password of the mysql instance to connect | `root` |
| `INTERVAl` | update interval in seconds | `86400` |
#### Volumes
Set the following volumes with the -v tag.
| Volume-Name | Container mount | Description |
| ----------- | --------------- | ----------------------------------- |
| `studip_data` | `/studip/src/data` | directory for studip files to be saved to |
#### docker-compose.yml
Copy `docker/docker-compose.yml` and change it to your needs.
Then run `docker-compose up`.
## Tested StudIP instances
- Carl von Ossietzky Universität Oldenburg

35
docker/Dockerfile Normal file
View File

@ -0,0 +1,35 @@
FROM python
LABEL maintainer "TiynGER <mail@martenkante.eu>"
ENV USER admin
ENV PSWD admin
ENV URL admin
ENV HOST mysql
ENV INTERVAL 86400
ENV DB_USER root
ENV DB_PSWD root
RUN git clone https://github.com/tiyn/stud.ip-crawler /studip
WORKDIR /studip
RUN pip install -r src/requirements.txt
ADD run.sh .
RUN chmod +x run.sh
RUN chmod +x src/run.py
VOLUME /studip/src/data
WORKDIR /studip/src
CMD ["/studip/run.sh"]

29
docker/docker-compose.yml Normal file
View File

@ -0,0 +1,29 @@
version: "3.1"
services:
studip:
image: tiyn/studip-crawler
restart: unless-stopped
environment:
USER: 'user'
PSWD: 'pswd'
URL: 'https://url.tld'
INTERVAL: 86400
HOST: mysql
DB_USER: root
DB_PSWD: root
volumes:
- studip_data:/studip/src/data
mysql:
image: mysql
restart: unless-stopped
environment:
MYSQL_ROOT_PASSWORD: root
volumes:
- studip_db:/var/lib/mysql
volumes:
studip_data:
driver: local
studip_db:
driver: local

3
docker/run.sh Executable file
View File

@ -0,0 +1,3 @@
#!/bin/bash
while true; do python /studip/src/run.py -o /studip/src/data -u $USER -p $PSWD -s $URL --db_user $DB_USER --db_passwd $DB_PSWD --host $HOST && sleep $INTERVAL; done