mirror of
https://github.com/tiyn/stud.ip-crawler.git
synced 2025-04-02 16:07:47 +02:00
docker: added build and docker-compose
This commit is contained in:
parent
564a9ba977
commit
4eb313eb9a
65
README.md
65
README.md
@ -2,30 +2,31 @@
|
||||
|
||||
This is a program that downloads all files available for a given Stud.IP user.
|
||||
It only downloads and searches through the courses in the current semester.
|
||||
If you run the program again it only downloads files that have changed since the last time running it.
|
||||
If you run the program again it only downloads files that have changed since
|
||||
the last time running it.
|
||||
|
||||
## Features/To-Dos
|
||||
|
||||
- [x] Downloads files of given users active semester via commandline
|
||||
- [x] Keeping file structure of Stud.IP
|
||||
- [x] Specify username
|
||||
- [x] Specify password
|
||||
- [x] Specify Stud.IP-URL
|
||||
- [x] Specify output directory
|
||||
- [x] Specify chunk size to download big files
|
||||
- [x] Specify all important database variables
|
||||
- [x] Keeping file structure of Stud.IP
|
||||
- [x] Specify username
|
||||
- [x] Specify password
|
||||
- [x] Specify Stud.IP-URL
|
||||
- [x] Specify output directory
|
||||
- [x] Specify chunk size to download big files
|
||||
- [x] Specify all important database variables
|
||||
- [x] Only download files after given date
|
||||
- [x] Save and read download date
|
||||
- [x] Possible reset of download date
|
||||
- [x] Save and read download date
|
||||
- [x] Possible reset of download date
|
||||
- [x] Incremental file download
|
||||
- [x] Store id and chdate of downloaded files
|
||||
- [x] Store id and chdate of downloaded files
|
||||
- [x] Logging
|
||||
- [x] Console log
|
||||
- [x] Log file
|
||||
- [x] Specify log level
|
||||
- [ ] Docker
|
||||
- DockerHub image
|
||||
- Docker-compose with db
|
||||
- [x] Console log
|
||||
- [x] Log file
|
||||
- [x] Specify log level
|
||||
- [x] Docker
|
||||
[x] DockerHub image
|
||||
[x] Docker-compose with db
|
||||
|
||||
## Installation
|
||||
|
||||
@ -37,11 +38,39 @@ If you run the program again it only downloads files that have changed since the
|
||||
## Usage
|
||||
|
||||
Just run the file via `python3 run.py [options]`.
|
||||
Alternatively to `python3 run.py` you can give yourself permissions using `chmod +x run.py [options]` and
|
||||
Alternatively to `python3 run.py` you can give yourself permissions using
|
||||
`chmod +x run.py [options]` and
|
||||
run it with `./run.py [options]`.
|
||||
There are several options required to work.
|
||||
Run `python3 run.py -h` for a help menu and see which ones are important for you.
|
||||
|
||||
### Docker
|
||||
|
||||
#### Environment-variables
|
||||
|
||||
Set the following variables with the -e tag.
|
||||
| Name | Usage | Default |
|
||||
| ---- |------ | ------- |
|
||||
| `USER` | username on the studip server | `admin` |
|
||||
| `PSWD` | password on the studip server | `admin` |
|
||||
| `URL` | url of the studip server | `admin` |
|
||||
| `HOST` | ip of the mysql instance to connect | `mysql` |
|
||||
| `DB_USER` | username of the mysql instance to connect | `root` |
|
||||
| `DB_PSWD` | password of the mysql instance to connect | `root` |
|
||||
| `INTERVAl` | update interval in seconds | `86400` |
|
||||
|
||||
#### Volumes
|
||||
|
||||
Set the following volumes with the -v tag.
|
||||
| Volume-Name | Container mount | Description |
|
||||
| ----------- | --------------- | ----------------------------------- |
|
||||
| `studip_data` | `/studip/src/data` | directory for studip files to be saved to |
|
||||
|
||||
#### docker-compose.yml
|
||||
|
||||
Copy `docker/docker-compose.yml` and change it to your needs.
|
||||
Then run `docker-compose up`.
|
||||
|
||||
## Tested StudIP instances
|
||||
|
||||
- Carl von Ossietzky Universität Oldenburg
|
||||
|
35
docker/Dockerfile
Normal file
35
docker/Dockerfile
Normal file
@ -0,0 +1,35 @@
|
||||
FROM python
|
||||
|
||||
LABEL maintainer "TiynGER <mail@martenkante.eu>"
|
||||
|
||||
ENV USER admin
|
||||
|
||||
ENV PSWD admin
|
||||
|
||||
ENV URL admin
|
||||
|
||||
ENV HOST mysql
|
||||
|
||||
ENV INTERVAL 86400
|
||||
|
||||
ENV DB_USER root
|
||||
|
||||
ENV DB_PSWD root
|
||||
|
||||
RUN git clone https://github.com/tiyn/stud.ip-crawler /studip
|
||||
|
||||
WORKDIR /studip
|
||||
|
||||
RUN pip install -r src/requirements.txt
|
||||
|
||||
ADD run.sh .
|
||||
|
||||
RUN chmod +x run.sh
|
||||
|
||||
RUN chmod +x src/run.py
|
||||
|
||||
VOLUME /studip/src/data
|
||||
|
||||
WORKDIR /studip/src
|
||||
|
||||
CMD ["/studip/run.sh"]
|
29
docker/docker-compose.yml
Normal file
29
docker/docker-compose.yml
Normal file
@ -0,0 +1,29 @@
|
||||
version: "3.1"
|
||||
services:
|
||||
studip:
|
||||
image: tiyn/studip-crawler
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
USER: 'user'
|
||||
PSWD: 'pswd'
|
||||
URL: 'https://url.tld'
|
||||
INTERVAL: 86400
|
||||
HOST: mysql
|
||||
DB_USER: root
|
||||
DB_PSWD: root
|
||||
volumes:
|
||||
- studip_data:/studip/src/data
|
||||
|
||||
mysql:
|
||||
image: mysql
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
MYSQL_ROOT_PASSWORD: root
|
||||
volumes:
|
||||
- studip_db:/var/lib/mysql
|
||||
|
||||
volumes:
|
||||
studip_data:
|
||||
driver: local
|
||||
studip_db:
|
||||
driver: local
|
3
docker/run.sh
Executable file
3
docker/run.sh
Executable file
@ -0,0 +1,3 @@
|
||||
#!/bin/bash
|
||||
|
||||
while true; do python /studip/src/run.py -o /studip/src/data -u $USER -p $PSWD -s $URL --db_user $DB_USER --db_passwd $DB_PSWD --host $HOST && sleep $INTERVAL; done
|
Loading…
x
Reference in New Issue
Block a user