The sigarra scraping library no one asked for

Feupy is a Python library that provides an interface to FEUP's information system, SIGARRA. SIGARRA stands for Sistema de Informação para Gestão Agregada dos Recursos e dos Registos Académicos (Academic Register and Aggregated Resource Management Information System, give or take).

All web requests that don't need a special permission (that is, you don't need to log in to see the page) are stored in a persistent cache with timeouts, in order to minimize latency and the number of requests made to sigarra. All web requests that need a special permission (i.e. you need to be logged in) are stored in a non-persistent cache.

Installation

pip install feupy

Documentation

Click me

Building the package and uploading to PyPI

Change your working directory to the root of this project in your computer

Run these two commands

python setup.py sdist bdist_wheel
python -m twine upload dist/*

Enter your PyPI credentials when prompted
Celebrate!

Building the docs and checking them out locally

Make sure you have the latest version of sphinx installed
Change your working directory to docs/
Run this command
```
sphinx-build -b  html source build
```
Change your working directory to docs/build/
Run this command
```
python -m http.server 8080
```
Open a web browser at localhost:8080

Running the tests

Change your working directory to the root of this project in your computer
Run this command
```
python -m unittest
```
Hope that the tests pass

Acknowledgements

I would like to thank the following people:

The maintainers of the requests package
The maintainers of the requests_futures package
The maintainers of the beautifulsoup4 package
The maintainers of the lxml package
The maintainers of the Pillow package
Miguel Ramalho for creating sigpy

Examples

Logging in

>>> # For a function to be able to access pages where you need to be logged in,
>>> # you need to pass a Credentials object as an argument to that function.
>>> 
>>> # You can tell whether or not a function needs a Credentials object by
>>> # checking if the function needs a parameter called "credentials"
>>> 
>>> from feupy import Credentials
>>> 
>>> credentials = Credentials()
Username?
:> up201806185
Password for 201806185?
:>
>>> credentials
Credentials(201806185)

Student info

>>> from feupy import Student
>>> from pprint import pprint
>>> 
>>> daniel = Student(201806185) # That's me!
>>> print(f"Hello, {daniel.name}!")
Hello, Daniel Filipe Amaro Monteiro!
>>> pprint(daniel.courses)
({'course': Course(742, 2019), # MIEIC
  'first academic year': 2018, # (2018/2019)
  'institution': 'Faculty of Engineering'},) # Best faculty

Teacher info

>>> from feupy import Teacher
>>> 
>>> jlopes = Teacher(230756)
>>> 
>>> print(f"Thanks for teaching us Python, {jlopes.name}!")
Thanks for teaching us Python, João António Correia Lopes!
>>> 
>>> print(jlopes.presentation)
Personal Presentation
João Correia Lopes is an Assistant Professor in Informatics Engineering at the Universidade do Porto and a senior researcher at INESC TEC. He has graduated in Electrical Engineering in the University of Porto in 1984 and holds a PhD in Computing Science by Glasgow University in 1997. His teaching includes undergraduate and graduate courses in databases and web applications, software engineering and programming, markup languages and semantic web. He has been involved in research projects in the area of data management, service-oriented architectures and e-Science. Currently his main research interests are e-Science and research data management.
ResearcherID  ORCID  Google Scholar Citations  DBLP Author  Scopus Author
>>> 
>>> pprint(vars(jlopes))
{'acronym': 'JCL',
 'base_url': 'https://sigarra.up.pt/feup/en/',
 'career': 'Pessoal Docente de Universidades',
 'category': 'Professor Auxiliar',
 'department': 'Department of Informatics Engineering',
 'email': None,
 'links': ('http://www.fe.up.pt/~jlopes/',
           'https://www.authenticus.pt/R-000-6RX',
           'http://orcid.org/0000-0002-9040-0889'),
 'name': 'João António Correia Lopes',
 'p_codigo': 230756,
 'personal_webpage': 'http://www.fe.up.pt/~jlopes/',
 'presentation': 'Personal Presentation\n'
                 'João Correia Lopes is an Assistant Professor in Informatics '
                 'Engineering at the Universidade do Porto and...', #etc
 'profession': 'Docente',
 'rooms': 'I129',
 'status': 'Active',
 'url': 'https://sigarra.up.pt/feup/en/func_geral.formview?p_codigo=230756',
 'voip': 3375}

Curricular unit info

>>> from feupy import CurricularUnit
>>> 
>>> fpro = CurricularUnit(419983)
>>> 
>>> fpro.name
'Programming Fundamentals'
>>> fpro.acronym
'FPRO'
>>> 
>>> pprint(vars(fpro))
{'ECTS_credits': 6.0,
 'academic_year': 2018,
 'acronym': 'FPRO',
 'base_url': 'https://sigarra.up.pt/feup/en/',
 'code': 'EIC0005',
 'curricular_years': (1,),
 'has_moodle': True,
 'is_active': True,
 'name': 'Programming Fundamentals',
 'number_of_students': 182,
 'pv_ocorrencia_id': 419983,
 'regents': (Teacher(230756),),
 'semester': '1',
 'teachers': (Teacher(230756),
              Teacher(230756),
              Teacher(520205),
              Teacher(552793),
              Teacher(209847)),
 'text': 'Teaching language\n'
         'Portuguese\n'
         'Objectives\n'
         '1 - BACKGROUND\n'
         'Fluency in the process of software development is a basic '
         'prerequisite to the work of Informatics Engineers. In order to use '
         'computers to solve problems effectively, students must be competent '
         'at reading and writing programs using higher-order programming '
         'languages.\n'
         '2 - SPECIFIC AIMS...', # etc
 'url': 'https://sigarra.up.pt/feup/en/ucurr_geral.ficha_uc_view?pv_ocorrencia_id=419983',
 'webpage_url': 'https://web.fe.up.pt/~jlopes/doku.php/teach/fpro/index'}
>>> 
>>> pprint(fpro.other_occurrences())
(CurricularUnit(436425),
 CurricularUnit(419983),
 CurricularUnit(399878),
 CurricularUnit(384923),
 CurricularUnit(368689),
 CurricularUnit(350482),
 CurricularUnit(332985),
 CurricularUnit(272575),
 CurricularUnit(272574),
 CurricularUnit(272573),
 CurricularUnit(272572),
 CurricularUnit(272571),
 CurricularUnit(272570),
 CurricularUnit(272569))
>>> [uc.academic_year for uc in fpro.other_occurrences()]
[2019, 2018, 2017, 2016, 2015, 2014, 2013, 2012, 2011, 2010, 2009, 2008, 2007, 2006]
>>> 
>>> pprint(fpro.classes(credentials)) # Remember the Credentials object we created earlier?
{
    '1MIEIC01': [Student(201800000),
                 Student(201800001),
                 Student(201800002),
                 Student(201800003)],
    '1MIEIC02': [Student(201800004),
                 Student(201800005),
                 Student(201800006),
                 Student(201800007),
                 Student(201800008),
                 Student(201800009),
                 Student(201800010),
                 Student(201800011),
                 Student(201800012)],
    '...'     : [...] # etc
}
>>>
>>> pprint(fpro.students(credentials))
# (student, status, number of registrations, student type)
[(Student(201800001), 'Ordinário', 1, 'Normal'), 
 (Student(201800002), 'Ordinário', 1, 'Normal'),
 (Student(201800003), 'Ordinário', 1, 'Normal'),
 (Student(201800004), 'Estudante internacional', 1, 'Normal'),
 (Student(201800005), 'Estudante internacional', 1, 'Normal'),
 (Student(201800006), 'Ordinário', 1, 'Normal'),
 (Student(201800007), 'Ordinário', 1, 'Normal'),
 (Student(201800008), 'Ordinário', 2, 'Normal'),
 (Student(201800009), 'Ordinário', 2, 'Normal'),
 (Student(201800010), 'Ordinário', 1, 'Normal'),
 (Student(201800011), 'Estudante internacional', 1, 'Normal'),
 (Student(201800012), 'Trabalhador-Estudante', 1, 'Normal'),
 ... # etc
 ]
>>> 
>>> pprint(fpro.results(credentials)) # Get the results from the exams
{'Época Normal (1ºS)': [(Student(201800001), 10),
                        (Student(201800002), 13),
                        (Student(201800003), 10),
                        (Student(201800004), 'RFE'),
                        (Student(201800005), 'RFF'),
                        (Student(201800006), 'RFF'),
                        ...], # etc
'Época Recurso (1ºS)': [(Student(201800008), 11),
                        (Student(201800009), 7),
                        (Student(201800010), 8),
                        (Student(201800011), 8),
                        (Student(201800012), 'RFE'),
                        (Student(201800013), 13),
                        (Student(201800014), 5),
                        (Student(201800019), 'RFC'),
                        ...]} # etc
>>> 
>>> pprint(fpro.timetable(credentials)) # Returns the classes from the timetable as dicts
[{'class type': 'TP',
  'classes': ('1MIEIC04',),
  'curricular unit': CurricularUnit(419983),
  'finish': datetime.time(10, 0),
  'room': ('B307',),
  'start': datetime.time(8, 0),
  'teachers': (Teacher(209847), Teacher(520205)),
  'weekday': 'Monday'},
 {'class type': 'TP',
  'classes': ('1MIEIC01',),
  'curricular unit': CurricularUnit(419983),
  'finish': datetime.time(10, 30),
  'room': ('B302',),
  'start': datetime.time(8, 30),
  'teachers': (Teacher(230756),),
  'weekday': 'Tuesday'},
 {'class type': 'T',
  'classes': ('1MIEIC01',
              '1MIEIC02',
              '1MIEIC03',
              '1MIEIC04',
              '1MIEIC05',
              '1MIEIC06',
              '1MIEIC07',
              '1MIEIC08'),
  'curricular unit': CurricularUnit(419983),
  'finish': datetime.time(13, 30),
  'room': ('B002',),
  'start': datetime.time(12, 0),
  'teachers': (Teacher(230756),),
  'weekday': 'Tuesday'},
 {'class type': 'TP',
  'classes': ('1MIEIC06',),
  'curricular unit': CurricularUnit(419983),
  'finish': datetime.time(13, 30),
  'room': ('B310',),
  'start': datetime.time(11, 30),
  'teachers': (Teacher(209847), Teacher(552793)),
  'weekday': 'Wednesday'},
  ...] # etc

Course info

>>> from feupy import Course
>>> 
>>> mieic = Course(742)
>>> 
>>> mieic.name
'Master in Informatics and Computing Engineering'
>>> mieic.acronym
'MIEIC'
>>> [director.name for director in mieic.directors] # Let's see the names of the directors
['João Carlos Pascoal Faria', 'Maria Cristina de Carvalho Alves Ribeiro']
>>> 
>>> pprint(vars(mieic))
{'acronym': 'MIEIC',
 'base_url': 'https://sigarra.up.pt/feup/en/'
 'directors': (Teacher(210006), Teacher(209566)),
 'involved_organic_units': ('https://sigarra.up.pt/feup/en/',),
 'name': 'Master in Informatics and Computing Engineering',
 'official_code': '9459',
 'pv_ano_lectivo': 2019,
 'pv_curso_id': 742,
 'text': 'The Integrated Master in Informatics and Computing Engineering has '
         'been awarded the international EUR-ACE quality label. This certifies '
         'MIEIC as a high-quality programme which meets the international '
         'standards for professional engineering education at the masters '
         'level. [+ info]\n'
         '\n'
         'Accreditation by A3ES\r\n'
         'The Agency for Assessment and Accreditation of Higher Education '
         '(A3ES), at 20th of June, 2014, and in accordance with the '
         'recomendation and fundamentation produced by the respective External '
         ... , # etc
 'url': 'https://sigarra.up.pt/feup/en/cur_geral.cur_view?pv_curso_id=742&pv_ano_lectivo=2019'}
>>> 
>>> pprint(mieic.exams()) # Let's see this courses's currently scheduled exams
[{'curricular unit': CurricularUnit(420037),
  'finish': datetime.datetime(2019, 9, 6, 20, 0),
  'observations': 'Tenho um exame de outra disciplina marcado para esse '
                  'horário e assim juntava os dois.\r\n'
                  'José Luís Moura Borges\r\n',
  'rooms': ('B222',),
  'season': 'Especial de Conclusão - SET-E-ESPECIAL',
  'start': datetime.datetime(2019, 9, 6, 17, 0)},
 ..., # etc
 {'curricular unit': CurricularUnit(438941),
  'finish': datetime.datetime(2019, 9, 26, 13, 0),
  'observations': None,
  'rooms': ('B104', 'B208', 'B213'),
  'season': 'Exames ao abrigo de estatutos especiais - Mini-testes (1ºS)',
  'start': datetime.datetime(2019, 9, 26, 9, 0)},
 {'curricular unit': CurricularUnit(438941),
  'finish': datetime.datetime(2019, 9, 26, 17, 30),
  'observations': None,
  'rooms': ('B104', 'B213', 'B208', 'B207'),
  'season': 'Exames ao abrigo de estatutos especiais - Mini-testes (1ºS)',
  'start': datetime.datetime(2019, 9, 26, 13, 30)}]
>>> 
>>> mieic.curricular_units() # All the curricular units (with a link) from that course
[CurricularUnit(446081), CurricularUnit(437142), CurricularUnit(438941), 
CurricularUnit(436401), CurricularUnit(436402), CurricularUnit(436403), 
CurricularUnit(436404), CurricularUnit(436405), CurricularUnit(436406), 
CurricularUnit(436407), CurricularUnit(436408), CurricularUnit(436409), 
CurricularUnit(436410), CurricularUnit(436411), CurricularUnit(436412), 
CurricularUnit(436413), CurricularUnit(436414), CurricularUnit(436415), 
CurricularUnit(436416), CurricularUnit(436417), CurricularUnit(436418), 
CurricularUnit(436419), CurricularUnit(436420), CurricularUnit(436421), 
CurricularUnit(436422), CurricularUnit(436423), CurricularUnit(436424), 
CurricularUnit(436425), CurricularUnit(436426), CurricularUnit(436427), 
CurricularUnit(436428), CurricularUnit(436429), CurricularUnit(436430), 
CurricularUnit(436431), CurricularUnit(436432), CurricularUnit(436433), 
CurricularUnit(436434), CurricularUnit(436435), CurricularUnit(436436), 
CurricularUnit(436437), CurricularUnit(436438), CurricularUnit(436439), 
CurricularUnit(436440), CurricularUnit(436441), CurricularUnit(436442), 
CurricularUnit(436443), CurricularUnit(436444), CurricularUnit(436445), 
CurricularUnit(436446), CurricularUnit(436447), CurricularUnit(436448), 
CurricularUnit(436449), CurricularUnit(436450), CurricularUnit(436451), 
CurricularUnit(436452), CurricularUnit(436453), CurricularUnit(436454), 
CurricularUnit(436455), CurricularUnit(436456), CurricularUnit(436457), 
CurricularUnit(436458), CurricularUnit(436459), CurricularUnit(436460), 
CurricularUnit(436461), CurricularUnit(436462), CurricularUnit(436463), 
CurricularUnit(436464), CurricularUnit(436465), CurricularUnit(436466), 
CurricularUnit(436467), CurricularUnit(436468), CurricularUnit(436469), 
CurricularUnit(436470), CurricularUnit(436471)]
>>> len(mieic.curricular_units()) # Just out of curiosity
74
>>> [uc for uc in mieic.curricular_units() if 2 in uc.curricular_years and uc.semester == 1] # The uc's I will have this semester
[CurricularUnit(436433), CurricularUnit(436434), CurricularUnit(436435), CurricularUnit(436436), CurricularUnit(436437)]

Personal info

>>> from feupy import User
>>> 
>>> me = User.from_credentials(credentials)
>>> me.course.acronym
'MIEIC'
>>> 
>>> pprint(me.courses_units())
# Curricular unit       , grade
[(CurricularUnit(436433), None),
 (CurricularUnit(436434), None),
 (CurricularUnit(436435), None),
 (CurricularUnit(436436), None),
 (CurricularUnit(436437), None),
 (CurricularUnit(436439), None),
 (CurricularUnit(436438), None),
 (CurricularUnit(436441), None),
 (CurricularUnit(436442), None),
 (CurricularUnit(436440), None),
 (CurricularUnit(419981), 10),
 (CurricularUnit(419982), 11),
 (CurricularUnit(419985), 12),
 (CurricularUnit(419983), 13),
 (CurricularUnit(419984), 14),
 (CurricularUnit(420521), 15),
 (CurricularUnit(419986), 16),
 (CurricularUnit(419987), 17),
 (CurricularUnit(419990), 18),
 (CurricularUnit(419989), 19),
 (CurricularUnit(419988), 20)]

Name		Name	Last commit message	Last commit date
Latest commit History 184 Commits
docs/source		docs/source
feupy		feupy
test		test
.gitattributes		.gitattributes
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
MANIFEST.in		MANIFEST.in
README.md		README.md
requirements.txt		requirements.txt
setup.py		setup.py

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

The sigarra scraping library no one asked for

Installation

Documentation

Building the package and uploading to PyPI

Building the docs and checking them out locally

Running the tests

Acknowledgements

Examples

Logging in

Student info

Teacher info

Curricular unit info

Course info

Personal info

About

Contributors 3

Languages

License

dfamonteiro/feupy

Folders and files

Latest commit

History

Repository files navigation

The sigarra scraping library no one asked for

Installation

Documentation

Building the package and uploading to PyPI

Building the docs and checking them out locally

Running the tests

Acknowledgements

Examples

Logging in

Student info

Teacher info

Curricular unit info

Course info

Personal info

About

Topics

Resources

License

Stars

Watchers

Forks

Contributors 3

Languages