Skip to content

Latest commit

 

History

History
273 lines (220 loc) · 12.1 KB

README.md

File metadata and controls

273 lines (220 loc) · 12.1 KB
Raw

Singapore Key Places Names


This repo contains the names of key places in Singapore (e.g. Bishan, Bedok, Tampines) in different data structures. You can either view the preset output text files, or use the command line program shown below.

Explore the docs »

· Report Bug · Request Feature

View preset output text files

Table of Contents
  1. About The Project
  2. Getting Started
  3. Usage
  4. Roadmap
  5. Contributing
  6. License
  7. Contact
  8. Acknowledgements

About The Project

CLI program and Web Scraper

This project generates a list of key places in Singapore in different data structures - list and dict. You may either view the preset output files here or clone this repo to use the command line program. The command line program allows you to choose your desired region (East, West, North, North East, Central) and Planning Area (i.e. places covered under Bedok, Bishan, Tampines etc.).

Wikipedia listener (automatically push to Github)

changes.py is a Python script that helps to detect changes to the Wikipedia page. The wikiChanges.txt file in git-helper-bot-update branch is the most updated version of the last Wikipedia page edit data. When running the script, if there are any changes (i.e. Wikipedia page last updated value is different from what we have in wikiChanges.txt), then the script will update the local wikiChanges.txt and push the updates to git-helper-bot-update.

Wikipedia listener with Crontab

You can also schedule changes.py to run at regular intervals with cron, which is how the branch git-helper-bot-update works. It uses github.com/git-helper-bot to push updates to the git-helper-bot-update branch. Read more about it here.

Demo

CLI Program

/path/to/directory/sg-places-list> python3 scripts/main.py

Type a command to choose a data structure. Invalid commands will be skipped:
0: list
1: dict

b: Go back to start
i: Show instructions again
q: Quit

$ 1

Choose the type of data you want:
0: By region only (e.g. east, west, north etc.)
1: By region and planning area (e.g. { east: { Bedok: [...] } })

b: Go back to start
i: Show instructions again
q: Quit

$ 1

Choose the region you want:
0: All
1: East
2: North
3: North East
4: Central
5: West

b: Go back to start
i: Show instructions again
q: Quit
You may combine them by listing (e.g. 1324 OR 35). If 0 is included, result will just return all items. Duplicates will be removed.

$ 1

{'east': {'Bedok': ['Bayshore', 'Bedok North', 'Bedok Reservoir', 'Bedok South', 'Frankel', 'Kaki Bukit', 'Kembangan', 'Siglap'], 'Changi': ['Changi Airport', 'Changi Point', 'Changi West'], 'Changi Bay': [], 'Pasir Ris': ['Flora Drive', 'Loyang East', 'Loyang West', 'Pasir Ris Central (Formerly called "Town" subzone.)', 'Pasir Ris Drive', 'Pasir Ris Park', 'Pasir Ris Wafer Fab Park (Formerly called "Pasir Ris West" subzone.)', 'Pasir Ris West (Formerly called "Elias" subzone.)'], 'Paya Lebar': ['Airport Road', 'Paya Lebar East', 'Paya Lebar North', 'Paya Lebar West', 'PLAB'], 'Tampines': ['Simei', 'Tampines East', 'Tampines North', 'Tampines West', 'Xilin']}}

$ q

changes.py

$ python3 scripts/scrapers/changes.py

08/30/2022, 16:09:02: A change is found:
Previous update: This page was last edited on 12 July 2022, at 12:13 (UTC).
Most recent update: This page was last edited on 29 July 2022, at 08:15 (UTC).
Text file updated successfully.
Successfully updated github repo at 08/30/2022, 16:09:05

$ python3 scripts/scrapers/changes.py
No changes detected.

When is this useful?

  • This is useful if you are writing a program which requires a list of key places in Singapore in string format only. Examples:
  1. Dropdown autocomplete suggestions for a search bar component
  2. Querying a list of places based on region (east, west etc.) and/or planning area (Bedok, Yishun etc.)
  • You may use scripts/tests.py to play around with the provided functions. You may modify the data structures by mapping them or you may even export it as CSV, Excel, etc.
  • This program only fetches the list of names in string format for each item (i.e. this is a quick and dirty way to easily get a list of key places in SG). If you need a much more detailed set of data for your use case (e.g. with coordinates, geocoding), please see Singapore's oneMap API instead.

How it works

Webscraper Python scripts built using the beautifulsoup Python library pull data from https://en.wikipedia.org/wiki/List_of_places_in_Singapore in HTML format. The program then cleans up and converts this data into the chosen data structures (list, dictByRegion, dictByRegionAndPlanning).

Built With

Getting Started

To get a local copy up and running follow these simple steps.

Installation

  1. Clone the repo 
    git clone https://github.com/raihahahan/singapore-places-names.git
cd singapore-places-names
  2. Install required packages 
    pip install -r requirements.txt
  3. Start the command line program 
    python3 scripts/main.py
  4. To work with changes.py, you will need to upload this clone into a Github repository, or create a fork instead. Make sure that git-helper-bot-update branch exists. Add a .env file to the root of the project and add in the following data: 
    githubPK=YOUR_GITHUB_ACCESS_TOKEN ## This is the Github account that will push to git-helper-bot-update branch. It can be a bot account.
user=YOUR_GITHUB_USERNAME ## This is the Github account that will push to git-helper-bot-update branch. It can be a bot account.
repoUser=YOUR_GITHUB_USERNAME ## This is the Github account that owns the cloned/forked repository.
fileLocation=ABSOLUTE_PATH_TO_wikiChanges.txt
  5. (optional) To schedule changes.py with Crontab, change the following in changes.py: 
    dotEnvLocation = ".env" ## CHANGE THIS TO ABSOLUTE PATH IF USING CRONTAB

Usage

The CLI will show instructions on how to use the program.

Type a command to choose a data structure. Invalid commands will be skipped:
 0: list
 1: dict

 b: Go back to start
 i: Show instructions again
 q: Quit

Roadmap

  • Remove "(Formerly known as...)" from list items if it exists.
  • Remove any square brackets from list items if it exists.

Contributing

Contributions are greatly appreciated. Please read the documentation first before starting. See the open issues for a list of proposed features and/or known issues.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

  1. Fork the Project
  2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
  3. Commit your Changes (git commit -m 'Add some AmazingFeature')
  4. Push to the Branch (git push origin feature/AmazingFeature)
  5. Open a Pull Request

License

Distributed under the MIT License. See LICENSE for more information.

Contact

Email: mraihandev@gmail.com

Acknowledgements