Shell scripts to work with git repos
This article explains the script that builds a Mac machine with "everything" needed by a professional developer.
This is a "bootstrapping" script to enable you to more easily manage the complexity of competing stacks of components and their different versions. Java, Python, Node and their most popular add-ons are covered here.
Logic in the script goes beyond what Homebrew does, and configures the component just installed:
- Install dependent components where necessary
- Display the version number installed (to a log)
- Add alias and paths in .bash_profile (if needed)
- Perform configuration (such as adding a missing file needed for mariadb to start)
- Edit configuration settings (such as changing default port within Nginx within config.conf file)
- Upgrade and uninstall if that is available
- Run a demo using the component to ensure that what has been installed actually works.
This bring DevSecOps-style "immutable architecture" to MacOS laptops. Immutability means replacing the whole machine instance instead of upgrading or repairing faulty components.
Like many, you may have an AirPort Time Capsule to back up everything using the MacOS Time Machine app. However, as this person discovered,
"I don't want all the crap I had in the old one."
This script helps you manage what you have installed so you can, but don't have to start from scratch.
Target users of this script are those who configure new laptops for developers joining the organization, so each developer doesn't waste days installing everything one at a time (and doing it differently than colleagues).
To do the above manually for each component would involve hours needlessly, and be error-prone.
Technical techniques for the Bash shell scripting are described separtely at Bash scripting page in this website.
This script references a folder in your Home folder named git-utilities, It contains a configuration file named secrets.sh which you edit to specify what you want installed and run. The file's name is suffixed with ".sh" because it is run to establish variables for the script to reference. You don't run the file yourself. It is run by script mac-install-all.sh which you initiate within a Terminal command line.
-
The starting point (generic version) of these files are in a public GitHub repository:
https://github.com/wilsonmar/git-utilities
If you know what I'm talking about and have a GitHub account, you may Fork the repo under your own account and, while in a Teminal window at your Home folder, git clone it locally under your Home folder. This approach would enable you to save your changes back up to GitHub under your own account.
Alternately, follow these steps to create an initial installation of what many developers use (but you won't be able to upload changes back to GitHub):
-
Triple-click this command and press command+C to copy to your invisible Clipboard:
sh -c "$(curl -fsSL https://raw.githubusercontent.com/wilsonmar/git-utilities/master/mac-install-all.sh)"
-
Open a Terminal and press keys command+V to paste it from Clipboard.
-
Press Enter to run it.
If your home folder does not contain a folder named "git-utilities", the script will create one by Git cloning, using a Git client it first installs if there isn't one already.
A folder is necessary to hold additional folders such as "hooks" used by Git (if marked for install.) File "mac-bash-profile.txt" contains starter entries to insert in ~/.bash_profile that is executed before MacOS opens a Terminal session. Ignore the other files.
-
Wait for the script to finish.
On a 4mbs network the run takes less than 5 minutes for a minimal install.
PROTIP: A faster network or a proxy nexus server providing installers within the firewall would speed things up a lot and ensure that vetted installers are used.
When the script ends it pops up a log file in the TextEdit program that comes with MacOS.
-
Within TextEdit, review the log file.
-
Close the log file.
-
click File and navigate to your Home folder then within git-utilities to open file secrets.sh in the repo so you can customize what you want installed.
textedit secrets.sh
PROTIP: The default specification in the file is for a "bare bones" minimal set of components. If you run it again, it will not install it again.
There is a key (variable name) for each category (MAC_TOOLS, etc.).
-
Among the comments (which begin with a pound sige) look for keywords for programs you want.
Keywords shown are for the most popular programs. The mac-install-all.sh script contains logic go get it setup fully (as summarized above).
-
Scroll to the bottom of the secrets.sh file and click between the double-quotes of TRYOUT="".
Paste or type the keyword of the components you want opened (invoked) by the script.
We don't want to automatically open every component installed because that would be overwhelming.
This way you have a choice.
-
Save the file. You need not exit the text editor completely if you want to re-run.
-
Run the script to carry out your changes:
chmod +x mac-install-all.sh mac-install-all.sh
There are several variations possible:
-
Upgrade to the latest versions of ALL components when "update" is added to the calling script:
chmod +x mac-install-all.sh mac-install-all.sh update
CAUTION: This often breaks things because some apps are not ready to use a newer dependency.
NOTE: This script does NOT automatically uninstall modules. But if you're brave enough, invoke the script this way to remove components so you recover some disk space:
mac-install-all.sh uninstall
This is not an option for components that add lines to ~/.bash_profile. It's quite dangerous because the script cannot differentiate whether customizations occured to what it installed.
There are lists of additional programs (components) you may elect to install.
-
At the Terminal, use TextEdit or a other text editor to view the script file:
textedit mac-install.sh
-
Press command+F to search for "others" (including the double-quotes).
PROTIP: Several categories have a list of brew commands to install additional components. (MAC_TOOLS, PYTHON_TOOLS, NODE_TOOLS, etc.)
-
For each additional component you want, delete the # to un-comment it.
Remember that each component installed takes more disk space.
Apps on Apple's App Store for Mac need to be installed manually. Popular apps include:
- Office for Mac 2016
- BitDefender for OSX
- CrashPlan (for backups)
- Amazon Music
- HockeyApp RDP (Remote Desktop Protocol client for controlling Microsoft Windows servers)
- Colloquy IRC client (at https://github.com/colloquy/colloquy)
- etc.
The brew "mas" manages Apple Store apps, but it only manages apps that have already been paid for. But mas does not install apps new to your Apple Store account.
Apps added by specifying in JAVA_TOOLS are GUI apps.
Most other Java dependencies are specified by manually added in each custom app's pom.xml file to specify what Maven downloads from the Maven Central online repository of installers at
http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.dbunit%22
Popular in the Maven Repository are:
-
yarn for code generation. JHipster uses it as an integrated tool in Java Spring development.
-
DbUnit extends the JUnit TestCase class to put databases into a known state between test runs. Written by Manuel Laflamme, DbUnit is added in the Maven pom.xml (or Ant) for download from Maven Central. See http://dbunit.wikidot.com/
-
mockito enables calls to be mocked as if they have been creted. Insert file java-mockito-maven.xml as a dependency to maven pom.xml See https://www.youtube.com/watch?v=GKUlQMrbtHE - May 28, 2016 and https://zeroturnaround.com/rebellabs/rebel-labs-report-go-away-bugs-keeping-your-code-safe-with-junit-testng-and-mockito/9/
-
TestNG See http://testng.org/doc/download.html and https://docs.mendix.com/howto/testing/create-automated-tests-with-testng
When using Gradle, insert file java-testng-gradle as a dependency to gradle working within Eclipse plug-in Build from source git://github.com/cbeust/testng.git using ./build-with-gradle
TODO: The Python edition of this will insert specs such as this in pom.xml files.
The script outputs logs to a file.
This is so that during runs, what appears on the command console are only what is relevant to debugging the current issue.
At the end of the script, the log is shown in an editor to enable search through the whole log.
To start the Jenkins server to a specified port:
<pre>jenkins --httpPort=$JENKINS_PORT &</pre>
The "&" puts the process in the background so that the script can continue running.
The response is a bunch of lines ending with "INFO: Jenkins is fully up and running".
Several other methods (which don't work now) are presented on the internet:
This blog, on Dec 29, 2014 recommends
sudo defaults write /Library/Preferences/org.jenkins-ci httpPort "$JENKINS_PORT" sudo launchctl unload /Library/LaunchDaemons/org.jenkins-ci.plist sudo launchctl load /Library/LaunchDaemons/org.jenkins-ci.plist
The command "jenkins" above is actually a bash script that invokes Java:
#!/bin/bash JAVA_HOME="$(/usr/libexec/java_home --version 1.8)" \ exec java -jar /usr/local/Cellar/jenkins/2.113/libexec/jenkins.war "$@"
The code within "$(...)" is run to obtain the value. In this case, it's:
<pre>/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home
</pre>
The link above is the folder where MacOS keeps the Java SDK. Java executables (java, javac, etc.) are in the bin folder below that location.
The path to jenkins.war and jenkins-cli.war executable files are physcally at:
ls /usr/local/opt/jenkins/libexec
Instead of specifying the port in the command, change the configuration file.
On MacOS, services are defined by plist files containing XML, such as this for Jenkins server:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>homebrew.mxcl.jenkins</string>
<key>ProgramArguments</key>
<array>
<string>/usr/libexec/java_home</string>
<string>-v</string>
<string>1.8</string>
<string>--exec</string>
<string>java</string>
<string>-Dmail.smtp.starttls.enable=true</string>
<string>-jar</string>
<string>/usr/local/opt/jenkins/libexec/jenkins.war</string>
<string>--httpListenAddress=127.0.0.1</string>
<string>--httpPort=8080</string>
</array>
<key>RunAtLoad</key>
<true/>
</dict>
</plist>
The "1.8" is the version of Java, described below.
The "httpPort=8080" default is customized using this variable in secrets.sh:
JENKINS_PORT="8082" # default 8080
The above is file homebrew.mxcl.jenkins.plist within folder /usr/local/opt/jenkins installed by brew. The folder is a symlink created by brew to the physical path where brew installed it:
/usr/local/Cellar/Jenkins/2.113/homebrew.mxcl.jenkins.plist
The "2.113" means that several versions of Jenkins can be installed side-by-side. This version number changes over time. So it is captured by command:
JENKINS_VERSION=$(jenkins --version) # 2.113
The folder is actually a symlnk which points to the physical folder defined by: JENKINS_CONF="/usr/local/Cellar/Jenkins/$JENKINS_VERSION/homebrew.mxcl.jenkins.plist"
The path is defined in a variable so simplify the sed command to make the change:
sed -i "s/httpPort=8080/httpPort=$JENKINS_PORT/g" $JENKINS_CONF
# --httpPort=8080 is default.
The command to view the server in the default internet browser (such as Safari, Chrome, etc.) is:
open "http://localhost:$JENKINS_PORT"
It's "http" and not "https" because a certificate has not been established yet.
When executed the first time, Jenkins displays this screen:
However, we don't want to open it from the command line script, but from a GUI automation script.
The script invokes a GUI automation script that opens the file mentioned on the web page above:
/Users/wilsonmar/.jenkins/secrets/initialAdminPassword
"/Users/wilsonmar" is represented by the environment variable named $HOME or ~ symbol, which would be different for you, with your own MacOS account name. Thus, the generic coding is:
JENKINS_SECRET=$(<$HOME/.jenkins/secrets/initialAdminPassword)
The file (and now $JENKINS_SECRET) contains a string in clear-text like "851ed535fd3249ab95a274d23242655c".
We then call a GUI automation script to get that string to paste it in the box labeled "Administrator Password" based on the id "security-token" defined in this HTML:
<input id="security-token" class="form-control" type="password" name="j_password">
This was determined by obtaining the outer HTML from Chrome Developer Tools.
The call is:
python tests/jenkins_secret_chrome.py chrome $JENKINS_PORT $JENKINS_SECRET
We use Selenium Python because it reads and writes system environment variables.
Use of Selenium and Python this way requires them to be installed before Jenkins and other web servers.
To shut down Jenkins,
PID="ps -A | grep -m1 'jenkins' | awk '{print $1}'"
fancy_echo "Shutting downn jenkins $PID ..."
kill $PID
The above is the automated approach to the manual on recommended by many blogs on the internet:
Some say in Finder look for Applications -> Utilities -> Activity Monitor
Others say use command:
ps -el | grep jenkins
Two lines would appear. One is the bash command to do the ps command.
The PID desired is the one that lists the path used to invoke Jenkins, described above:
/usr/bin/java -jar /usr/local/Cellar/jenkins/2.113/libexec/jenkins.war
kill 2134
That is the equivalent of Windows command "taskkill /F /PID XXXX"
There is also:
sudo service jenkins stop
Either way, the response expected is:
INFO: JVM is terminating. Shutting down Winstone
If the title is not found an error message like this appears on the console:
File "tests/jenkins_secret_chrome.py", line 30, in
assert "Jenkins [Jenkins]" in driver.title # bail out if not found.
AssertionError
Some put in a 5 second delay:
time.sleep(5)
Use of this feature requires a library to be specified at the top of the file:
import sys
Some also take a photo to "prove" that the result was achieved:
driver.save_screenshot('jenkins_secret_chrome.py' +utc_offset_sec+ '.png')
We put the name of the script file in the picture name to trace back to its origin. We put a time stamp in ISO 8601 format so that several png files sort by date.
utc_offset_sec = time.altzone if time.localtime().tm_isdst else time.timezone datetime.datetime.now().replace(tzinfo=datetime.timezone(offset=utc_offset_sec)).isoformat()
The long explanation is https://docs.python.org/2/library/datetime.html
NOTE:- webDriver.Close() - Close the browser window that currently has focus
- webDriver.Quit() - Calls Dispose()
- webDriver.Dispose() Closes all browser windows and safely ends the session
driver.quit() means that someone watching the script execute would only see the web app's screen for a split second.
We prefer to use id rather than name fields because the HTML standard states that id's are supposed to be unique in each web page.
Other similar scripts (listed in "References" below) run
http://groovy-lang.org/install.html
Dropbox, OneDrive, Google Drive, Amazon Drive
http://download.eclipse.org/releases/juno
Within Eclipse IDE, get a list of plugins at Help -> Install New Software -> Select a repo -> select a plugin -> go to More -> General Information -> Identifier
eclipse -application org.eclipse.equinox.p2.director \ -destination d:/eclipse/ \ -profile SDKProfile \ -clean -purgeHistory \ -noSplash \ -repository http://download.eclipse.org/releases/juno/ \ -installIU org.eclipse.cdt.feature.group, \ org.eclipse.egit.feature.group
"Equinox" is the runtime environment of Eclipse, which is the reference implementation of OSGI. Thus, Eclipse plugins are architectually the same as bundles in OSGI.
Notice that there are different versions of Eclipse repositories, such as "juno".
PROTIP: Although one can install several at once, do it one at a time to see if you can actually use each one. Some of them:
org.eclipse.cdt.feature.group, \ org.eclipse.egit.feature.group, \ org.eclipse.cdt.sdk.feature.group, \ org.eclipse.linuxtools.cdt.libhover.feature.group, \ org.eclipse.wst.xml_ui.feature.feature.group, \ org.eclipse.wst.web_ui.feature.feature.group, \ org.eclipse.wst.jsdt.feature.feature.group, \ org.eclipse.php.sdk.feature.group, \ org.eclipse.rap.tooling.feature.group, \ org.eclipse.linuxtools.cdt.libhover.devhelp.feature.feature.group, \ org.eclipse.linuxtools.valgrind.feature.group, \
NOTE: A feature group is a list of plugins and other features which can be understood as a logical separate project unit for the updates manager and for the build process.
Some developers have not put their stuff from GitHub into Homebrew. So we need to read (scrape) the website and see what is listed, then grab the text and URL to download.
Such is the situation with font files at https://github.com/adobe-fonts/source-code-pro/releases/tag/variable-fonts The two files desired downloaded using the curl command are:
- https://github.com/adobe-fonts/source-code-pro/releases/download/variable-fonts/SourceCodeVariable-Italic.ttf
- https://github.com/adobe-fonts/source-code-pro/releases/download/variable-fonts/SourceCodeVariable-Roman.ttf
The files are downloaded into where MacOS holds fonts available to all users: /Library/Fonts/
ITerm2 can make use of these font files.
-
https://github.com/paulirish/dotfiles/blob/master/brew-cask.sh (one of the earliest ones by a legend at Google)
-
https://github.com/andrewconnell/osx-install described at http://www.andrewconnell.com/blog/rapid-complete-install-reinstall-os-x-like-a-champ-in-three-ish-hours separates coreinstall.sh from myinstall.sh for personal preferences.
-
https://www.reddit.com/r/osx/comments/3u6mob/what_are_the_top_10_osx_applications_you_use/
-
https://github.com/jaywcjlove/awesome-mac/blob/master/README.md
-
https://medium.com/@ankushagarwal/maximize-developer-productivity-on-a-mac-a9ae6fbaedab