Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix citations and travis build #100

Merged

Conversation

JamesOwers
Copy link
Contributor

@JamesOwers JamesOwers commented Jan 11, 2021

Issue

Unfortunately, pandoc-citeproc is depreciated; citeproc-hs is now installed by default and used by pandoc. This means the installation (either via travis, install.sh, or hacking through the README.md) is broken (because installing pandoc-citeproc kicks a depreciation error).

If the user proceeds without pandoc-citeproc, citations don't work!

Fix

  • Fix and extend travis build
    • tests both mac and linux now linux needs work to install texlive (see comments below, it's 90% done though)
  • Fix install script by removing pandoc-citeproc, and
    • made separate files for mac and linux (in case they have different requirements in future)
    • make install accounts for the diff files and uses the correct one automatically
  • Make citations work for the example generated from source/*.md by:
    • Change commands in Makefile to use --citeproc argument
    • Fix style/template.tex with new contents of pandoc -D latex relating to citations

Other minor contributions

  • Extended examples in source/*.md files including:
    • some advanced equations (showing how to back off to latex if needed)
    • section referencing with pandoc-secnos
  • Tidy commands in Makefile to highlight differences and make the output format consistent
  • Version control all output types thesis.{pdf,tex,html,docx}, such that it is obvious if changes are made (not true for docx, so there's an argument we shouldn't version control that file since it is large)
  • Add a target to Makefile so the user can make all to generate thesis.{pdf,tex,html,docx} with one command
  • Output pandoc logs to separate files: `pandoc.{pdf,tex,html,docx}.log
  • Extended the README.md:
    • Add quickstart for mac users
    • Add high level explanation of how pandoc processes all the files
    • Add troubleshooting

Pandoc is not installed including citeproc.
@JamesOwers JamesOwers mentioned this pull request Jan 11, 2021
@JamesOwers JamesOwers changed the title Remove pandoc-citeproc install Fix citations and travis build Jan 13, 2021
@JamesOwers JamesOwers marked this pull request as ready for review January 14, 2021 07:10
@JamesOwers
Copy link
Contributor Author

JamesOwers commented Jan 14, 2021

This is 99% there. Travis build passes for all commands bar make pdf, which is erroring with

LaTeX Error: File `l3backend-xetex.def' not found.

If this can be fixed, this will be a decent PR to merge ☺️. @tompollard - please may you review?

Current stack trace for the error
[makePDF] Run #1
This is XeTeX, Version 3.14159265-2.6-0.999992 (TeX Live 2020) (preloaded format=xelatex)
 restricted \write18 enabled.
entering extended mode
(/private/var/folders/nz/vv4_9tw56nv9k3tkvyszvwg80000gn/T/tex2pdf.-1ca2a0397ad4
70e2/input.tex
LaTeX2e <2020-02-02> patch level 5
L3 programming layer <2021-01-09>
(/usr/local/texlive/2020/texmf-dist/tex/latex/base/report.cls
Document Class: report 2019/12/20 v1.4l Standard LaTeX document class
(/usr/local/texlive/2020/texmf-dist/tex/latex/base/size12.clo))
(/usr/local/texlive/2020/texmf-dist/tex/latex/lm/lmodern.sty)
(/usr/local/texlive/2020/texmf-dist/tex/latex/float/float.sty)
(/usr/local/texlive/2020/texmf-dist/tex/latex/truncate/truncate.sty)
(/usr/local/texlive/2020/texmf-dist/tex/latex/caption/caption.sty
(/usr/local/texlive/2020/texmf-dist/tex/latex/caption/caption3.sty
(/usr/local/texlive/2020/texmf-dist/tex/latex/graphics/keyval.sty)))
(/usr/local/texlive/2020/texmf-dist/tex/latex/tocloft/tocloft.sty)
(/usr/local/texlive/2020/texmf-dist/tex/latex/amsfonts/amssymb.sty
(/usr/local/texlive/2020/texmf-dist/tex/latex/amsfonts/amsfonts.sty))
(/usr/local/texlive/2020/texmf-dist/tex/latex/amsmath/amsmath.sty
For additional information on amsmath, use the `?' option.
(/usr/local/texlive/2020/texmf-dist/tex/latex/amsmath/amstext.sty
(/usr/local/texlive/2020/texmf-dist/tex/latex/amsmath/amsgen.sty))
(/usr/local/texlive/2020/texmf-dist/tex/latex/amsmath/amsbsy.sty)
(/usr/local/texlive/2020/texmf-dist/tex/latex/amsmath/amsopn.sty))
(/usr/local/texlive/2020/texmf-dist/tex/generic/iftex/ifxetex.sty
(/usr/local/texlive/2020/texmf-dist/tex/generic/iftex/iftex.sty))
(/usr/local/texlive/2020/texmf-dist/tex/generic/iftex/ifluatex.sty)
(/usr/local/texlive/2020/texmf-dist/tex/latex/base/fixltx2e.sty
Package fixltx2e Warning: fixltx2e is not required with releases after 2015
(fixltx2e)                All fixes are now in the LaTeX kernel.
(fixltx2e)                See the latexrelease package for details.
) (/usr/local/texlive/2020/texmf-dist/tex/xelatex/mathspec/mathspec.sty
(/usr/local/texlive/2020/texmf-dist/tex/latex/etoolbox/etoolbox.sty)
(/usr/local/texlive/2020/texmf-dist/tex/latex/fontspec/fontspec.sty
(/usr/local/texlive/2020/texmf-dist/tex/latex/l3packages/xparse/xparse.sty
(/usr/local/texlive/2020/texmf-dist/tex/latex/l3kernel/expl3.sty
! LaTeX Error: File `l3backend-xetex.def' not found.
Type X to quit or <RETURN> to proceed,
or enter new name. (Default extension: def)
Enter file name: 
! Emergency stop.
<read *> 
         
l.148   { \sys_load_backend:n { } }
                                    
No pages of output.
Transcript written on /private/var/folders/nz/vv4_9tw56nv9k3tkvyszvwg80000gn/T/
tex2pdf.-1ca2a0397ad470e2/input.log.
[makePDF] Run #2
This is XeTeX, Version 3.14159265-2.6-0.999992 (TeX Live 2020) (preloaded format=xelatex)
 restricted \write18 enabled.
entering extended mode
(/private/var/folders/nz/vv4_9tw56nv9k3tkvyszvwg80000gn/T/tex2pdf.-1ca2a0397ad4
70e2/input.tex
LaTeX2e <2020-02-02> patch level 5
L3 programming layer <2021-01-09>
(/usr/local/texlive/2020/texmf-dist/tex/latex/base/report.cls
Document Class: report 2019/12/20 v1.4l Standard LaTeX document class
(/usr/local/texlive/2020/texmf-dist/tex/latex/base/size12.clo))
(/usr/local/texlive/2020/texmf-dist/tex/latex/lm/lmodern.sty)
(/usr/local/texlive/2020/texmf-dist/tex/latex/float/float.sty)
(/usr/local/texlive/2020/texmf-dist/tex/latex/truncate/truncate.sty)
(/usr/local/texlive/2020/texmf-dist/tex/latex/caption/caption.sty
(/usr/local/texlive/2020/texmf-dist/tex/latex/caption/caption3.sty
(/usr/local/texlive/2020/texmf-dist/tex/latex/graphics/keyval.sty)))
(/usr/local/texlive/2020/texmf-dist/tex/latex/tocloft/tocloft.sty)
(/usr/local/texlive/2020/texmf-dist/tex/latex/amsfonts/amssymb.sty
(/usr/local/texlive/2020/texmf-dist/tex/latex/amsfonts/amsfonts.sty))
(/usr/local/texlive/2020/texmf-dist/tex/latex/amsmath/amsmath.sty
For additional information on amsmath, use the `?' option.
(/usr/local/texlive/2020/texmf-dist/tex/latex/amsmath/amstext.sty
(/usr/local/texlive/2020/texmf-dist/tex/latex/amsmath/amsgen.sty))
(/usr/local/texlive/2020/texmf-dist/tex/latex/amsmath/amsbsy.sty)
(/usr/local/texlive/2020/texmf-dist/tex/latex/amsmath/amsopn.sty))
(/usr/local/texlive/2020/texmf-dist/tex/generic/iftex/ifxetex.sty
(/usr/local/texlive/2020/texmf-dist/tex/generic/iftex/iftex.sty))
(/usr/local/texlive/2020/texmf-dist/tex/generic/iftex/ifluatex.sty)
(/usr/local/texlive/2020/texmf-dist/tex/latex/base/fixltx2e.sty
Package fixltx2e Warning: fixltx2e is not required with releases after 2015
(fixltx2e)                All fixes are now in the LaTeX kernel.
(fixltx2e)                See the latexrelease package for details.
) (/usr/local/texlive/2020/texmf-dist/tex/xelatex/mathspec/mathspec.sty
(/usr/local/texlive/2020/texmf-dist/tex/latex/etoolbox/etoolbox.sty)
(/usr/local/texlive/2020/texmf-dist/tex/latex/fontspec/fontspec.sty
(/usr/local/texlive/2020/texmf-dist/tex/latex/l3packages/xparse/xparse.sty
(/usr/local/texlive/2020/texmf-dist/tex/latex/l3kernel/expl3.sty
! LaTeX Error: File `l3backend-xetex.def' not found.
Type X to quit or <RETURN> to proceed,
or enter new name. (Default extension: def)
Enter file name: 
! Emergency stop.
<read *> 
         
l.148   { \sys_load_backend:n { } }
                                    
No pages of output.
Transcript written on /private/var/folders/nz/vv4_9tw56nv9k3tkvyszvwg80000gn/T/
tex2pdf.-1ca2a0397ad470e2/input.log.
[makePDF] Run #3
This is XeTeX, Version 3.14159265-2.6-0.999992 (TeX Live 2020) (preloaded format=xelatex)
 restricted \write18 enabled.
entering extended mode
(/private/var/folders/nz/vv4_9tw56nv9k3tkvyszvwg80000gn/T/tex2pdf.-1ca2a0397ad4
70e2/input.tex
LaTeX2e <2020-02-02> patch level 5
L3 programming layer <2021-01-09>
(/usr/local/texlive/2020/texmf-dist/tex/latex/base/report.cls
Document Class: report 2019/12/20 v1.4l Standard LaTeX document class
(/usr/local/texlive/2020/texmf-dist/tex/latex/base/size12.clo))
(/usr/local/texlive/2020/texmf-dist/tex/latex/lm/lmodern.sty)
(/usr/local/texlive/2020/texmf-dist/tex/latex/float/float.sty)
(/usr/local/texlive/2020/texmf-dist/tex/latex/truncate/truncate.sty)
(/usr/local/texlive/2020/texmf-dist/tex/latex/caption/caption.sty
(/usr/local/texlive/2020/texmf-dist/tex/latex/caption/caption3.sty
(/usr/local/texlive/2020/texmf-dist/tex/latex/graphics/keyval.sty)))
(/usr/local/texlive/2020/texmf-dist/tex/latex/tocloft/tocloft.sty)
(/usr/local/texlive/2020/texmf-dist/tex/latex/amsfonts/amssymb.sty
(/usr/local/texlive/2020/texmf-dist/tex/latex/amsfonts/amsfonts.sty))
(/usr/local/texlive/2020/texmf-dist/tex/latex/amsmath/amsmath.sty
For additional information on amsmath, use the `?' option.
(/usr/local/texlive/2020/texmf-dist/tex/latex/amsmath/amstext.sty
(/usr/local/texlive/2020/texmf-dist/tex/latex/amsmath/amsgen.sty))
(/usr/local/texlive/2020/texmf-dist/tex/latex/amsmath/amsbsy.sty)
(/usr/local/texlive/2020/texmf-dist/tex/latex/amsmath/amsopn.sty))
(/usr/local/texlive/2020/texmf-dist/tex/generic/iftex/ifxetex.sty
(/usr/local/texlive/2020/texmf-dist/tex/generic/iftex/iftex.sty))
(/usr/local/texlive/2020/texmf-dist/tex/generic/iftex/ifluatex.sty)
(/usr/local/texlive/2020/texmf-dist/tex/latex/base/fixltx2e.sty
Package fixltx2e Warning: fixltx2e is not required with releases after 2015
(fixltx2e)                All fixes are now in the LaTeX kernel.
(fixltx2e)                See the latexrelease package for details.
) (/usr/local/texlive/2020/texmf-dist/tex/xelatex/mathspec/mathspec.sty
(/usr/local/texlive/2020/texmf-dist/tex/latex/etoolbox/etoolbox.sty)
(/usr/local/texlive/2020/texmf-dist/tex/latex/fontspec/fontspec.sty
(/usr/local/texlive/2020/texmf-dist/tex/latex/l3packages/xparse/xparse.sty
(/usr/local/texlive/2020/texmf-dist/tex/latex/l3kernel/expl3.sty
! LaTeX Error: File `l3backend-xetex.def' not found.
Type X to quit or <RETURN> to proceed,
or enter new name. (Default extension: def)
Enter file name: 
! Emergency stop.
<read *> 
         
l.148   { \sys_load_backend:n { } }
                                    
No pages of output.
Transcript written on /private/var/folders/nz/vv4_9tw56nv9k3tkvyszvwg80000gn/T/
tex2pdf.-1ca2a0397ad470e2/input.log.
Error producing PDF.
! LaTeX Error: File `l3backend-xetex.def' not found.
Type X to quit or <RETURN> to proceed,
or enter new name. (Default extension: def)
Enter file name: 
! Emergency stop.
<read *> 
         
l.148   { \sys_load_backend:n { } }
My next step would be to check the issue is not due to travis brew being out of date...

Currently in the build log, we have:

Error: 
  homebrew-core is a shallow clone.
  homebrew-cask is a shallow clone.
To `brew update`, first run:
  git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core fetch --unshallow
  git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask fetch --unshallow
These commands may take a few minutes to run due to the large size of the repositories.
This restriction has been made on GitHub's request because updating shallow
clones is an extremely expensive operation due to the tree layout and traffic of
Homebrew/homebrew-core and Homebrew/homebrew-cask. We don't do this for you
automatically to avoid repeatedly performing an expensive unshallow operation in
CI systems (which should instead be fixed to not use shallow clones). Sorry for
the inconvenience!

Travis config currently has:

addons:
   ...
  homebrew:
    casks:
    - mactex
    update: true 

So google around this issue.

@JamesOwers
Copy link
Contributor Author

JamesOwers commented Jan 14, 2021

For .travis.yml, linux build is nearly there now too, stuck on:

$ make install
bash /home/travis/build/tompollard/phd_thesis_markdown/install_linux.sh
(running on Debian, switching to user mode!)
cannot setup TLPDB in /home/travis/texmf at /usr/bin/tlmgr line 5604.
Makefile:37: recipe for target 'install' failed
make: *** [install] Error 2
The command "make install" failed and exited with 2 during .

Edit

Reviewing advice here and the linked linux texlive install advice it links to here, it may be best to comment out the linux part of the travis build and raise a separate PR for that.

Next steps for linux

Follow advice here to install a vanilla distribution: https://www.tug.org/texlive/debian.html

Hopefully this solves some issues and maybe even cuts the build time
This issue suggests we could be using an old install of texlive.
Conda should be putting pandoc on the path anyway. I wonder how it
choses what tex version to use.
https://tex.stackexchange.com/questions/567050/how-can-i-install-l3backend-pdfmode-def
Trying this to compare with previous build
Else we get the following stack:
```
brew
Installing Homebrew Packages
6.51s$ rvm $brew_ruby do brew bundle --verbose --global
/usr/local/bin/brew tap homebrew/bundle
==> Tapping homebrew/bundle
Cloning into '/usr/local/Homebrew/Library/Taps/homebrew/homebrew-bundle'...
remote: Enumerating objects: 101, done.
remote: Counting objects: 100% (101/101), done.
remote: Compressing objects: 100% (95/95), done.
remote: Total 101 (delta 4), reused 28 (delta 1), pack-reused 0
Receiving objects: 100% (101/101), 51.37 KiB | 2.23 MiB/s, done.
Resolving deltas: 100% (4/4), done.
Tapped (109 files, 268.1KB).
Error: Unknown command: bundle
before_install.1
0.01s$ echo $PATH
0.05s$ if [ "$TRAVIS_OS_NAME" = "osx" ]; then
    eval "$(/usr/libexec/path_helper)"
    sudo tlmgr update --self
  elif [ "$TRAVIS_OS_NAME" = "linux" ]; then
    tlmgr init-usertree
    echo MSG FROM JO:
    echo on linux we cannot update apt installed texlive
    echo next steps to fix see commit msg: tompollard@85fd9e7
    echo c.f. https://tex.stackexchange.com/questions/551383/cant-run-tex-lives-tlmgr-in-a-github-action
    echo and https://tex.stackexchange.com/questions/1092/how-to-install-vanilla-texlive-on-debian-or-ubuntu

  fi

sudo: tlmgr: command not found
```
@JamesOwers
Copy link
Contributor Author

Ok, that's the build fixed for mac. Paved the way for fixing for linux next PR.

Added:
* Troubleshooting
* More info to "how does it work"?
Also added comments re updating all
* Section references - chapter 1
  * Uses plain LaTeX \ref{...} to indicate how to refer to each section.
* Use of URL links - chapter 1

Still todo:
* footnotes
* label arbitrary sections like it was an equation (e.g. the code snippet)
I think this makes sense, given that we are tracking both thesis.pdf (a large
binary file), and thesis.html (a file which refers to these images).
Debatable whether this is really required. I do it for completeness. I don't
know if we will be able to track changes the the file i.e. will it stay the same
if we generate from the same markdown?
@dendrondal
Copy link
Collaborator

This is phenomenal work! Thank you for putting the time in to make all of these improvements. There are a few other PRs (#97 and #96) that I'm also wanting to merge, but I'm going to review yours first, as I think having a more through CI process will mean less headaches down the road. This will probably take a second, so bear with me.

@@ -37,6 +37,8 @@ For bold, add ** on either side of the text
For bold and italic, add _** on either side of the text
-->

This is a brief outline of what went into each chapter. **Chapter 1** gives a background on duis tempus justo quis arcu consectetur sollicitudin. **Chapter 2** discusses morbi sollicitudin gravida tellus in maximus. **Chapter 3** discusses vestibulum eleifend turpis id turpis sollicitudin aliquet. **Chapter 4** shows how phasellus gravida non ex id aliquet. Proin faucibus nibh sit amet augue blandit varius.
This is a brief outline of what went into each chapter, and a section which shows how to reference headers (which are labelled automatically for you). This chapter, +@sec:introduction-with-a-citation, shows how to use citations and how to reference section headers. \*@sec:literature-review-with-maths shows how use and reference equations. \*@sec:first-research-study-with-code shows how to use and reference code. \*@sec:research-containing-a-figure shows how to use, reference, and resize pdf and jpg figures. \*@sec:research-containing-a-table shows how to use and reference tables. \*@sec:final-research-study is truly revolutionary (but shows nothing functional). **[Appendix 1](#appendix-1-some-extra-stuff)** shows how to add chapters which are not numbered, and has to be referenced manually, as does **[Appendix 2](#appendix-2-some-more-extra-stuff)**. See the base [`README.md`](https://github.com/tompollard/phd_thesis_markdown/blob/master/README.md) for how References are handled - leave `*_references.md` alone, and provide it to `pandoc` last.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kudos for adding this, I didn't realize we had no secnos examples in here.

@@ -152,13 +158,36 @@
\fi
$endif$

% $if(csl-refs)$
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the purpose of this block?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good question - I just looked at what the current pandoc standard template was and fiddled until I fixed things! It was necessary to make the references section to actually work.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check out pandoc -D latex to see what the standard template currently is.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh, interesting. Whatever works, and it certainly doesn't seem to be breaking anything!

@dendrondal dendrondal mentioned this pull request Jan 28, 2021
@JamesOwers
Copy link
Contributor Author

This is phenomenal work! Thank you for putting the time in to make all of these improvements. There are a few other PRs (#97 and #96) that I'm also wanting to merge, but I'm going to review yours first, as I think having a more through CI process will mean less headaches down the road. This will probably take a second, so bear with me.

As an aside - I'm v happy to help in future since I'll be using this template to write my phd actively. The linux travis build is nearly there, I could have a crack at that next.

@dendrondal
Copy link
Collaborator

As an aside - I'm v happy to help in future since I'll be using this template to write my phd actively. The linux travis build is nearly there, I could have a crack at that next.

Absolutely! If you want to add docker integration to the travis.yml also (compatible with #97, see #96 for an example), that would also be appreciated, and well within that scope. Always happy to have more contributors, and may the writing process be as painless as possible!

@JamesOwers
Copy link
Contributor Author

As an aside - I'm v happy to help in future since I'll be using this template to write my phd actively. The linux travis build is nearly there, I could have a crack at that next.

Absolutely! If you want to add docker integration to the travis.yml also (compatible with #97, see #96 for an example), that would also be appreciated, and well within that scope. Always happy to have more contributors, and may the writing process be as painless as possible!

Yeah will do. Never used docker before so should be fun. This PR is complicated enough so I’ll wait for this to merge (fixing any changes you’d like), then raise a separate PR for Linux + docker travis improvements.

@dendrondal
Copy link
Collaborator

I definitely think it's a good idea to have that as a separate request. I think this is ready to merge.

@dendrondal dendrondal merged commit 7c882bf into tompollard:master Jan 30, 2021
@tompollard
Copy link
Owner

tompollard commented Jan 30, 2021

@JamesOwers @dendrondal thanks for all of your work on this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants