AfNOG 2019 Workshop on Network Technology

Track SS-E: Scalable Internet Services

Introduction

This course on Scalable Internet Services in English (SS-E) is part of the AfNOG 2019 Workshop on Network Technology, held in conjunction with the AfNOG meeting in Dakar, Senegal, April-May 2019.

We use hands-on training in a well-equipped classroom over a five-day period to teach skills required for the configuration and operation of large scale Internet services.

Who should attend

Technical staff who are now providing Internet Services, or those who will be involved in the establishment and/or provisioning of basic Internet Services.

Prerequisites

Experience using and administering *NIX Servers, Name Servers, Web Servers, Mail Servers and similar services. Knowledge of Networking is a plus!

Help!

You can get free help with any of the tutorials or materials on this site by joining the AfNOG mailing list and sending an email to afnog@afnog.org with your question or problem. Please note that you must subscribe to the list first, otherwise your post will be rejected automatically.

Instructors

Participants

Timetable

Management

Details for project management of the SS-E workshop.

Editing this page

Please file an issue requesting to be added as an administrator of the AfNOG organisation on GitHub.

Mailing list

The instructors group on Google Groups:

• Email address: afnog-sse@googlegroups.com
• Web interface: https://groups.google.com/forum/#!forum/afnog-sse

There is a wiki page on course development.

Equipment

To host this track you will probably need the following equipment:

• 2 x Mac Minis or similar, quad core i7 2Ghz+, 16GB Ram, 250 GB SSD (to host 16 virtual machines each)
• Projector: VGA required, HDMI optional, screen/wall. Mac VGA and HDMI adaptors.
• Wifi: ideally wired and wireless on the same SSID with /24.
• Wired Ethernet ports: probably 4-8 ports for people with broken wireless and for instructors in our classroom.
• Power strips: 12 x 4 socket.
• Spare machines: 4 x reasonable desktop/laptop with 1 GB RAM and permission to reformat and install Ubuntu or FreeBSD.
• Sun shading: to be able to read the projected screen and not overheat in our rooms.
• White board, pens and eraser: at least 3 pens in 2 different colours.

Setup

We usually use LXC containers, although not everything is supported (e.g. you can’t run an iSCSI Target in an LXC container yet), so there is an alternative setup using KVM virtual machines. These are much heavier-weight, so you can’t run as many on the same host (especially RAM is a limiting factor, since each VM needs ~512 MB of dedicated RAM).

• Server setup (Mac Mini)
• LXC setup for ~40 containers
• KVM setup for ~24 containers (alternative)
• Admin howto for the containers

Administration

All done by CW unless anyone else wants to.

• Student numbers and names
• Classroom setup, networking, virtual machine images, cable management
• Notices - door, timetable, complaints box, wifi password
• Introductory talk - welcome, topic poll, complaints box
• Time management during the workshop (breaks, lunch, etc)
• Set alarms for break times
• Ensure that every topic has an instructor and enough time allocated
• Ensure that participants are receiving any assistance necessary
• Student name verification for certificates
• Coordinate the class group photo
• Liaise with the secretariat on any other issues that may be required
• Ensure that all course materials are placed on the workshop folder for CD burning at the end of the workshop
• Ensure that the participants complete and return the feedback form
• Download and serve any files needed, e.g. FreeBSD ISO images (for virtualisation) and packages (for pkg_add mirror)

TODO

Meta (about this site)

Quick Start (editing)

If someone else is hosting, then all you have to do is:

• Request write access to the repositories below, or clone them (and use the clone URLs instead)
• Install SparkleShare
• Add git@github.com:afnog/sse.git to it
• Edit the Markdown files in ~/SparkleShare/sse/.../*.md
• The host will detect your changes, generate the HTML and upload to Github and the local server.

Quick Start (hosting)

Only one person should host a repository, otherwise you will duplicate work and maybe have race conditions (although you should both be producing and updating the same content, so it should actually work if you have more than one, but there’s no point or advantage to it.)

If you want to host, you need a Linux or Mac desktop with lsyncd, rsync and ruby installed. Then follow these steps:

• Add git@github.com:afnog/afnog.github.io.git to SparkleShare
• Install Jekyll for Ubuntu/Debian: sudo gem install jekyll execjs therubyracer
• Open a command prompt and go to ~/SparkleShare/sse
• Run make serve
• View the results in your browser at http://localhost:4000/ (generated HTML files)

Syncing to the Afnog webserver (ws.afnog.org) which runs FreeBSD

For updates to automatically sync to the webserver in use at the AFNOG workshop, do the following:

• Login to the NOC Box
• Install Jekyll sudo pkg install rubygem-jekyll
• Install the Jekyll gem if necessary sudo gem install jekyll
• Go to (replace XX with year) cd /u/vol/www/vhosts/www.ws.afnog.org/data/afnog20XX
• sudo mkdir sse-git-afnog20XX
• cd sse-git-afnog20XX/
• git init
• git pull -p https://github.com/afnog/sse.git
• jekyll build --destination /u/vol/www/vhosts/www.ws.afnog.org/data/afnog20XX/sse/
• look at this script for a sample cron job

Source code (Markdown)

• Latest master is in GitHub.
• Edit online using GitHub’s web editor, or:
• Clone an offline copy with Markdown source (no HTML) at git@github.com:afnog/sse.git.
• The syntax is parsed with Kramdown, which adds some useful extensions.

Generated HTML

• HTML files are auto-generated from Markdown by Jekyll - do not edit by hand!
• All files except those starting with --- (front matter) are copied literally from the source (sse) repository.
• Possibly outdated copy in GitHub, browsable online at http://afnog.github.io/sse/.
• Clone an offline copy (HTML, not Markdown) at git@github.com:afnog/afnog.github.io.git.
• Some variables are stored in _config.yml, e.g. the year, and used with { { page.year } } in HTML and Markdown files (spaces added to stop Markdown from replacing this with the year number!)

Presentations

Presentations use a special format to invoke remark on the Markdown source files:

• The Markdown source is called presentation.md (so there can be only one per directory).
• In the same directory is a file called index.md, which tells Jekyll to use a specific layout (template file) to generate the HTML: _layouts/presentation.html.
• This file is generic and the same for all presentations. It loads the Remark source code, and then loads the presentation.md file from the same directory using AJAX. So the URL that you use to load it is very important in locating the correct presentation.md file.
• This means that you cannot use Kramdown extensions in presentations. No presentation.html files are generated because the presentation.md files deliberately do not have a “front matter” section which Jekyll requires.

Generating the HTML

If you’re using Ruby 1.8, you may need to install Ruby 1.9 first.

Warning: This command by default will overwrite ../afnog.github.io/sse, since it assumes that you have both https://github.com/afnog/sse/ and https://github.com/afnog/afnog.github.io/ checked out side-by-side (for example in SparkleShare).

If you want it to overwrite a different directory (where it will write the generated HTML files), you can specify it as a command-line argument to Make:

make DST_DIR=/tmp/site

You will need to install Jekyll to generate the HTML files:

sudo gem install jekyll execjs therubyracer

Then run make to build them once, in the destination directory:

make

Or run make watch to tell Jekyll to stay running, watch for source files changing, and generate a new HTML file when they do (ideal for modifying presentations on the fly):

make watch

Publishing the HTML

You can use make sync to run lsyncd (which you must have installed, for example with brew install lsyncd) to automatically rsync the content to the workshop server, http://www.ws.afnog.org. You will need to check the SYNC_HOST and SYNC_DIR in the Makefile, which must point to the destination host and directory which will be overwritten.

make sync

Automatic publishing

If you want to have a server automatically fetch changes from Git and update the static website, you can’t use SparkleShare on the server because it’s a GUI tool. What you can do is checkout both Git repositories, using a fresh personal access token to give it write access to the afnog.github.io repository, and “git pull” in a loop (or automatically from Cron), generate the HTML and commit and publish it if different (which is almost what SparkleShare does):

git config --global user.name "Your Name"
git config --global user.email you@example.com
sudo apt install lsyncd rsync ruby ruby-dev gcc g++ make
sudo gem install jekyll execjs therubyracer
mkdir ~/website
cd ~/website
git clone https://github.com/afnog/sse.git
git clone https://<your GitHub username>:<your token>@github.com/afnog/afnog.github.io.git
git config --global push.default simple
while true; do cd ~/website/sse; git pull; make autocommit; sleep 5; done

Or replace the last line with a Cron job:

* * * * *	cd ~/website/sse; git pull; make autocommit