This guide has been developed as support material for March 5, 2021, online seminar “Singularity Container Capabilities”. This event has been made possible by the EuroHPC project “National Competence Centers in the framework of EuroHPC” (EuroCC). This project has started on September 1, 2020, and Latvia is represented in this project by the HPC Center of Riga Technical University together with the Institute of Numerical Modelling of the University of Latvia. The EuroCC project aims to create a European network of supercomputing competence centres. 33 countries are involved in the project; it will last for 2 years with total funding of more than 56 million EUR.

The task of the competence centres established during the project will be to create a unified support structure to promote the use of supercomputing opportunities in higher education, research, public administration, and industry. The Competence Centres will bring together the competencies, experience, and computing resources available in all EU countries.

Introduction

When using high-performance computing (HPC) resources, it often becomes important to use non-standard software that is not available in the form of installed, ready-to-use software modules. In this situation, you can ask the administrator for help installing the software. But often this does not completely solve the problem:

• The desired software may not be compatible with the operating system used in the cluster.
• Many different software packages with specific versions need to be installed.
• The user may want to use an identical environment on different clusters/workstations.

In the described situation, it is very suitable to use the software in the form of containers – a fully equipped and encapsulated software environment, which allows you to perform your desired calculation tasks on different systems. To use the required software environment (container), all you have to do is copy/obtain the desired container file.

Demonstration of basic functionality

Below is an example of the basic functionality of software containers on the LU INM HPC cluster. The example considers a case where a container with Anaconda Python is already stored in a user’s folder (a tool with a series of Python tools that is typically used to process HPC data with Python).

# See that base OS offers only old and incomplete Python version (2.x.x.)
python --version

# It is possible to use software module provided by the administrators
# See that the module indeed offers feature-rich Anaconda Python (3.x.x.)
module avail
module load anaconda3/anaconda-2020.11
python --version

# However, there is an alternative – use of containers!
# Detach previously loaded modules
module purge

# Load 'singularity' module
module load singularity/3.4.1

# Use of available container (for executing a single command)
# See that, indeed, container features Anaconda Python (3.x.x.)
singularity exec ~/my_containers/my_anaconda.simg python --version

# Use of available container (for interactive shell)
singularity shell ~/my_containers/my_anaconda.simg
python --version

# While being in the container, see that all the user files are accessible
ls ~

From the example shown, it can be seen that containers provide the use of a particular software environment. However, it is not as cumbersome as using a separate virtual machine, which is usually not possible or suitable for HPC clusters. It should be noted that the use of containers can, of course, be specified in queue system scripts like any other command.

Singularity and Docker comparison

In recent years, software containerization has gained popularity in the form of the widely used platform Docker – the community of this platform has created many different, freely available container images in the DockerHub repository. However, there is an alternative containerization solution that is more suitable and popular in the HPC environment: Singularity. This solution has several advantages that make it particularly suitable for HPC:

• Unlike Docker, the use of Singularity containers does not require administrator rights. This allows you to make full use of these containers in an HPC environment. (Administrator rights are still required to create containers, however.)
• Singularity containers offer ready-made solutions for parallel calculations (OpenMPI) as well as GPU calculations.
• Singularity is compatible with Docker containers, allowing you to take advantage of the points above while allowing the use of the DockerHub repository.

Based on these advantages, as well as the fact that Singularity is available on both the LU INM cluster and the RTU HPC Center cluster, this guide specifically addresses the use of Singularity containers.

Obtaining DockerHub containers

The previous demonstration with the Anaconda Python container was possible because it was obtained from the DockerHub repository. To obtain/build this container file, proceed as follows:

# Load 'singularity' module
module load singularity/3.4.1

# Obtain container using the link seen on DockerHub
# ‘pull’ will download the container within few seconds
singularity pull ~/my_containers/my_anaconda.simg docker://continuumio/anaconda3

# Optionally – you can also use ‘build’ command
# This converts the image to the newest available image format
# In this case, build takes around 3 min
singularity build ~/my_containers/my_anaconda.simg docker://continuumio/anaconda3

The DockerHub repository contains a number of different containers, which contain sets of software tools suitable for typical tasks in various fields – physics calculations, data processing, and IT solution tools (e.g., databases). However, there are several repositories besides DockerHub, where software containers are freely available.

Installing Singularity and creating containers from scratch

Despite the extensive repository resources, sometimes it is necessary to create your own, specific container. Because this operation requires administrator privileges, it would typically be performed on your personal computer (or a Linux virtual machine available to you).

The example below considers the creation of a container for using ESPResSo software, a specific tool for soft matter physics simulations. Installing this software requires several different additional tools, which can be time-consuming for the user and HPC cluster administrators. This situation makes the ESPResSo program a great candidate for a container demonstration.

Installing Singularity on your personal computer

To be able to build your own containers, you need to install Singularity on a Linux computer with administrator privileges. For those who do not use Linux on their personal computer, it is worth mentioning that any computer with Windows 10 Enterprise / Pro already has a built-in option to run virtual instances of Linux. The example below was implemented on an Ubuntu 20.04 LTS virtual machine. Following this Singularity installation guide, a number of dependencies were first installed:

sudo apt-get update && sudo apt-get install -y \
    build-essential \
    libssl-dev \
    uuid-dev \
    libgpgme11-dev \
    squashfs-tools \
    libseccomp-dev \
    wget \
    pkg-config \
    git \
    cryptsetup

Lai būtu iespējas kompilēt un iegūt lietošanā Singularity, nepieciešamas programmēšanas valodas Go rīki, kurus var lejupielādēt golang.org mājaslapā. Instalācija veicama sekojoši:

# Atarhivē lejupielādēto arhīvu
tar -C /usr/local -xzf go1.16.linux-amd64.tar.gz

# Pievieno ‘go’ mapi datora PATH
export PATH=$PATH:/usr/local/go/bin

# Restartē datoru vai arī izpilda komandu:
$HOME/.profile

To be able to compile and use Singularity, you need Go programming tools, which can be downloaded from golang.org. The installation is performed as follows:

# Extracting the downloaded archive
tar -C /usr/local -xzf go1.16.linux-amd64.tar.gz

# Adding the ‘go’ folder to system PATH
export PATH=$PATH:/usr/local/go/bin

# Reboot computer or execute command:
$HOME/.profile

You can continue with the installation of the Singularity program itself. The code of the latest version of the program can be downloaded from the GitLab site.

# Extracting archive and changing directory
tar -xzf singularity-3.7.1.tar.gz
cd singularity

# Install singularity by specifying location (here /usr/local is chosen)
./mconfig -b ./buildtree -p /usr/local
cd ./singularity/buildtree
make
sudo make install

# Check if installation was successful
singularity version

After completing the steps above, Singularity has been successfully installed and it is possible to use Singularity on your computer and prepare containers for use elsewhere, e.g., HPC clusters.

Create container using a recipe file

You can create your own containers using the container recipe file. Its contents indicate what software, what parameters and other functional aspects should be included within the container file that is about to be built. You can read about the blocks and parameters in the contents of these recipe files in this manual. However, it is extremely valuable to get acquainted with and base your work on the various examples of recipe files compiled in this GitHub repository.

Below is an example of a recipe file that creates a container with the aforementioned ESPResSo software. As you can see from the contents of this file, the operation is similar as with a typical computer: for the most part, the file consists of the %post section, with contents identical to the ESPResSo installation guide.

Bootstrap: docker
From: ubuntu:18.04


%help
    Container with ESPResSo
    ESPResSo means: Extensible Simulation Package for Research on Soft Matter
    Website: http://espressomd.org/wordpress/
    Main binary for using ESOResSo via this container is 'pypresso'.

%post
    # Updating repositories and packages
    apt-get update && apt-get -y upgrade
    
    # Installing all dependencies (as shown on ESPResSo website)
    apt-get -y install wget tar build-essential \
        cmake cython3 python3-numpy libboost-all-dev \
        openmpi-common fftw3-dev libhdf5-dev \
        libhdf5-openmpi-dev python3-opengl libgsl-dev
   
    # Installing ESPResSo (as shown on ESPResSo website)
    cd ~
    wget -c \
    https://github.com/espressomd/espresso/releases/download/4.1.4/espresso-4.1.4.tar.gz
    tar zxvf espresso-4.1.4.tar.gz
    mkdir /usr/bin/espresso && cd /usr/bin/espresso
    cmake ~/espresso
    cmake --build .
    
    # Cleaning up
    apt-get clean
    apt-get autoremove
   
%environment
    # Something about locale... everyone seems to be doing this.
    export LC_ALL=C
    
    # Adding ESPResSo directory to PATH, so that the command 'pypresso' is available
    export PATH=/usr/bin/espresso:$PATH

Once the recipe file has been created and saved as ESPResSo_container_recipe.def, you can build a container using the command below.

sudo singularity build ESPResSo_container.sif ESPResSo_container_recipe.def

In this case, the process took 5-10 minutes (downloading the basic environment and packages, compiling the specific software). The size of the container file created at the end is about 400 MB. Once a container file has been created, it can be used in the same way as previously shown, with exec and shell commands. The container file can be copied and used elsewhere, including an HPC cluster.

singularity exec ESPResSo_container.sif pypresso <pypresso script file>

Using containers with GPU-compute capabilities

Although containers are encapsulated software environments, their execution can be linked to base operating system resources and hardware: enabling the use of system GPU cards and supplying external software resources (such as CUDA libraries for use inside a container). Below is a brief example of using a Tensorflow container with GPU acceleration. Tensorflow is a popular tool used to perform machine learning tasks.

# Load 'singularity' module
module load singularity/3.4.1

# Load CUDA library
# This module containes information where to look for CUDA commands
# within the system (‘/bin’ folder), and defines the CUDA_HOME variable
module load cuda/cuda-10.2

# Using the container from DockerHub while enabling use of GPU (‘--nv’),
# as well as bind the CUDA library folder with ‘--bind ..’.
singularity shell --nv --bind ${CUDA_HOME} \ 
                     ~/my_containers/tensorflow_2.3.1-gpu.sif

Similarly, any other external software can be passed to the container by using the bind parameter. Thus, containers can be made smaller: for example, it is not necessary to include Anaconda Python in a container if it is clearly known that it can be passed through with the bind parameter. In this case, only the specific/non-standard software should be included in the container.

Topic discussed in this last section is thoroughly analyzed in the presentation that was shown during the seminar on March 5. It is made available to you as PDF file:

The post Capabilities of Singularity containers appeared first on Institute of Numerical Modelling.

Mg. Phys. Kristaps Bergfelds is a researcher at the Institute of Numerical Modelling (University of Latvia), where his daily duties include acquisition, processing, and plotting of scientific data. Additional to his research, Kristaps teaches programming for physics students (programming and data science basics using Python language). This article is based on Kristaps’ experience and observation on the daily needs of his colleagues and students to collaborate in the development of various programming projects.

Almost anyone who has collaborated with colleagues on creating a text report has encountered version control issues. Often you end up browsing and organizing such documents as document.docx, document_final.docx, document_final_edited.docx, document_final_edited_2_really_completed.docx etc. This process can be confusing, error-prone, and cumbersome for all those involved.

To make this process easier, you can place the document in a place accessible to everyone and edit it collaboratively: either in the form of a cloud document (Google Drive, OneDrive, Dropbox) or in a shared local network folder. Versioning is available on these document sharing platforms so that you can restore an earlier version of your documents if necessary. Edit highlighting is also available to inspect exactly what has changed between versions.

Programmers have been facing very similar problems since the beginning of software development itself – several developers need to interact during program code development, keep track of changes, test various elements of program code, and sometimes obtain earlier code versions. As with text documents, several specialized tools have been developed for controlling code versions, offering a variety of options for specific code development needs, such as the ability to create parallel code development paths and branched version structures.

Currently, the de facto code versioning system standard among programmers is the Git platform. In addition to various technical advantages and increasing popularity, supporting web-platforms (such as GitHub, GitLab, BitBucket) should be emphasized. Therefore, this article is devoted to the use of the Git platform specifically. This article is intended for Windows users who want to take advantage of the versioning system in a graphical interface (rather than through the command line).

The article provides an insight into the basic principles of using Git, shows the required software to be installed in the Windows environment, provides an insight into the sharing of Git repositories with the help of the GitLab portal. For further reading, it is highly recommended to use this free tutorial provided by the Nordic e-Infrastructure Collaboration (NeIC).

Setting up Git on your computer and basic use of repositories

To start using the Git versioning system, you need to install Git for Windows (keep all default settings during installation). Installing this program makes it possible to use Git only in the command-line interface. Further sections of this article will show you how to do this using a graphical interface, however, a brief demonstration of the command-line interface use is given here to illustrate the basics of Git and introduce some terminology.

When you open the installed Git command line (Git Bash), execute the following commands to create a folder whose contents will be subject to versioning control (i.e., a repository will be created). The comments written on each line reveal the essence of the commands.

$ mkdir MyProject    # Create folder
$ cd MyProject       # Change location to the new folder
$ Git init           # Initialize repository

You can place/create program code files in the folder using your usual tools. In the example below, the folder contains a Python script (my_program.py) that plots some mathematical expressions (myplot.pdf).

To preserve the current stage of .py script development in the versioning history, you need to execute commands that (1) add the file to the version control (staging) and (2) mark the changes made in the version history (commit).

git add my_program.py                    # Staging for version control
git commit -m "Initial code development" # Saving changes as a version

Once this is done, you can run the git status command to see what the current state of the repository is from the versioning system’s point of view. The command will show if changes have been made since the last version of the content was committed and if there are files that are not being tracked. In this example, the file myplot.pdf is not tracked, because it is the output of the developed script, the exact content of which is not important. It is important, however, to follow the code of the program that created it.

After changing the repository files (e.g., by continuing to develop the program code), the changes are saved by re-executing the already mentioned commands (git add, git commit). When you run the git log command, the version development history is displayed. This example shows that two submissions have been made (‘Initial code development’ and ‘Small edits’).

If you need to undo changes made to the repository, you can run the git restore command. There are many capabilities of Git: you can return the repository to its previous states, you can create parallel branches of development, you can temporarily examine older versions. However, using Git via the command line interface can be inconvenient, frustrating, and unfamiliar to regular Windows users. Therefore, further in this article we look at ways to use Git repositories through a graphical interface.

Using Git Extensions for graphical experience

Installation and configuration

To use the graphical interface to manage Git repositories in Windows, it is recommended to install Git Extensions. To take full advantage of this program, you need to install two additional applications: (1) WinMerge, which will be used to visualize file changes, and (2) PuTTY, which will make it easier to work with remote repositories (e.g., GitLab).

After installing Git Extensions, it is very important to configure all the required program settings – for this purpose, the program settings window offers a convenient checklist. Make sure all settings are shown as valid (green).

In the ‘difftools’ section select the WinMerge tool mentioned above. Additionally, select ‘PuTTY’ in the ‘SSH’ section of the settings.

Basic usage

After installing the programs and configuring all the settings, use the Git repositories without the help of the command line are now possible. In any Windows folder, a right-click reveals the options provided by Git Extensions: browse existing or create new repositories, copy (clone) the repository from another location (e.g., GitLab).

In folders where repositories are already located, ‘GitExt open repository’ command is available. This option opens a window where you can review the history of repository changes, see their authors and added comments.

You can click the ‘Commit’ button to create a version of the repository content similarly as shown previously in the command line example. However, in this case, an additional visualization is provided by the Git Extensions environment.

A very valuable and frequently used option is to edit the .gitignore file (Repository / Edit .gitignore), which lists the files and filetypes whose changes to ignore and not subject to version control. In the example provided, PDF file output should be added here.

Using GitLab for collaboration

So far, this article considered only local Git repositories. However, a significant advantage of using a versioning system is the ability to collaborate easily and transparently. That is why several Git-based Internet services have been created with the possibility to deploy and share remote repositories. This section demonstrates the use of one of such services: GitLab.com. It should be noted that the University of Latvia provides a GitLab-based platform git.lu.lv, which can be accessed with a LUIS username and password.

Registering and creating a repository

You can register on GitLab with an email or a Google account. The platform is free to use, but you need to notice and use the ‘Skip Trial’ button during registration.

After successful account creation, it is possible to create remote repositories – specify the title, description, privacy level (private, shared to group, public) and other parameters.

Adding an SSH key

To upload (git push) the local repositories to the remote GitLab environment, as well as download (git clone) your private repositories, you need a way to authenticate your local computer to work with your GitLab account. SSH keys are used for this purpose.

To enable this type of authentication, you need to create a pair of private and public SSH key. A private SSH key is nothing more than a very long and secure password. The SSH public key, on the other hand, is a publicly visible authentication token that matches this password. In fact, it would be more appropriate to call the public key a ‘lock’ which the user can install in a place of his choice – in this case, his GitLab account.

If you have the aforementioned PuTTY program installed on your computer, the PuTTYgen tool is available for you to easily create SSH key pairs. Following the instructions in the program window, it is possible to obtain both the private key (saved in a .ppk file) and the public key (it must be copy/pasted to the GitLab portal section User Settings / SSH Keys).

Cloning remote repositories

To obtain a repository located on the GitLab portal (including newly created/empty repositories), you need to copy its SSH address.

On your computer, right-click within a folder and select the GitExt Clone command. You must specify the address obtained on the GitLab portal and select the appropriate private SSH key for your account. Press the ‘Clone’ button to copy the repository.

After the successful copy (clone) of the repository, you will be able to continue/start working with its content as demonstrated previously. However, now, in addition to the ‘Commit’ action discussed earlier, there is a ‘Push’ action that uploads versions created in the local repository to the remote location. Similarly, you can perform a ‘Pull’ action that will download changes made by others.

Cloning public repositories

If the repository is set to be public, anyone can download it (git clone) using the repository’s HTTPS address provided on the GitLab platform. In the example below, the public repository shown in this article is downloaded to the INM High Performance Computing Cluster using the command line.

Conclusion

This article provides an insight into working with local Git repositories in the Windows environment by using the Git Extensions GUI. Basics of using GitLab environment, creating remote repositories, performing clone/push/pull operations is also shown. However, the article does not discuss several advantages of the Git system, as well as the GitLab portal. At least one of them should be emphasized before closing this article: project documentation.

In the GitLab repository interface, you can display the repository documentation using the README.md file. I.e., if the repository contains a text file named README.md, it will be previewed on the GitLab site. These documentation files can be formatted with various text markdown commands.

* Nulla vel sem volutpat, volutpat sapien sed, egestas ligula.
* _Vestibulum fermentum lectus et finibus volutpat._
* __Vestibulum tristique turpis quis enim porta vestibulum.__
* `Nam facilisis sapien vitae gravida ultricies.`
* [Nam commodo purus in faucibus fermentum.](https://modinst.lv)

The text formatting commands shown above, if placed in the README.md file, will be displayed on the GitLab portal as follows:

Hopefully, this article will help you to get started with Git and will allow you to collaborate more efficiently and quickly when working on code projects.

The post Basics of Git version control appeared first on Institute of Numerical Modelling.

Mg. Phys. Kristaps Bergfelds is a researcher at the Institute of Numerical Modelling (University of Latvia), where his daily duties include acquisition, processing, and plotting of scientific data. Additional to his research, Kristaps teaches programming for physics students (programming and data science basics using Python language). This article is based on Kristaps’ experience and observation on the daily computer tasks of his colleagues and students.

Introduction

Modern science is unimaginable without the use of a computer! However, much more than web-browsing, e-mail, and text editing is meant by this claim. Today’s IT environment and its rapid development require technology professionals and scientists to be familiar with data processing techniques, data visualization, and many other areas. The computer is the permanent ally of the scientist – the Swiss Army knife, which must always be kept sharp in your pocket!

But unlike a knife, a computer has no tangible blades or tools. Instead, these various functions are performed by applications installed on your computer. Knowing exactly which programs will be useful in a scientist’s daily life is often a long and painful search process. This article tries to help with this problem.

The article describes a list of free software that will be useful for anyone facing data processing problems and other typical tasks of a scientist. This article is intended for a beginner who uses the Windows operating system. However, it should be noted that some of the software is also available in the Linux environment.

This article is written as supplementary material for the Association of Latvian Young Scientists online event “Scientist’s Survival Kit: From Data to a Proper Graphic” on January 6, 2021. Event recording available on YouTube (in Latvian).

Management of files and raw data

Before you start the data analysis, you need to do several preparations, as well as answer a series of questions: What is the filetype of the data? What is the structure of these files? How many of these data files are there? How can you arrange and prepare them to proceed with your analysis effectively? Clarifying these issues often takes up most of the workflow and often is a source of many mistakes. The software listed in this section facilitates this stage of work.

Notepad++

You can use Notepad++ to view the contents of text files. This is especially important as it is a typical data storage format (.csv, .dat, etc. – these are all plain text data formats). You can also view these files with Windows tool Notepad, but as the name Notepad++ implies, this program offers several benefits.

The program provides tabbed browsing, advanced find/replace operations (see figure), non-standard input methods (e.g., to write in several lines at once), the ability to turn on invisible characters, compare the contents of two files, syntax highlighting for numerous programming languages, etc. The possibilities and use-cases of this lightweight program are extremely wide.

Total Commander

When you need to work with many files and various filetypes, Windows built-in file manager Windows Manager may not have enough capabilities, but Total Commander certainly does.

This program offers various specialized operations for file management: Bulk file renaming (with text find/replace and numbering), comparison and synchronization of file folders (see picture), specialized file search options, etc.

7-Zip

When finishing work on a project/data processing task, it is recommended to store the whole set of files (data, plotting scripts, generated images) compressed in a single ZIP archive file. This operation is important and relevant even though the capacity of modern data storage devices (flash drives, hard drives) has increased considerably in recent years.

Archive file creation is recommended not because of the storage size reduction, but instead for more convenient file management. Combining many small files into a single ZIP file makes several operations incomparably faster: such as copying files and synchronizing folders (shown above with Total Commander). Additionally, when you use archived data, you avoid the risk of the operating system performing unwanted actions (for example, Windows Photos application interpreting plotted data images like photographs and including them in albums).

The recommended software for working with various data archive formats is 7-Zip. It can create and open various types of archive files (.zip, .rar, .tar, .7z, etc.). It should be emphasized that, despite the support for various formats, it is recommended to create archives as .zip files, as it is an ISO standardized file format.

SpaceSniffer

You can visualize the use of your storage device with SpaceSniffer. The program displays folders and files in the form of rectangles, the area of which is proportional to the amount of data in it. As you can see in the figure below, the contents of my C: drive are largely used by some very large programs (ANSYS, Wolfram Mathematica, Anaconda Python) rather than Windows files or my personal data.

Working with images and video

A large and important part of scientific work often is the creation and management of images. Very often photographs and other types of images are the data sources for scientific data analysis. Or vice versa – they are the main result of data processing operations. There are several tools recommended for working with this type of data that can facilitate this process.

WebPlotDigitizer

Sometimes it is necessary to get numerical data from an image. For example, data from old scientific publication must be used in your calculations or be redrawn. Instead of placing a ruler on the monitor, it is recommended to use WebPlotDigitizer for this task.

With the help of this program, you can read numerical data from any image – define your own reference system and, using the mouse cursor, read the data and save it in a data file (see the picture above). If the data is displayed with a contrasting colour, the reading can be done automatically without selecting the data points with mouse clicks.

GIMP

Image processing program GIMP is extremely popular – its capabilities are not that far behind commercial alternatives such as Adobe PhotoShop. It is possible to work with raster (pixel) images in layers, create effect layers and layer masks. Various filters and tools are available. The figure below shows the use of a relatively rare and unique tool: layer perspective correction to show what the logo would look like on a hallway wall as a signboard.

However, it should be noted that learning the program takes time and patience. However, if you do not need all the extensive features of GIMP, other simpler programs exist.

Paint.NET

In case you do not need all the features of GIMP, Paint.NET is highly recommended. This relatively simple and intuitive program allows you to work with image layers, various effects and tools. For example, the image below shows how to use the Magic Wand tool to selectively highlight a white background for a logotype – for deletion or recolouring.

ImageMagick

In the case of many images, even simple operations such as resizing or converting the image filetype are not easy with the programs listed above. In such cases, ImageMagick can help. It is a command line program that allows efficient batch image processing. For example, the command below (executed in the Command Prompt console or written in a Windows batch script) will process all the JPG files in the folder – crops the biggest available square part of the image and resizes it to 1000×1000 px.

magick mogrify -gravity center -crop 1:1 +repage -resize 1000x1000 *.jpg

This is a typical operation when preparing images for further use on web pages or data processing programs.

Inkscape

Unlike the raster (pixel) image processing programs described above, Inkscape is designed for creating and processing vector graphics. It can be used to create various diagrams and drawings. Inkscape in many cases allows to avoid use of specialized software (CAD software, Microsoft Visio, etc.). If creation of schematics and other vector graphics are not routine operations for you, Inkscape is a particularly attractive choice.

The image above shows both the capabilities of Inkscape and the benefits of vector graphics in general: A pixelated small image is transformed into a vector graphics object using the appropriate Inkscape tool Bitmap Trace. It can be resized while maintaining image quality. It is possible to change its colours and modify it many other ways.

Avidemux and OpenShot

Sometimes you must deal with situations where you need to work with video files – image frames need to be merged into a video (time-lapse), video file formats need to be converted, or a short video scene requires editing. However, buying commercial software (Adobe Premier, Final Cut Pro, etc.) might not be an option. In this case, free tools such as Avidemux and OpenShot will come in handy.

Avidemux is a very capable program for converting various video formats and changing the technical parameters of video files. It is also possible to combine individual images (.jpg, .png) into a video file (see example in the image above). OpenShot, on the other hand, is a very powerful video editing tool that goes beyond the Video Editor included in Windows 10. At the same time, the program is intuitive and simple enough to learn quickly.

Data visualization and creating reports

Once you have found out the structure of your data files and arranged them for easy processing (for example, they are named with a numbered pattern), you can begin processing and plotting them and summarize the results in a report. This section shows some tools and approaches for this process.

Gnuplot

If you only need to display (not process) data files, Gnuplot is very useful. This program can be used as a command-line application or used via plotting scripts. Because the program is intended for data plotting – not processing – the commands used are very concise and minimalistic. It is relatively easy to learn this program, while still allowing you to process very large data sets and obtain various types of output (line plots, bar plots, histograms, 2D heatmaps, etc.). You can output graphics in both vector (PDF) and raster (PNG) image formats.

Below is a small script that plots the contents of three data files in a single image: speed versus time during car breaking and stopping. The comments within the script describe the meaning of each section of the code. Here you can download the whole example (input data, plotting script, output image).

# Defining canvas size, line thickness, and font size
set terminal pngcairo size 900,700 enhanced lw 3 fontscale 2.0

# Setting filename of the output
set output "example_gnuplot_output.png"

# Formatting plot (axis titles, range, tickmarks)
set yrange [-5:105]
set xlabel "t, s"
set ylabel "v, km/h" # Data in m/s! Need to scale by 3.6!
set xtics 1
set ytics 20
set grid

# Cycle for plotting all three datafiles on single canvas
plot for [i=1:3] \
	'breaking_speed_data_'.i.'.dat' using ($1):($2*3.6) with line title 'File '.i

The plot (see below) shows that the car has stopped from 100, 70 and 50 km/h within 4.0, 2.7 and 2.0 seconds respectively.

Python

If it is necessary not only to display the data, but also to process it at the same time, then the Python programming language and various additional components (packages) are very suitable for this purpose. A convenient way to get the full set of tools (Python, frequently used packages, graphical code editor) is to install the Anaconda software package.

The capabilities of the Python programming language and related tools certainly go beyond the scope of this article, but below is an example of what a data processing and plotting script looks like. It uses Numpy/Scipy numerical data processing capabilities as well as Matplotlib plotting library. The comments within the script describe the meaning of each section of the code. Here you can download the whole example (input data, plotting script, output image).

# Importing necesarry packages
import numpy as np
import matplotlib.pyplot as plt
import glob
from scipy.integrate import cumtrapz

# Reading all filenames ending with '.dat' from the current folder
text_filenames = glob.glob("*.dat")

# Defining canvas size for plotting
plt.figure(figsize = (4.5,3.5), dpi = 200)

# Cycle through all filenames for data processing and plotting
for filename in text_filenames:

    # Reading each file (tab-delimited data)
    time, velocity = np.loadtxt(
        filename,
        delimiter = "\t",
        skiprows = 1,
        unpack = True
        )
    
    # Using numerical integration to get distance from velocity data
    distance = cumtrapz(velocity, time, initial=0.0)
    
    # Determining maximum velocity and distance (for later use in legend)
    velocity_max = velocity.max()
    distance_max = distance.max()
    
    # Plotting distance vs. time and putting the max distance in the legend
    plt.plot(
        time,
        distance,
        label = "{:.0f} km/h - breaking dist. is {:.1f} m".format(
            velocity_max*3.6,
            distance_max
            )
        )

# Formatting of the plot (added grid, axis titles, legend)
plt.grid()
plt.legend(loc="upper left")
plt.xlabel("t, s")
plt.ylabel("d, m")
plt.xlim(0,5)

# Saving the plot as PNG image
plt.savefig("example_python_output.png")

The script sums up (integrates) the previously displayed car velocity data to show the braking distance of the car. It can be seen that the car’s braking distance from 100 km/h is 55 m, but from 50 km/h only 14 m. This result is quite un-intuitive, as twice the speed does not mean twice the braking distance. In fact, it is four times greater!

LaTeX typesetting

Usually, the main result of a scientist’s work is not constructed equipment, programmed software or other production-grade product. Instead, it is new information and conclusions. The way you present the results of your research – this new information – is extremely important. It must thorough, technically sound, and easy to comprehend. Most of these informative materials (reports, publications, scientific articles) rely heavily on the use of figures, tables, mathematical expressions, bibliographic citations.

It is possible to create this type of material in Microsft Word, but it is certainly cumbersome. LaTeX typesetting solutions are much more suitable for this purpose. When using LaTeX, documents are created in the form of a code, specifying only the content – where the sections start and end, where to insert references, where to place images. The formatting is left to the LaTeX engine, which arranges text and images optimally, performs automatic reference numbering and so on. This approach is so suitable for creating scientific literature that scientific journals often accept articles only in LaTeX code. Below is a visual example of a LaTeX typesetting result that would be extremely difficult to create with Word (example from this StackExchange thread).

To start creating documents with LaTeX, you can install the necessary software on your computer: MiKTeX package (allows you to convert LaTeX code into a PDF document), as well as TeXstudio or Texmaker editor (GUI for easy code creation, file management). But lately, creating LaTeX documents on the Overleaf cloud platform has become extremely popular. There is no need to install anything on your computer – documents can be created directly in a web browser.

LaTeX document creation and language syntax are also too broad a topic for the scope of this article, but for an impression of how document creation with LaTeX/Overleaf works, see this example where comments show the meaning of each code section.

Conclusion

The list of recommended software as well as their capabilities is by no means fully described in this article. There are other open-source programs that perform the described functions. Additionally, the software shown here can be used for much more that is shown. Nevertheless, I hope that this article will serve as the first step for those looking for ways to expand their capabilities.

Appendix

Following the publication of this article, several readers have identified other valuable open access programs that are relevant to the topic of this article.

The post From data to beautiful graphs: The tools of a modern scientist appeared first on Institute of Numerical Modelling.