Benchmarks



Introduction

The estimation of the performance of an entire programming language is not an easy task.
Since the scope of our work is to develop a new programming language, EveC, in order to obtain a more efficient language than Java, the results report the comparison of performances for EveC and Java, both in terms of memory usage and computational speed. For this reason, we present a large number of benchmarks, that can be divided into two groups:

  • Micro-benchmarks, that try to evaluate the performance of a particular construct of the language, such the addition of two integer numbers, or the cost of a method call.
  • Macro-benchmarks, that are bigger application that should represent parts of real world applications.

Micro-benchmarks are important in order to evaluate single constructs of the programming language, and can also be useful to find bottlenecks that can be target of further optimizations that are not implemented yet.
Unfortunately, at this stage of development, a huge real world application cannot be ported or written using the EveC programming language because not all the functionalities of a complete library are implemented yet. For example, is not possible to read or write a file.
Nevertheless, the results reported in the following sections will show that the EveC programming language achives performances that are at least comparable to Java as computational speed, and that are always better than Java as memory usage.


Java Grande Benchmarks

Java Grande Benchmarks is a suite of benchmarks relevant for testing the performance of Java for scientific computations. It can be divided into three different sections: low-level benchmarks that test general language features and operations (e.g. method invocation); scientific and numerical application kernel; full-scale science and engineering application.
Currently, only benchmarks belonging to the first two categories are used.

The low-level benchmarks consist in:

  • Arith: it executes basic arithmetic operations.
  • Assign: it does variable assignments.
  • Cast: it does cast between primitive types.
  • Create: it generates different kinds of objects
  • Math: it executes all the mathematical functions present in the class java.lang.Math.
  • Method : it performs method calls of different types.



The scientific and numerical computational benchmarks test well-known computational benchmarks included in an huge number of real world application.
For this reason and also because they are bigger and more complex than the low-level benchmarks, scientific and numerical computational benchmarks are included into the macro-benchmarks section. They are composed by:

  • Fourier: it computes fourier coefficients for a given function.
  • Heap Sort: it sorts an array of integers using the heap sort algorithm, and tests generic integer performances.
  • Crypt: it performs IDEA (International Data Encryption Algorithm) encryption and decryption on an array of N bytes.
  • FFT: it performs a one-dimensional forward transformation of N complex numbers. This kernel exercises complex arithmetic, shuffling, non-constant memory references and trigonometric functions.
  • SOR: it performs 100 iterations of successive over-relaxation on a NxN grid. The performance reported is in iterations per second.
  • Sparse Matrix Multiplication: it uses an unstructured sparse matrix stored in compressed-row format with a prescribed sparsity structure. This kernel exercises indirection addressing and non-regular memory references. A NxN sparse matrix is used for 200 iterations.

jBYTE

The jBYTE benchmark is a collection of well-known algorithms proposed by the BYTE Magazine. This benchmark is used in different works.
The ported benchmarks are:

  • Numeric Sort: it sorts an array of 32-bit integer using the heap sort algorithm.
  • Bitfield: it executes a variety of bit manipulation functions.
  • Emulated floating-point: it performs floating-point operation via software.
  • Fourier Coefficients: it executes a numerical analysis routine for calculating series approximations of waveforms.
  • Huffman compression: a well-known text and graphics compression algorithm.
  • IDEA encryption: a block cipher algorithm.
  • Neural Net: a small but functional back-propagation network simulator.
  • LU Decomposition: a robust algorithm for solving linear equations.

Simplex

The Simplex benchmark is a simple implementation of the well-known optimization algorithm, which finds an optimal solution in a minimization problem (or maximization) of a linear function with linear constraints. This benchmark is considered as a macro-benchmark. The algorithm uses an implicit enumeration technique to explore all the possible solutions terminating when the optimal solution is found. The implementation of the algorithm is taken here , and a simplified version is used.
The implementation relies on the standard random generator defined for Java to initialize the data processed by the algorithm.
The Simplex benchmark is executed with a different number of variables and constraints. The reported results are the arithmetical means on five run, in order to achieve better results stability (i.e. results with higher confidence due to multiple runs).

Sorting Algorithms

Three well-known sorting algorithms are also used.
Bubble Sort:
Bubble sort is a simple sorting algorithm that works by repeatedly stepping through the list to be sorted, comparing each pair of adjacent items and swapping them if they are in the wrong order. The pass through the list is repeated until no swaps are needed, which indicates that the list is sorted. The algorithm gets its name from the way smaller elements “bubble” to the top of the list. It performs the sorting of an array with a quadratic performance with respect to the size of the array.

Insertion Sort:
Insertion sort is a simple sorting algorithm that builds the final sorted array (or list) one item at a time. Insertion sort iterates, consuming one input element each repetition, and growing a sorted output list. On a repetition, insertion sort removes one element from the input data, finds the location it belongs within the sorted list, and inserts it there. It repeats until no input elements remain. Sorting is typically done in-place, by iterating up the array, growing the sorted list behind it. At each array-position, it checks the value there against the largest value in the sorted list (which happens to be next to it, in the previous array-position checked). If larger, it leaves the element in place and moves to the next. If smaller, it finds the correct position within the sorted list, shifts all the larger values up to make a space, and inserts into that correct position.

Quick Sort:
Quick Sort is a “divide and conquer” sorting algorithm. In fact it recursively divides the array into two parts by selecting a pivot element. Then it searches in each array an element that is in wrong position with respect to the pivot (hence for the first part of the array, it searches an element that is greater then the pivot if considering ascending sorting; vice versa for the second part of the array). Once found the two elements it swaps them and repeats the procedure until all the element are in the right position with respect to the pivot element. Then the algorithm is recursively applied to the two parts of the array.