The blog post is divided into 4 steps:

1. Simple Querying Without Batch Operations
2. Implementing Batch Queries
3. Using Generator Functions for Batch Queries
4. Clean Code: Extract the batch logic into its own function

1. Simple Querying Without Batch Operations

Suppose we want to send a notification to all our users, we would need to retrieve all users from the database using TypeORM, iterate over each individual user and send the actual notification. The code is clean and split up into the more general function sendOfferToUsers(), whereas the database query logic is encapsulated into findAllConfirmedUsers()

async function sendOfferToUsers() {
  for (const user of await findAllConfirmedUsers()) {
    const message = `Special Offer just for ${user.name}`;
    await notifyUser(user.email, message);
  }
}

async function findAllConfirmedUsers() {
  return await userRepository.find({
    where: { confirmed: true }
  });
}

async function notifyUser(email: string, message: string) {
  // omitted for the brevity
}

While this approach works fine for smaller datasets, it becomes inefficient and memory-intensive when dealing with larger ones. To address this issue, we need to query the database in batches.

2. Implementing Batch Queries

To efficiently query large datasets from a database, it’s necessary to retrieve the data in subsets rather than all at once. This is achieved through batch operations, which involve querying the database multiple times and limiting the returned rows to a specific number while skipping the previously fetched rows.

TypeORM allows us to set take and skip attributes for batch querying, but it requires explicit configuration. Here’s an example of implementing batch operations:

async function sendOfferToUsers() {
  const batchSize = 1000;
  let batchIndex = 0;
  let users: User[];

  do {
    users = await findAllConfirmedUsers(batchSize, batchIndex);
    for (const user of users) {
      const message = `Special Offer just for ${user.name}`;
      await notifyUser(user.email, message);
    }
    batchIndex++;
  } while (users.length === batchSize);

}

async function findAllConfirmedUsers(batchSize: number, batchIndex: number) {
  return await userRepository.find({
    where: { confirmed: true },
    take: batchSize,
    skip: batchIndex * batchSize
  });
}

Obviously, the code is less readable than the previous example due to littering the batch operation from the database-specific method findAllConfirmedUsers() into sendOfferToUsers(). However, I want to enforce a stricter separation of concerns and keep all database-related code in findAllConfirmedUsers().

To address this requirement, we can employ a generator function.

3. Using Generator Functions for Batch Queries

Let’s begin by examining the refactored code, where we consolidate all database-related logic into findAllConfirmedUsers() and transform it into a generator function.

async function sendOfferToUsers() {
  for await (const user of findAllConfirmedUsers()) {
    const message = `Special Offer just for ${user.name}`;
    await notifyUser(user.email, message);
  }
}

async function* findAllConfirmedUsers() {
  const batchSize = 1000;
  let batchIndex = 0;
  let users: User[];

  do {
    users = await userRepository.find({
      where: { confirmed: true },
      take: batchSize,
      skip: batchIndex * batchSize
    });
    for (const user of users) {
      yield user;
    }
    batchIndex++;
  } while (users.length === batchSize);
}

The signature of findAllConfirmedUsers() has changed from function to function* (a star at the end) and includes the yield keyword in the function body. Beside from these syntactical changes, the function itself works much different. It’s a special type of function that can be paused and resumed during execution, allowing for the generation of a sequence of values. findAllConfirmedUsers() retrieves users from the database in batches and yields one user at a time. The for await...of loop in the sendOfferToUsers() function then iterates over the generated values, providing a clean and readable way to process each user without loading the entire dataset into memory. The code becomes more efficient and memory-friendly when dealing with large datasets.

Using a generator function makes the code more readable and maintainable. The sendOfferToUsers() function remains focused on its main purpose, while the details of batch querying are handled by the findAllConfirmedUsers() generator function.

4. Clean Code: Extract the batch logic into its own function

To enhance the code further, we can extract the batch logic into its own function to make it reusable. This function can be used to retrieve arbitrary entities in batches, eliminating the need to duplicate the batch logic when fetching different types of entities.

async function sendOfferToUsers() {
  for await (const user of findAllConfirmedUsers()) {
    const message = `Special Offer just for ${user.name}`;
    await notifyUser(user.email, message);
  }
}

async function* findAllConfirmedUsers() {
  yield* findInBatches(userRepository, { where: { confirmed: true } });
}

async function* findInBatches<T extends ObjectLiteral>(repository: Repository<T>, findOptions: FindManyOptions<T>) {
  const batchSize = 1000;
  let batchIndex = 0;
  let entities: T[];

  do {
    entities = await repository.find({
      take: batchSize,
      skip: batchIndex * batchSize,
      ...findOptions
    });
    for (const entity of entities) {
      yield entity;
    }
    batchIndex++;
  } while (entities.length === batchSize);
}

The yield* keyword is used to pass on the iteration control from one generator function to another. In the findAllConfirmedUsers() function, yield* is employed to transfer the iteration of the batched users to the findInBatches() generator function. This helps to maintain a clear separation of concerns: the findAllConfirmedUsers() function concentrates on fetching users from the database, while the findInBatches() function handles the logic for querying in batches.

Conclusion

In Node.js applications, querying a database in batches is essential for handling large datasets efficiently. By utilizing generator functions, we can simplify the code, improve readability, and maintainability. The extracted batch logic also promotes clean code organization and reusability. Generator functions are a powerful language feature that I just learned about a few weeks ago.

Comments are welcome on Twitter or LinkedIn.

OpenTelemetry is the new shining star on the Observability horizon. It aims to standardize the three pillars of observability: Tracing, Logging, and Metrics. OpenTelemetry is a set of vendor-independent APIs and SDKs for collecting and exporting telemetry data. To obtain distributed traces of an application, only the general OpenTelemetry interface needs to be integrated without a vendor-specific implementation or library. The three major cloud providers, AWS, Azure and Google, already provide an interface from OpenTelemetry to their hosted tracing solutions. I expect OpenTelemetry to be the only widely used observability API in the future. It already provides good support for tracing and metrics, but they are still working on the specification for logging.

AWS X-Ray has two key advantages over a self-hosted solution like Jaeger: It is a fully AWS-hosted service that includes setup, authentication, and updates. In addition, many other AWS services such as API Gateway or DynamoDB report their traces to AWS X-Ray. This provides visibility into the entire microservices infrastructure, not just the self-developed services.

In this article I want to show how to integrate OpenTelemetry into a Java Quarkus microservice in order to send distributed traces to AWS X-Ray. No javaagent is required and it supports the Quarkus native mode. This solution enables observation of the entire microservice infrastructure, including services hosted by AWS.

The setup consists of four steps:

1. Integrate OpenTelemetry into the Quarkus application
2. Adjust the Trace-ID generation
3. Use the AWS Context Propagation
4. Run the OpenTelemetry Collector

1. Integrate OpenTelemetry into the Quarkus application

There are two different ways to instrument an application to get tracing data with OpenTelemetry: autoinstrumentation using a javaagent or manual instrumentation by modifying the code. But Quarkus offers a third and much better option: An extension for OpenTelemetry that automatically collects tracing data.

This is actually very simple and is described in detail in the Quarkus OpenTelemetry Guide. I just want to point out the two changes that actually need to be made:: add the quarkus-opentelemetry-exporter-otlp extension to the pom.xml and configure it in application.properties.

<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-opentelemetry-exporter-otlp</artifactId>
</dependency>

quarkus.application.name=myservice
quarkus.opentelemetry.enabled=true
quarkus.opentelemetry.tracer.exporter.otlp.endpoint=http://localhost:4317

As a result the application can push traces to a OpenTelemetry Collector running on localhost:4317. To forward the data to a local Jaeger instance, please follow the Quarkus OpenTelemetry Guide.

2. Adjust the Trace-ID generation

Every request that hits the microservice application gets a unique Trace-ID. By the specification of OpenTelemetry the Trace-ID is by default randomly generated, but it offers an API to customize the Trace-ID generation:

The SDK MUST by default randomly generate both the TraceId and the SpanId.
The SDK MUST provide a mechanism for customizing the way IDs are generated for both the TraceId and the SpanId.
Source

Unfortunately, AWS X-Ray has a more specific format: It expects a version number and the unix timestamp in the beginning of the Trace-ID. If X-Ray receives a randomly generated Trace-ID it might be silently ignored because it considers the Trace-ID as invalid or outdated.

To solve such issues, OpenTelemetry provides an interface to overwrite the Trace-ID generation. For this purpose, a class must be added to the application that references it.

<dependency>
  <groupId>io.opentelemetry.contrib</groupId>
  <artifactId>opentelemetry-aws-xray</artifactId>
  <version>1.14.0</version>
</dependency>

@Singleton
public class AwsOpenTelemetryIdGeneratorProducer {

  @Produces
  @Singleton
  public IdGenerator idGenerator() {
    return AwsXrayIdGenerator.getInstance();
  }
}

3. Use the AWS Context Propagation

Whenever microservice A calls microservice B the same Trace-ID must be used to report the tracing information in order to link both requests into a single distributed trace. To achieve this requirement, microservice A needs to pass the Trace-ID as part of the request to microservice B. This feature is called “Context Propagation”.

Some AWS services, such as the AWS API Gateway or the AWS Elastic Load Balancer, can be configured to add the X-Ray Trace-ID to each incoming request. Typically, these services are the entry point into the microservices landscape and thus the beginning of the trace context. It is important to use the X-Ray context propagation format to receive the trace context from AWS services like API Gateway or ELB and pass it to downstream services like DynamoDB.

There are diffent HTTP header formats, but the result is the same. The W3C Trace Context is specified by the W3C and is used by OpenTelemetry by default. Unfortunately, X-Ray uses a different, more specific format. However, OpenTelemetry provides an extension for the X-Ray format, so it can be easily adopted.

Add the OpenTelemetry extension to the pom.xml

<dependency>
  <groupId>io.opentelemetry</groupId>
  <artifactId>opentelemetry-extension-aws</artifactId>
</dependency>

and enable the xray propagator in the application.properties

quarkus.opentelemetry.propagators=xray

As a result the traceparent header is changed to X-Amzn-Trace-Id.

>> GET /42 HTTP/1.1
>> Accept: text/plain
>> traceparent: 00-fde5c586b403d130a7cb756ccb05fb02-a01a988d41694782-01
>> Host: localhost:8080
>> User-Agent: Apache-HttpClient/4.5.13 (Java/18.0.1)

>> GET /42 HTTP/1.1
>> Accept: text/plain
>> X-Amzn-Trace-Id: Root=1-641c1bd3-d485eee618a90c8ef3bc3978;Parent=64f85711e0109ce6;Sampled=1
>> Host: localhost:8080
>> User-Agent: Apache-HttpClient/4.5.13 (Java/18.0.1)

4. Run the OpenTelemetry Collector

A key conecpt of OpenTelemetry is the Collector. It is a single service that receives tracing data from an application, processes the data and exports them to the specific backend such as Jaeger or AWS X-Ray. The OpenTelemetry Collector can be deployed as a standalone service or as a sidecar container in Kubernetes and others.

Support for Jaeger is included in the standard version of OpenTelemetry, but it does not come with support for AWS X-Ray. Fortunately, AWS steps in and provides a customization for OpenTelemetry called the AWS Distro for OpenTelemetry that provides AWS X-Ray support. It is quite easy to run the collector as a sidecar in ECS or EKS, AWS provides it as a Docker image along with good documention for various environments. For ECS it is even simpler, the AWS console provides a checkbox to enable tracing support with OpenTelemetry.

Afterwards, the microservice running in ECS can access the OpenTelemetry collector on localhost:4317. Therefore there is no need to change the oltp-endpoint in the application.properties

Observing the results with a sample application

To show the results, I created an almost useful example microservice called is-odd that determines if a given number is odd. Calling the example application with GET /42 returns false. A second microservice is-even uses is-odd to negate its result and indicate whether a given number is even. To complete the overall microservice architecture, both microservices are deployed on ECS and equipped with an API gateway that has X-Ray support enabled. Both applications can be found on Github

The result is no surprise: both the service map and the segments timeline in CloudWatch show that the request travels from the client through the API gateway to the sample applications is-even and is-odd in AWS ECS. It can be observed that the API gateway causes the largest delay for the request shown in the screenshot, however, the overall response time of just 8 ms is pretty fast.

In summary, the proposed solution offers two key benefits:

• Traces cover the entire microservice landscape, including self-developed applications and AWS-hosted services.
• There is no need to integrate a javaagent. The native executable mode of Quarkus is supported.

Comments are welcome on Twitter or LinkedIn.

Until today.

I discovered, that the anchor-element has a download attribute.

<a href="35567e83.pdf" download="report-2022.pdf">
  Download Report
</a>

By clicking this link, the browser will download the file instead of linking to it and saves it as report-2022.pdf.

The browser support for this attribute is pretty good with support by all major desktop and mobile browsers.

More details can be found at the MDN docs.

Comments are welcome on Twitter or LinkedIn.

Introducing the Example

In my last post, I showed how to use TypeScript to check translations and their usage at compile time. I would like to briefly recall this example.

I defined a type DeepKeysOf, which returns all possible nested keys from one object.

type DeepKeysOf<T, Key extends keyof T = keyof T> = Key extends string
  ? T[Key] extends string ? Key : `${Key}.${DeepKeysOf<T[Key]>}`
  : never;

If this type definition is used for the translate method now, the compiler can already point out faulty translations.

const translation = {
  dialog: {
    title: 'Bestätigung',
    description: 'Möchten Sie fortfahren?'
  }
}

type TranslationKey = DeepKeysOf<typeof translation>; // 'dialog.title' | 'dialog.description'

const translate = (translationKey: TranslationKey): string => {
  // omit the actual implementation
}

translate('dialog.title'); // compiles fine
translate('dialog.header'); // compiler error: Argument of type '"dialog.header"' is not assignable to parameter of type 'TranslationKey'.

The full example with more detailed explanations can be found in my previous post or in this TypeScript Playground.

Testing

Looking at the implementation, some issues stand out that need to be addressed:

• The definition is quite complex and not obvious at first sight.
• Check if the implementation works as expected, in the good case as well as in the bad case.
• Refactorings should not lead to new errors.
• Document the use of DeepKeysOf.

All these points apply in general to any code, not only to type definitions. The key is to write unit tests. And this is exactly what I want to do for TypeScript’s type definitions.

The big difference to usual unit tests is that type definitions exist only at compile time. This implies that the tests also have to be executed by the compiler rather than at runtime like all other unit tests. This sounds complicated, but is actually fairly easy. In the following I want to briefly introduce the possibilities with ts-expect-error and expect-type.

ts-expect-error

ts-excpect-error was introduces with TypeScript 3.9. This instructs TypeScript to expect a compiler error on the following line. If a line is preceded by a // @ts-expect-error comment, TypeScript suppresses the reporting of this error; but if there is no error, TypeScript reports that // @ts-expect-error was not necessary.

translate('dialog.title');

// @ts-expect-error
translate('dialog.header');

The actual type definition of DeepKeysOf can be tested without using the translate method:

// @ts-expect-error
const invalidKey: DeepKeysOf<typeof translation> = 'dialog.header';

A complete example can be found at TypeScript Playground.

This is a very simple approach and can be used without additional libraries. However, the solution is not very verbose, in unit test usually the expected result is explicitly given.

expect-type

expect-type is a library developed for exactly this purpose: Testing type definitions.

const translation = {
  dialog: {
    title: 'Bestätigung',
    description: 'Möchten Sie fortfahren?'
  }
}

type TranslationKey = DeepKeysOf<typeof translation>;

expectTypeOf<'dialog.title'>().toMatchTypeOf<TranslationKey>();
expectTypeOf<'dialog.header'>().not.toMatchTypeOf<TranslationKey>();

In my opinion expectTypeOf is a very smart solution, but it adds another dependency to the project.

Comments are welcome on Twitter or LinkedIn.

const translation = {
  helloWorld: 'Hallo Welt!',
  dialog: {
    title: 'Bestätigung',
    description: 'Möchten Sie fortfahren?'
  },
  actions: {
    buttons: {
      submit: 'Bestätigen',
      cancel: 'Abbrechen'
    }
  }
}

To retrieve and display the translation at runtime there is a method called translate which returns the corresponding translation for a key from the translation object:

type StringOnlyJson = string | { [property: string]: StringOnlyJson };

const translate = (translationKey: string): string => {
  const result = translationKey
    .split('.')
    .reduce(
      (json, propertyName) => typeof json === 'object' ? json[propertyName] : undefined,
      translation as StringOnlyJson | undefined
    );
  return typeof result === 'string' ? result : translationKey;
};

If the method gets called with dialog.title, it will return Bestätigung:

translate('dialog.title'); // returns 'Bestätigung'

But what happens if an invalid or incorrect translation key is passed to the method? The method cannot find the corresponding translation and returns the passed key as a fallback. In the following example dialog.header is passed instead of dialog.title.

translate('dialog.header'); // returns 'dialog.header'

In my experience this error pattern occurs quite often. A developer simply makes a typo in the translations or changes the naming of a translation key without adjusting them at each source code location. This results in the user seeing only the technical key instead of the expected translation. Such a fallback is helpful because the user can likely continue working with the application instead of getting a blank label or even worse an error message.

As mentioned at the beginning, translations are usually loaded at runtime. Therefore, such errors do not occur earlier than at runtime. To detect and avoid such defects, an extensive test suite or a high manual testing effort is required. In worst case, such defects occur in production and are displayed to the end user.

For this reason, errors should be found as early as possible and that is usually at compile time.

Wouldn’t it be great if the translation keys could be checked automatically? Let’s jump into the power of TypeScript.

In TypeScript there are these String Literal Types. Isn’t it possible to check the translation keys at compile time and inform the developer about his mistake? It would only need a list of all possible translation keys. That is worth a try:

type TranslationKey =
  | 'helloWorld'
  | 'dialog.title'
  | 'dialog.description'
  | 'actions.buttons.submit'
  | 'actions.buttons.cancel';

Afterwards the signature of the translate method can be changed to use the type TranslationKey for the translationKey parameter instead of just string:

const translate = (translationKey: TranslationKey): string => {
  // same code as above
};

This change leads to the fact that the method translate must only be called with one of the previously defined values. All other strings are treated as errors by the compiler.

Back to the example from above. What happens if the translate method gets called with the correct dialog.title and the incorrect dialog.header?

translate('dialog.title'); // compiles fine
translate('dialog.header'); // compiler error: Argument of type '"dialog.header"' is not assignable to parameter of type 'TranslationKey'.

The compiler gives an error message on the second call. The program does not compile and the developer is forced to correct the mistake.

This solution works very well and is easy to implement. Problem solved. :-)

Well…

This solution requires the developer to maintain all translation keys twice: once in the actual translation and a second time in the definition of the TranslationKey type. These two definitions must always be kept in sync to avoid the above mentioned errors of missing translations. This process is tedious, error-prone and in the end does not lead to any improvement.

Is there no way to create the TranslationKey type automatically? The TypeScript compiler would only have to extract the translation keys from the JSON object and concatenate them with a dot.

Indeed, TypeScript can derive the TranslationKey!

type DeepKeysOf<T, Key extends keyof T = keyof T> = Key extends string
  ? T[Key] extends string ? Key : `${Key}.${DeepKeysOf<T[Key]>}`
  : never;

type TranslationKey = DeepKeysOf<typeof translation>;

The type TranslationKey is identical to the manual definition from above. The behavior of the compiler is as well identical, a call to the translate method with an incorrect translation key will raise an error.

But what exactly does the DeepKeysOf type do? Let’s look at the crucial part first:

T[Key] extends string ? Key : `${Key}.${DeepKeysOf<T[Key]>}`

T is the type definition of the translation object, Key is a property of the translation object, T[Key] is therefore the value of this property. T[Key] can be either a string, like 'Hello World!' or another object, like the value of dialog. In the first case, the execution can be stopped and the result is simply helloWorld. In the second case a recursion is called, which adds . to the Key and again uses the type definition DeepKeysOf for the object of T[Key]. With the help of this recursion it is possible to use arbitrarily deeply nested translation objects.

A practical example for DeepKeysOf<typeof translation>: T is the entire translation object, Key is a property of this, i.e. helloWorld, dialog or action. T[Key] is the value of this property, for the Key helloWorld it is 'Hello World!' , for dialog it is the object { title: 'Bestätigung', description: 'Möchten Sie fortfahren?' }. Thus, if Key is helloWorld then the expression T[Key] extends string holds true and thus the result of the expression will be helloWorld. On the other hand, if Key is dialog, then T[Key] is an object, the expression holds false, and the result is a concatenation of dialog. (including the dot) with the result of DeepKeysOf<T['dialog']>.

However, it still has to be clarified how to iterate through the different properties within an object. For this purpose keyof and a type alias named Key is used: Key extends keyof T = keyof T. keyof T is an alias for all properties of the object T and allows in that way an iteration through all properties. Key then contains the current property selected by the iteration through keyof T. The actual iteration is performed by TypeScript itself.

As a last point there is the wrapper Key extends string ? ... : never around the actual expression (abbreviated by …). In the translation object Key is always a string, so this expression is actually not relevant. But TypeScript does not actually know this, because this has not been defined. But for the later concatenation with . TypeScript expects a string (or several other types). By the way, the else branch with the result never is not called with the translation object. But if the object would contain some keys which are not string (i.e. number, or similar), then using never the corresponding invalid branches in the input object would be ignored.

This entire implementation is also available for simple follow up on the TypeScript Playground.

In the end, this solution is very powerful, thanks to TypeScript’s extensive type system. Translations are no longer as error-prone as I know from my past. Overall, a single type definition increases the quality of the software and this quality can also be checked automatically at compile-time.

Comments are welcome on Twitter or LinkedIn.

• The events in the timeline are not equally distributed and the space between the events need to follow that.
• Since the timeline might contain a huge amount of events, a user needs to be able to zoom into the timeline.
• If the full timeline is not visible anymore, the user needs to be able to scroll left and right by grabbing the timeline.

I’ll omit any labels or fancy details and focus on the main challenges only.

1. Fully working example
2. Easy Peasy - the Flexbox magic
3. Scrolling
4. Zooming first attempt
5. Fix the calculations

Fully working example

First, let’s take a look at the fully working example. It will give a better impression of the solution than the dry list of requirements.

Use the mouse wheel to scroll in and out. Grab the timeline to move it left and right.

It’s implemented with pure HTML, CSS and JavaScript without a single dependency. 😇

You can find the full source code on Github.

Easy Peasy - the Flexbox magic

Let’s get started: A simple timeline requires only very few lines of CSS. It fills the whole screen and rescales well for different screen sizes. It employs a simple Flexbox layout with a flex-direction: row to keep the events in a row. But the events should not be distributed evenly, the space between them should differ and be customizable.

Thanks to Flexbox it’s quite simple: Just put a div between the events that has a flex-grow property. A div with flex-grow: 8 will get four times the space of a flex-grow: 2. The browser will then scale the size between the events accordingly to the flex-grow property. To put this abstract property to the timeline we can assume that the number of days between two events is set to the flex-grow property, e. g. if two events are 30 days apart, we would use flex-grow: 30. Easy peasy, right? I cannot image how painful an implemention would look like without the power of Flexbox.

The result looks quite good. And yet, it wasn’t a challenge to make it happen. So far, there is no reason for a longish blog post like this. But the task will get much harder in the next section. Be warned!

For now the scrollable and zoomable classes are needless, but they will get important very soon. I want to drop the full HTML and CSS here so that I don’t need to paste it later on again.

<div class="scrollable">
  <div class="timeline zoomable">
    <div class="spacer" style="width: 26px;"></div>
    <div class="event"></div>
    <div class="spacer" style="flex-grow: 1;"></div>
    <div class="event"></div>
    <div class="spacer" style="flex-grow: 2;"></div>
    <div class="event"></div>
    <div class="spacer" style="flex-grow: 3;"></div>
    <div class="event"></div>
    <div class="spacer" style="flex-grow: 5;"></div>
    <div class="event"></div>
    <div class="spacer" style="flex-grow: 8;"></div>
    <div class="event"></div>
    <div class="spacer" style="width: 26px;"></div>
  </div>
</div>

.scrollable {
  overflow-x: hidden;
  width: 100%;
}

.timeline {
  display: flex;
  flex-direction: row;
  align-items: center;
  height: 104px;
  background-color: #f2f3f3;
}

.event {
  width: 26px;
  height: 26px;
  border-radius: 100%;
  background-color: rgb(0, 127, 255);
}

.spacer {
  height: 6px;
  background-color: rgba(0, 127, 255, 0.5);
}

Scrolling

Scrolling consists of multiple features:

• change the mouse point to grabbing when pressing the mouse button and back to grab when releasing it
• save the mouse position whe pressing the mouse button
• move the timeline accordingly to the left or right when the mouse is moved and the button is pressed

The latter feature is implemented by the scrollLeft property of the container. Every div has a scrollLeft property which is set to 0 by default. If the content of the container is wider than the container itself, the scrollLeft property can be used to drag the content to left so that the wider content can be capped on the left side.

An example: The visible container is 350 px wide and overflow-x is set to hidden. It contains an element with double width, i. e. 700 px. Without further specification, the browser would set the child element left-aligned with the visible container and cut off the right half. But with the help of the scrollLeft property the too wide child element can be dragged to the left, so that an arbitrary section can become visible. To show the center and cut off the same amount on both sides, scrollLeft would have to be set to 175 px in this example.

Since there is no way to ask the browser if a user has the mouse button pressed right now, we need to listen to the mousedown and mouseup events and to remember the state manually. In addition when the mousedown event fires, we save the current position of the mouse pointer and the current scrollLeft position as initial values. We need those values later on to calculate the mouse distance.

Actually, it’s quite simple to implement that. The full source code can be viewed on Github, I’ll walk through some major steps.

mousedown

When the mouse button gets pressed, the current mouse position and the current scrollLeft value needs to be stored. In addition, the cursor style is changed to grabbing.

scrollableElement.addEventListener('mousedown', (mouseEvent) => {
  mouseDown = true;
  scrollableElement.style.cursor = 'grabbing';
  initialGrabPosition = mouseEvent.clientX;
  initialScrollPosition = scrollableElement.scrollLeft;
});

mouseup

When the user releases the mouse button, the values that were changed in the mousedown event must be reset.

scrollableElement.addEventListener('mouseup', () => {
  mouseDown = false;
  scrollableElement.style.cursor = 'grab';
});

mousemove

When the user moves the mouse, a very few mathematical calculations are required. The distance between the starting point of grabbing action and the current position of the mouse pointer must be computed and the value for scrollLeft must be reduced by exactly the same value.

scrollableElement.addEventListener('mousemove', (mouseEvent) => {
  if (mouseDown) {
    const mouseMovementDistance = mouseEvent.clientX - initialGrabPosition;
    scrollableElement.scrollLeft = initialScrollPosition - mouseMovementDistance;
  }
});

That’s it. Now the user can move the timeline left and right if it doesn’t fit on the screen. By default, the timeline is exactly as wide as the window, so the next task will be to implement the zoom behavior to actually use the scrolling feature.

Zooming - first attempt

The first challenge will be a draft implementation of the zooming feature. Spoiler: it won’t work as excepted and it took me ages to figure out whats going wrong.

The idea of the zoom function is to enlarge the timeline beyond the visible width by using the width property and set it to more than 100 % for the timeline itself and overflow-x: hidden for the its parent. The result is a timeline that is larger than the visible screen, but cropped to the original size. The spacing between events is doubled, so it feels like the timeline has been zoomed in.

If the width of the child element changes, it gets stretched to the right and therefore cropped on the right. To change the focus, the scrollLeft property needs to be changed accordingly.

My first attempt was to calculate the scrollLeft property according to the mouse position. If the mouse pointer is between the third and the fourth quarter and the user zooms in, the mouse pointer has to stay on that point. So the user has to move the timeline in such a way that the he gets the impression of thetimeline moving around his mouse pointer. It looks like he zoomed in at that exact spot. My first (and incorrect) assumption was to simply calculate the space difference and increase the scrollLeft property to match the mouse position accordingly.

An example: The timeline has a width of 100 px, we zoom in to make the timeline 200 px wide, so it has doubled in size. The mouse pointer is positioned between the third and fourth quarter, where it must be positioned also after zooming.

scrollLeft must be computed in such a way that 75 % of the timeline is still on the left of the mouse pointer after zooming. In this example scrollLeft is increased by 100 px * 0.75, so it gets the value 263.

Looks simple? Yes, it actually is. But …

Does it work?

Let’s look again at the example of the timeline, which is zoomed in by the factor 2. This time, however, the example timeline will be in a more detailed way, representing the events and spaces between them.

The mouse pointer is again positioned at three-quarters, which in this example is at the end of the last element.

In the next step, the user zooms into the timeline so that the width increases to twice the original width.The algorithm is applied and the mouse pointer is moved back to its original position. Before and after zooming, the mouse pointer is over a point that is about three quarters wide.

The result is as expected. Or is it? The mouse pointer has the correct position relative to the timeline itself, but the mouse pointer is far away from the last element and no longer right next to it. To the user, it no longer seems as if the center of the zooming is below the mouse pointer. But that is exactly the expected behavior.

What should it look like properly?

It is clear to see that the timeline is pushed significantly further to the right than the algorithm did. As can be seen, the scrollLeft property must be only roughly half the size. Where is the difference compared to the first simple example?

Well, … The timeline consists of variable spacing between the points and the points themselves with fixed width. When the user scales the timeline, the spaces are changed, but the events are not.

Simple example: Assume a Timeline which consists on the left half of 2 points with one length unit each, in addition there are 2 spaces with two length units each. In total, the left half of this imaginary Timeline is 6 length units (2*1 + 2*2). The right half consists of only one distance with 6 length units.

If the user increases the timeline by a factor of two, all spacings get doubled, but not the events. On the left side the resulting size is 10 (2*1 + 2*4), whereas on the right half the distance doubles to 12. Thus the center of the timeline has shifted.

For single elements like an image the simple algorithm works well. Even if all elements scale the same, it works. But the timeline consists of the variable spacing between the points and the points with fixed widths. The effect is that the timeline does not scale proportionally.

This finding took me several days (and sleepless nights). In the end, the problem was simply some elementary school math.

Fix the calculations

The idea of the algorithm was quite correct. The timeline must be moved so that the event under the mouse pointer is fixed. The only difference is that this point not point of the timeline as a whole, but a child element of the timeline.

First, it needs to be identified at which event or space the mouse pointer is located and relative to that element the original algorithm can be applied. In the example from above, the last point of the timeline is identified as the element below the mouse pointer. Relative to this element, the mouse pointer is at the very end. And it is exactly this position that we would have to reach again after zooming.

This sounds quite simple, but for the calculation of the scrollLeft property there are some additional information required. The position of the mouse pointer is measured from the edge of the screen. To compute the correct value the algorithm also needs all other relevant values in relation to the edge of the screen. This includes the distance of the timeline, its parent container as well as the the child element within the timeline where the mouse pointer is positioned.

The code for this looks like this:

const mousePosition = wheelEvent.clientX;
const elementUnderMouseLeft = getLeft(elementUnderMouse);
const zoomableLeft = getLeft(zoomableElement);
const containerLeft = getLeft(containerElement);
const moveAfterZoom = getWidth(elementUnderMouse) * mousePositionRelative;

containerElement.scrollLeft =
  elementUnderMouseLeft
  - zoomableLeft
  - mousePosition
  + containerLeft
  + moveAfterZoom;

You can find the full source of the zooming feature code on Github.

Comments are welcome on Twitter or LinkedIn.

The application takes a huge amount of memory during processing the data. But when the processing job has been completed, the memory can be released and returned to the operating system. Unfortunately, this doesn’t happen. The JVM seems to keep the memory forever.

What’s going on?

Currently the G1 garbage collector may not return committed Java heap memory to the operating system in a timely manner. G1 only returns memory from the Java heap at either a full GC or during a concurrent cycle. Since G1 tries hard to completely avoid full GCs, and only triggers a concurrent cycle based on Java heap occupancy and allocation activity, it will not return Java heap memory in many cases unless forced to do so externally. – JEP 346

The motivation of JEP 346 describes perfectly what I observed from my application. When it should release memory, there’s no need anymore to run the garbage collector at all. Therefore, it will keep the memory until the next huge processing phase starts and requires the memory again.

How can this be fixed?

JEP 346 has exactly this issue in mind: Promptly Return Unused Committed Memory from G1. JEP 346 has been implemented in Java 12. Unfortunately, I have to use Java 11 and cannot benefit from this improvment.

But there are other garbage collectors than the default G1. Ruslan Synytsky has a well-written blog post about the memory consumption of different garbage collectors.

Based on his observations there’s a huge difference in the behaviour of the memory consumption. It seems that the Shenandoah GC might be a really good option for my application. Next to ZGC, Shenandoah is one of the newest garbage collectors and available as production ready in Java 15. But Shenandoah has been backported to OpenJDK 11 and is available since version 11.0.9. Therefore I’m able to use it. Let’s give it a try:

java -XX:+UseShenandoahGC -jar app.jar

It works! As you can see, some time after finishing the hard processing work, Shenandoah returns most of the memory back to the operating system.

Since reducing the heap is an expensive operation, Shenandoah takes a delay of 5 minutes (300,000 ms) by default to release any memory back to the operating system. You can set Shenandoah to be more aggressive, but the command line options for tuning Shenandoah are still marked as experimental. However, I’m fine with the default 5 minute delay.

java \
  -XX:+UseShenandoahGC \
  -XX:+UnlockExperimentalVMOptions \
  -XX:ShenandoahUncommitDelay=1000 \
  -XX:ShenandoahGuaranteedGCInterval=10000 \
  -jar app.jar

Comments are welcome on Twitter or LinkedIn.

Links

• JEP 346: Promptly Return Unused Committed Memory from G1
• Garbage Collector Tuning as the First Step to Java Memory Usage Optimization
• Shenandoah GC in production: experience report
• Stackoverflow: Does GC release back memory to OS?
• Stackoverflow: Does G1 GC release back memory to the OS even if Xms = Xmx?

Even if Java Streams are a good choice for many problems, they might be not the best choice for readability. I want to show you an example where I struggeled for a good solution. Java Streams sound like the perfect solution for that kind of problem.

But let’s get started:

The problem

I cannot give any details about the actual business requirement about that. But I hopefully got a really neat story instead.

I want to get a list of all people and their pets and the possibiliets they can go for a walk. Each person wants to walk with all of their pets in all possible combinations. But it is not allowed two walk two pets of the same kind at the same time.

Example: Thomas has two dogs, two cats and a pig. The result shold be:

Keep in mind that even if Thomas would have no cat, he wants wo walk his dogs and his pig.

Start with some coding

The people and animals are fetched from independent external sources and could look like that:

List<String> people = getPeopleInTown("Freiburg");

Map<String, List<Cat>> = getCatsForPeople(people);
Map<String, List<Dog>> = getDogsForPeople(people);
Map<String, List<Pig>> = getPigsForPeople(people);

To model a Walk the simple Java class will be used:

public class Walk {

	String person;
	Cat cat;
	Dog dog;
	Pig pig;

	public Walk(String person, Cat cat, Dog dog, Pig pig) {
		this.person = person;
		this.cat = cat;
		this.dog = dog;
		this.pig = pig;
	}

	// getters and setters omitted for brevity
}

To build the walk, we introduce a buildWalks method.

public List<Walk> buildWalks(
    List<String> people,
    Map<String, List<Cat>> cats,
    Map<String, List<Dog>> dogs,
    Map<String, List<Pig>> pigs) {

  List<Walk> walks = new ArrayList<>();

  // do the magic

  return walks;
}

The naive implementation

My first appraoch was to use just a couple of for-loops. It looks gorgeous and is easy to read.

public List<Walk> buildWalks(
    List<String> people,
    Map<String, List<Cat>> cats,
    Map<String, List<Dog>> dogs,
    Map<String, List<Pig>> pigs) {

  List<Walk> walks = new ArrayList<>();

  for(String person : people) {
    for (Cat cat : cats.get(person)) {
      for (Dog dog : dogs.get(person)) {
        for (Pig pig : pigs.get(person)) {
          walks.add(new Walk(person, cat, dog, pig));
        }
      }
    }
  }

  return walks;
}

Unfortunately, it doesn’t work. If Thomas has no cat, he won’t be able to do any walk. That’s bad for his other pets.

If the list of cats is empty, the for-loop will not be executed and therefore the walks.add()-method will never be called.

Ok, let’s fix it. We need to make sure that every for-loop will be executed at least once:

public List<Walk> buildWalks(
    List<String> people,
    Map<String, List<Cat>> cats,
    Map<String, List<Dog>> dogs,
    Map<String, List<Pig>> pigs) {

  List<Walk> walks = new ArrayList<>();

  for(String person : people) {
    for (Cat cat : atLeastOnce(cats.get(person))) {
      for (Dog dog : atLeastOnce(dogs.get(person))) {
        for (Pig pig : atLeastOnce(pigs.get(person))) {
          walks.add(new Walk(person, cat, dog, pig));
        }
      }
    }
  }

  return walks;
}

private <T> List<T> atLeastOnce(List<T> animals) {
  if (animals == null || animals.isEmpty()) {
    return getNulledList();
  }
  return animals;
}

private <T> List<T> getNulledList() {
  List<T> list = new ArrayList<>();
  list.add(null);
  return list;
}

Streams to the rescue?

Let’s use Java Streams to implement the buildWalks() again.

public List<Walk> buildWalks(
    List<String> names,
    Map<String, List<Cat>> cats,
    Map<String, List<Dog>> dogs,
    Map<String, List<Pig>> pigs) {

  List<Walk> walks = new ArrayList<>();

  names.forEach(name -> {
    forEachAtLeastOnce(cats.get(name), cat -> {
      forEachAtLeastOnce(dogs.get(name), dog -> {
        forEachAtLeastOnce(pigs.get(name), pig -> {
          walks.add(new Walk(name, cat, dog, pig));
        });
      });
    });
  });

  return walks;
}

private T forEachAtLeastOnce(List<T> animals, Consumer<T> consumer) {
  if (animals == null || animals.isEmpty()) {
    consumer.accept(null);
  }
  animals.forEach(consumer);
}

If you look at the code, do you see at a glance what’s going on? Indee, I need some time to read through every single line to know what the method actually returns.

Conclusion

The main difference of the implementations is the idea how the permuations will be created.

for(String person : people) {
  for (Cat cat : atLeastOnce(cats.get(person))) {
    for (Dog dog : atLeastOnce(dogs.get(person))) {
      for (Pig pig : atLeastOnce(pigs.get(person))) {
        walks.add(new Walk(person, cat, dog, pig));
      }
    }
  }
}

names.forEach(name -> {
  forEachAtLeastOnce(cats.get(name), cat -> {
    forEachAtLeastOnce(dogs.get(name), dog -> {
      forEachAtLeastOnce(pigs.get(name), pig -> {
        walks.add(new Walk(name, cat, dog, pig));
      });
    });
  });
});

If you read the code, do you prefer nested for-loops or streams?

IMHO the nested for-loops can be understood more easily than streams. Therefore, I prefer the nested for-loops.

Please keep in mind, that Java Streams is a powerful performance improvement when you have to deal with large data sets. But you need to take special attention on readability if you don’t want to get a trade-off for your readers.

Comments are welcome on Twitter or LinkedIn.

userList.stream()
  .filter(user -> user != null)
  .filter(user -> !user.isActivated())
  .count();

The idea of this code snippet is to count all deactivated users. At first we have to filter all nulls and secondly remove all activated users. The remaining list entries can just be counted.

I want to focus on the two filter methods. IMHO using method references should be preferred. Let me show that by an example which filters for the opposite. Please keep in mind that this will throw a NullPointerException since we call isActivated only on nulls.

userList.stream()
  .filter(Objects::isNull)
  .filter(User::isActivated)
  .count();

Wouldn’t you agree that the method reference User::isActivated is much easier to read than user -> !user.isActivated()? But we do not want to filter for activated users, but for de-activated users. We could achieve that be implementing a isDeactivated method in the User class, but I want to show you a different way.

It’s the very same idea for remove null objects from the list. It’s really nice to just write .filter(Objects::isNull) to find only nulls, but I always wrote .filter(u -> u != null) in the past to get only non-nulls.

Until today, I never took the time to look for an easier way. But there are two very nice (and obvious) solutions:

Objects::nonNull

In the Objects class there exists not only the isNull method, but also nonNull to achieve exactly the behaviour I need.

userList.stream()
  .filter(Objects::nonNull)
  .count()

Predicate::not

The Predicate interface has a handy not method to negate any Predicate. It’s concise and easy to understand when using a static import for the Predicate.not method

userList.stream()
  .filter(not(Objects::isNull))
  .filter(not(User::isActivated))
  .count();

Predicate.not is more powerful, it allows to negate any filter option, e.g. to remove empty lists out of a list of lists:

By combining the two approaches, we are again able to use method references to count all deactivated users:

userList.stream()
  .filter(Objects::nonNull)
  .filter(not(User::isActivated)
  .count();

Filtering Java Streams can be very concise by using method references. To achieve the opposite of a provided method, Predicate.not is a handy solution.

Comments are welcome on Twitter.

import static java.util.Collections.unmodifiableList;
import static java.util.stream.Collectors.toList;

import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
import java.util.function.BinaryOperator;
import java.util.stream.Stream;

public final class CartesianProductUtil {

  private CartesianProductUtil() { }

  /**
   * Compute the cartesian product for n lists.
   * The algorithm employs that A x B x C = (A x B) x C
   *
   * @param listsToJoin [a, b], [x, y], [1, 2]
   * @return [a, x, 1], [a, x, 2], [a, y, 1], [a, y, 2], [b, x, 1], [b, x, 2], [b, y, 1], [b, y, 2]
   */
  public static <T> List<List<T>> cartesianProduct(List<List<T>> listsToJoin) {
    if (listsToJoin.isEmpty()) {
      return new ArrayList<>();
    }

    listsToJoin = new ArrayList<>(listsToJoin);
    List<T> firstListToJoin = listsToJoin.remove(0);
    Stream<List<T>> startProduct = joinLists(new ArrayList<T>(), firstListToJoin);

    BinaryOperator<Stream<List<T>>> noOp = (a, b) -> null;

    return listsToJoin.stream() //
        .filter(Objects::nonNull) //
        .filter(list -> !list.isEmpty()) //
        .reduce(startProduct, CartesianProductUtil::joinToCartesianProduct, noOp) //
        .collect(toList());
  }

  /**
   * @param products [a, b], [x, y]
   * @param toJoin   [1, 2]
   * @return [a, b, 1], [a, b, 2], [x, y, 1], [x, y, 2]
   */
  private static <T> Stream<List<T>> joinToCartesianProduct(Stream<List<T>> products, List<T> toJoin) {
    return products.flatMap(product -> joinLists(product, toJoin));
  }

  /**
   * @param list   [a, b]
   * @param toJoin [1, 2]
   * @return [a, b, 1], [a, b, 2]
   */
  private static <T> Stream<List<T>> joinLists(List<T> list, List<T> toJoin) {
    return toJoin.stream().map(element -> appendElementToList(list, element));
  }

  /**
   * @param list    [a, b]
   * @param element 1
   * @return [a, b, 1]
   */
  private static <T> List<T> appendElementToList(List<T> list, T element) {
    int capacity = list.size() + 1;
    ArrayList<T> newList = new ArrayList<>(capacity);
    newList.addAll(list);
    newList.add(element);
    return unmodifiableList(newList);
  }
}

The code is also available on this GitHub Gist.

Comments are welcome on LinkedIn.