Since sharing posts on Medium has more social features, future posts will be located there. Projects and other things will remain on this site.

While adjusting some environment variables recently, I came across an odd issue with Docker, Spring Boot and JAVA_OPTS. JAVA_OPTS comes from the Tomcat/Catalina world and when searching for “Docker and javaopts” on Google you’ll find many references to just adding JAVA_OPTS to the Docker environment. After some testing, I found this to be incorrect when running a Spring Boot jar in a Docker container, I’ll explain why and give a solution in this post.

Before I start, let’s setup a basic test environment that prints out the current memory setup so we can test in various situations. I’ve created this repo as a test case and you can refer back to it when needed.

Test Application Setup

Looking at the sample code, we have a basic pom file that imports Spring Boot dependencies:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

<modelVersion>4.0.0</modelVersion>

<groupId>vc.c4.javaopts</groupId>
<artifactId>example-javaopts</artifactId>
<version>0.0.1-SNAPSHOT</version>

<name>Spring Boot Tomcat JAVA_OPTS</name>
<description>Spring Boot Tomcat JAVA_OPTS Example</description>
<url>https://github.com/cl4r1ty/spring-boot-javaopts/</url>

<properties>
<java.version>1.7</java.version>
<main.basedir>${basedir}/../..</main.basedir>
<spring-boot.version>1.3.1.RELEASE</spring-boot.version>
</properties>

<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>${spring-boot.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>

<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>${spring-boot.version}</version>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

</project>

The main application is a single Java file that prints out memory details:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
package vc.c4.javaopts;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

import java.text.NumberFormat;

@SpringBootApplication
public class JavaOptsApp {

private static Log logger = LogFactory.getLog(JavaOptsApp.class);

public static void main(String[] args) throws Exception {
SpringApplication.run(JavaOptsApp.class, args);
Runtime runtime = Runtime.getRuntime();

final NumberFormat format = NumberFormat.getInstance();

final long maxMemory = runtime.maxMemory();
final long allocatedMemory = runtime.totalMemory();
final long freeMemory = runtime.freeMemory();
final long mb = 1024 * 1024;
final String mega = " MB";

logger.info("========================== Memory Info ==========================");
logger.info("Free memory: " + format.format(freeMemory / mb) + mega);
logger.info("Allocated memory: " + format.format(allocatedMemory / mb) + mega);
logger.info("Max memory: " + format.format(maxMemory / mb) + mega);
logger.info("Total free memory: " + format.format((freeMemory + (maxMemory - allocatedMemory)) / mb) + mega);
logger.info("=================================================================\n");
}

}

Running this app in IntelliJ with no memory settings will show some basic output with memory settings:

1
2
3
4
5
6
========================== Memory Info ==========================
Free memory: 221 MB
Allocated memory: 245 MB
Max memory: 3,641 MB
Total free memory: 3,617 MB
=================================================================

Changing the memory values to -Xmx3g -Xms3g we get a different output:

1
2
3
4
5
6
========================== Memory Info ==========================
Free memory: 2,713 MB
Allocated memory: 2,944 MB
Max memory: 2,944 MB
Total free memory: 2,713 MB
=================================================================

Now that we have a baseline, let’s move onto the Docker stage.

Docker

The first thing we’ll do is setup a basic Dockerfile similar to the one in the Spring Boot guide. Since the jar will be built in the target folder we grab it from there and place a renamed copy into the container.

1
2
3
4
5
6
7
8
# Base java:8
FROM java:8

# Add jar to container
ADD /target/example*.jar javaopts.jar

# Entry in json format
ENTRYPOINT ["java", "-jar", "/javaopts.jar"]

To generate the jar, just run maven real quick.

1
mvn clean package

The jar will then be in the target folder ready for building the Docker container.

1
2
3
4
5
6
7
8
9
$ ls -ahl target
total 11760
drwxr-xr-x 7 clarity clarity 238B Dec 22 12:42 .
drwxr-xr-x 13 clarity clarity 442B Dec 22 12:42 ..
drwxr-xr-x 3 clarity clarity 102B Dec 22 12:42 classes
-rw-r--r-- 1 clarity clarity 5.7M Dec 22 12:42 example-javaopts-0.0.1-SNAPSHOT.jar
-rw-r--r-- 1 clarity clarity 3.3K Dec 22 12:42 example-javaopts-0.0.1-SNAPSHOT.jar.original
drwxr-xr-x 3 clarity clarity 102B Dec 22 12:42 maven-archiver
drwxr-xr-x 3 clarity clarity 102B Dec 22 12:42 maven-status

With the jar and the Dockerfile ready we can build the container from the base folder of the repository.

1
2
3
4
5
6
7
8
9
10
11
12
$ docker build -t spring-boot-javaopts .
Sending build context to Docker daemon 6.254 MB
Step 1 : FROM java:8
---> d4849089125b
Step 2 : ADD /target/example*.jar javaopts.jar
---> 518b9e05c3a9
Removing intermediate container cf4ff4db572a
Step 3 : ENTRYPOINT java -jar /javaopts.jar
---> Running in 15f032c1cc5c
---> 29a4656cdd5c
Removing intermediate container 15f032c1cc5c
Successfully built 29a4656cdd5c

We can see the newly built image by listing our Docker images.

1
2
3
spring-boot-javaopts$  docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
spring-boot-javaopts latest 29a4656cdd5c 8 seconds ago 648 MB

With our newly built container that will run Spring boot application let’s run it for the first time! Note: I’ll be cutting out some of the output lines to save space since we only care about the memory detail output.

1
2
3
4
5
6
7
8
docker run spring-boot-javaopts

2015-12-22 20:44:12.516 INFO 1 --- [ main] vc.c4.javaopts.JavaOptsApp : ========================== Memory Info ==========================
2015-12-22 20:44:12.516 INFO 1 --- [ main] vc.c4.javaopts.JavaOptsApp : Free memory: 17 MB
2015-12-22 20:44:12.517 INFO 1 --- [ main] vc.c4.javaopts.JavaOptsApp : Allocated memory: 30 MB
2015-12-22 20:44:12.517 INFO 1 --- [ main] vc.c4.javaopts.JavaOptsApp : Max memory: 485 MB
2015-12-22 20:44:12.518 INFO 1 --- [ main] vc.c4.javaopts.JavaOptsApp : Total free memory: 471 MB
2015-12-22 20:44:12.518 INFO 1 --- [ main] vc.c4.javaopts.JavaOptsApp : =================================================================

First run looks good. Let’s try setting the JAVA_OPTS as we’re used to!

1
2
3
4
5
6
7
8
docker run -e JAVA_OPTS='-Xmx3g -Xms3g' spring-boot-javaopts

2015-12-22 20:45:40.030 INFO 1 --- [ main] vc.c4.javaopts.JavaOptsApp : ========================== Memory Info ==========================
2015-12-22 20:45:40.030 INFO 1 --- [ main] vc.c4.javaopts.JavaOptsApp : Free memory: 16 MB
2015-12-22 20:45:40.031 INFO 1 --- [ main] vc.c4.javaopts.JavaOptsApp : Allocated memory: 30 MB
2015-12-22 20:45:40.031 INFO 1 --- [ main] vc.c4.javaopts.JavaOptsApp : Max memory: 485 MB
2015-12-22 20:45:40.032 INFO 1 --- [ main] vc.c4.javaopts.JavaOptsApp : Total free memory: 471 MB
2015-12-22 20:45:40.034 INFO 1 --- [ main] vc.c4.javaopts.JavaOptsApp : =================================================================

Woah! The Xmx and Xms settings didn’t have any affect on the Java settings in the container! Something must be wrong here and it’s not immediately obvious if all you search for is for “Docker and javaopts” on Google. Spring Boot is smart enough to handle many passed in environment variables but those are application specific. For example, when you look for a value x.y.z, Spring Boot will look in application.properties by default for x.y.z and in the environment variables for X_Y_Z. Due to this, passing in the environment variable -e X_Y_Z=1234 can be used in your application when using a Docker container. However, since JAVA_OPTS are used by Java and not in the application we run into this problem.

After some searching, I came across the fact that JAVA_OPTS are very specific to Catalina (Tomcat). Looking in the bin folder of a tomcat install you’ll find a shell script that handles passing JAVA_OPTS into the exec lines. With this info, we can now look to change the Dockerfile to adjust for handling passed in environment variables. I named this new Dockerfile ExecDockerfile so I can have both in the same repo.

The change needed to enable JAVA_OPTS is to execute the Java line with an ENTRYPOINT exec command and place the environment variable into the line.

1
2
3
4
5
6
7
8
# Base java:8
FROM java:8

# Add jar to container
ADD /target/example*.jar javaopts.jar

# Entry with exec
ENTRYPOINT exec java $JAVA_OPTS -jar /javaopts.jar

Let’s rebuild the container but specify the new ExecDockerfile specifically using the -f flag.

1
2
3
4
5
6
7
8
9
10
11
12
$ docker build -f ExecDockerfile -t spring-boot-javaopts .
Sending build context to Docker daemon 6.254 MB
Step 1 : FROM java:8
---> d4849089125b
Step 2 : ADD /target/example*.jar javaopts.jar
---> Using cache
---> 518b9e05c3a9
Step 3 : ENTRYPOINT exec java $JAVA_OPTS -jar /javaopts.jar
---> Running in 87f0d69e6171
---> a5622e7d302e
Removing intermediate container 87f0d69e6171
Successfully built a5622e7d302e

Let’s run the container again.

1
2
3
4
5
6
7
8
$ docker run -e JAVA_OPTS='-Xmx3g -Xms3g' spring-boot-javaopts

2015-12-22 21:04:03.662 INFO 1 --- [ main] vc.c4.javaopts.JavaOptsApp : ========================== Memory Info ==========================
2015-12-22 21:04:03.662 INFO 1 --- [ main] vc.c4.javaopts.JavaOptsApp : Free memory: 2,740 MB
2015-12-22 21:04:03.663 INFO 1 --- [ main] vc.c4.javaopts.JavaOptsApp : Allocated memory: 2,969 MB
2015-12-22 21:04:03.663 INFO 1 --- [ main] vc.c4.javaopts.JavaOptsApp : Max memory: 2,969 MB
2015-12-22 21:04:03.663 INFO 1 --- [ main] vc.c4.javaopts.JavaOptsApp : Total free memory: 2,740 MB
2015-12-22 21:04:03.664 INFO 1 --- [ main] vc.c4.javaopts.JavaOptsApp : =================================================================

Conclusion

By starting the jar with ENTRYPOINT exec java $JAVA_OPTS, we’ve successfully enabled changing memory settings using Docker environment variables. If you think you’re changing the memory settings currently but are not passing in $JAVA_OPTS into your start line in some fashion, I suggest you look at your Dockerfile config.

We’re all fans of GitHub and their mention system is pretty sweet. The normal use cases are straightforward but what if there were other possible use cases for the same feature? What if a programming language used a similar format as GitHub mentions in the language. Luckily Java does just that.

When writing Java code you can end up using a lot of annotations. Spring makes excellent use of these annotations and has a myriad of uses for them behind the scenes. These include everything from @SpringBootApplication which is a combinations of three other annotations, to @Bean which just defines a bean. Sometimes there are so many annotations you don’t know what to do with them.

[SpringBootApplication annotation]

At this point the idea dawned on me that I could passively monitor GitHub issues, pull requests, etc using the mention system.

The first task was registering the username @Autowired because it sounded like a decent alias so grabbing it seemed like a good idea. Initially there was some oddities in the GitHub account as it was detected as not being human. An email to GitHub support got that changed. As this was the first test of what kind of emails I would get, I sat on it for a few days. Here’s the results over a 5 day period:

[@Autowired]

Looking at the mails that were sent it was obvious that not only was this useful for Spring information but possibly also for any other public project.

After deciding this could be turned into a useful tool I registered a myriad of other usernames that related to Java annotations. After dealing with the not a human problem they’re not just sitting collecting data that I can turn into a feed of information about open source projects.

As an example use case I took these usernames:

  • PreAuthorize
  • PostAuthorize
  • PreFilter
  • PostFilter

These annotations (@PostFilter, etc) are some of the core spring security annotations. It’s possible some of the information I’d receive would be about incorrectly used annotations leading to security issues. Using these users I’m now just siphoning public data looking for security issues

On a side note - You don’t get mentions for private repos which the user is not a part of.

Many months ago I was faced with a normal problem, my ec2 instances disk was full. But why I asked? Why was it running out of space so quickly. What was the actual cause of my loss of disk space. This is the story of my investigation and the a final word on security.

It all started with a page, a message telling me an instance had a full disk, well time to investigate. (I’m replaying this a tad with a vagrant instance as this was so long ago)

First let’s bust out some bash:

1
2
3
4
5
6
7
8
9
ubuntu# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.8G 7.4G 252K 100% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 492M 12K 492M 1% /dev
tmpfs 100M 332K 99M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 497M 0 497M 0% /run/shm
none 100M 0 100M 0% /run/user

Well we’ve got a problem, lets find the root cause. I poked around a little bit by hand and I came across a file with an odd name.

1
257M Feb  2  2012 enron_mongo.tar.bz2

Well that’s unusual, where in the world did that come from. After many find, sort and various other commands later I also found a file called messages.bson which was what was inside the enron_mongo.tar.bz2.

TIP: du -xak .|sort -n|tail -50 can be used to find large files quickly. Try running it from a specific folder or from /.

1
1.4G May 30 01:21 messages.bson

Around this time we were working on analyzing many open source projects and testing out various methods for finding data. Many of these instances weren’t exactly stable but it was early so hard drives got filled up and things crashed here and there. What was unusual was that I appeared to have a copy of the Enron emails now on my server. No suspicious commands were in the history on the server so I went about the task of finding out where the emails came from.

The first idea I had was to search GitHub due to the scanning of various repos so I went about that. The first few items alerted me quickly that this data was a common mongo test setup but specifically I was led to a specific repo https://github.com/mongodb/mongo-hadoop.

Since this appears to be a gradle project I started looking at the build.gradle file https://github.com/mongodb/mongo-hadoop/blob/master/build.gradle where we find a few interesting items.

1
2
3
4
5
6
7
8
9
10
test {
dependsOn 'jar', 'testsJar', ':startCluster', ':downloadEnronEmails'
}
....
project(":examples/enron") {
uploadArchives.onlyIf { false }
dependencies {
compile project(':core')
}
}

Inside functions.gradle we will also find:

1
2
3
4
task downloadEnronEmails() << {
extract(dataHome, dataHome,
downloadFile('https://s3.amazonaws.com/mongodb-enron-email/enron_mongo.tar.bz2'))
}

Well it looks like we’ve found out where this came from and it was our very early days of testing gradle. The more import point here is to remember that build tools can do almost anything and to be careful when running them. I’ll leave that discussion for another time.

I was recently emailed about my work with Mallory and decoding the FIX protocol. It could be applied to any protocol of course but I had never given out the python file I used except to one person who emailed me a month or two after the original writing.

Original blog post: http://blog.opensecurityresearch.com/2012/05/mallory-mitm-fix-ssl-decryption.html

Amazingly this was 3 years ago and I’ve got no idea where Mallory is at this stage or if my code will even still work but I tracked down the file I had left on a server and am adding here it here for anyone who might want it.

Mallory base.py file

RabbitMQ has become a staple for building job queues between the myriad of spring boot micro-serivces I’ve built at SRC:CLR. The Spring abstraction has allowed for quick and mostly painless development. What I hadn’t found a need for was RabbitMQ’s “Dead Letter Exchange” setup. Multiple times there had been discussions about using the dead letter pattern but I’d never gone that route. During one of my latest development sessions I decided to use the dead letter pattern and felt a quick example setting one up would be useful to share. I’ll also cover some of the new Spring features that allowed me to write less code.

I started off with a blank spring boot project (which I now realize should have been a post prior to this one). I removed the extra items I didn’t need and went about configuring a couple queues and setting some properties. I’ll leave out some of the basic spring boot setup items but for the full code go here.

First a quick properties file:

1
2
3
4
5
6
7
8
# RABBIT
spring.rabbitmq.host=localhost
spring.rabbitmq.username=guest
spring.rabbitmq.password=guest
spring.rabbitmq.virtual-host=/

# Uncomment for full debugging
#logging.level.=DEBUG

The spring.rabbitmq. items are part of spring-boots auto-configuration setup and a full list can be found here. Those properties will load the RabbitAutoConfiguration spring-boot class which gives a set of default beans that you’d otherwise write yourself. More info on that class here.

Next we get into the RabbitMQ config:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
@Autowired
private ConnectionFactory cachingConnectionFactory;

// Setting the annotation listeners to use the jackson2JsonMessageConverter
@Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory() {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(cachingConnectionFactory);
factory.setMessageConverter(jackson2JsonMessageConverter());
return factory;
}

// Standardize on a single objectMapper for all message queue items
@Bean
public Jackson2JsonMessageConverter jackson2JsonMessageConverter() {
return new Jackson2JsonMessageConverter();
}

@Bean
public Queue outgoingQueue() {
Map<String, Object> args = new HashMap<String, Object>();
// The default exchange
args.put("x-dead-letter-exchange", "");
// Route to the incoming queue when the TTL occurs
args.put("x-dead-letter-routing-key", INCOMING_QUEUE);
// TTL 5 seconds
args.put("x-message-ttl", 5000);
return new Queue(OUTGOING_QUEUE, false, false, false, args);
}

@Bean
public RabbitTemplate outgoingSender() {
RabbitTemplate rabbitTemplate = new RabbitTemplate(cachingConnectionFactory);
rabbitTemplate.setQueue(outgoingQueue().getName());
rabbitTemplate.setRoutingKey(outgoingQueue().getName());
rabbitTemplate.setMessageConverter(jackson2JsonMessageConverter());
return rabbitTemplate;
}

@Bean
public Queue incomingQueue() {
return new Queue(INCOMING_QUEUE);
}

The ConnectionFactory came from the auto-configuration and the SimpleRabbitListenerContainerFactory will be used for spring-rabbit’s new annotations later. An important item is deciding on a message converter, I’m a fan of the Jackson2JsonMessageConverter so I added that. You’ll also need to add jackson into your package manager to use this. (Note: As of writing this spring-boot only supports jackson 2.4.x and not 2.5)

The outgoingQueue is where we will set the important arguments to enable the dead-letter setup and the outgoingSender is a basic sender that can be autowired into services for quick sending to a specific queue.

  • x-dead-letter-exchange - The exchange where the message will be republished. (In this case the same exchange)
  • x-dead-letter-routing-key - The routing key used to route the message. Essentially the queue name.
  • x-message-ttl - Defines how long the message will stay on the queue before its Time-To-Live expires and it’s placed on the incoming queue.

The incomingQueue will receive the dead-lettered messages.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
@Component
public class DeadLetterSendReceive {

private static final Logger LOGGER = LoggerFactory.getLogger(DeadLetterSendReceive.class);

@Autowired
private RabbitTemplate outgoingSender;

// Scheduled task to send an object every 5 seconds
@Scheduled(fixedDelay = 5000)
public void sender() {
ExampleObject ex = new ExampleObject();
LOGGER.info("Sending example object at " + ex.getDate());
outgoingSender.convertAndSend(ex);
}

// Annotation to listen for an ExampleObject
@RabbitListener(queues = MQConfig.INCOMING_QUEUE)
public void handleMessage(ExampleObject exampleObject) {
LOGGER.info("Received incoming object at " + exampleObject.getDate());
}

}

Using a scheduled task I send an ExampleObject message every 5 seconds with the autowired outgoingSender. The handleMessage method will receive the messages as their TTL expires and they are placed on the incomingQueue.

The @RabbitListener annotation is new in spring-rabbit as of version 1.4. Previously I’d need to setup around 4 beans to receive messages, with this new annotation it’s reduced to declaring the queue and setting a global factory to use the proper message convertor. Once those are set you’re free to add the annotation in your project.

Running this code you will see log messages like the ones below (colored logs from spring-boot are awesome!):

1
2
3
4
INFO 35325 --- [pool-4-thread-1] v.c.deadletter.mq.DeadLetterSendReceive  : Sending example object at Fri Apr 24 08:35:41 PDT 2015
INFO 35325 --- [cTaskExecutor-1] v.c.deadletter.mq.DeadLetterSendReceive : Received incoming object at Fri Apr 24 08:35:36 PDT 2015
INFO 35325 --- [pool-4-thread-1] v.c.deadletter.mq.DeadLetterSendReceive : Sending example object at Fri Apr 24 08:35:46 PDT 2015
INFO 35325 --- [cTaskExecutor-1] v.c.deadletter.mq.DeadLetterSendReceive : Received incoming object at Fri Apr 24 08:35:41 PDT 2015

Full code can be found here: https://github.com/cl4r1ty/spring-rabbitmq-dead-letter

You’ll need RabbitMQ installed and running. The easiest way to do that is brew install rabbitmq then rabbitmq-server.

Once RabbitMQ is up from the root of the repo quickly run the example with the spring-boot maven plugin command mvn spring-boot:run.

Now you’ve got a quick setup for a dead-letter pattern using spring-rabbit which is much less code than RabbitMQ’s Java implementation.