Using Handbrake to Encode DVDs for the Roku

In a previous post, I noted that I use ffmpeg to convert DVDs to MP4s which will play on my Roku.  I recently encountered two problems with this approach.  First, the resulting MP4s only include a stereo audio track, so are highly sub-optimal for surround sound systems.  Second, for a yet-undetermined reason, Plex (my current preferred media client) will transcode the videos rather than play them directly.

To solve both problems, I have changed from ffmpeg to Handbrake and I am following the encoding guidance of Rokoding.  Specifically, since most of my source material is DVDs, I use the 480p Original settings.

Just in case the Rokoding site disappears, I have reproduced the 480p DVD Settings below in a slightly modified form.  These instructions apply to Handbrake 0.9.9.

Use mediainfo to determine the aspect ratio and framerate of the source VOB.  Then, in Handbrake, start with the High Profile preset and then make the following changes:

  1. In Output Settings:
    1. Set Container = Mp4
    2. Ensure Web Optimized is checked
    3. Ensure Large File Size and iPod 5G Support are unchecked
  2. In Picture:
    1. If the movie is widescreen or hard letterboxed, set Anamorphic = Strict.  If the movie is 4:3, set Anamorphic = None and ensure Keep Aspect Ratio is checked.
    2. Set Cropping = Automatic.
  3. In Filters:
    1. Set Detelecine = Off
    2. Set Decomb = Fast
    3. Set Deinterlace = Off
    4. Set Denoise = Off
    5. Set Deblock = Off
  4. In Video:
    1. Set Video Codec = H.264 (x264)
    2. Set Framerate (FPS) to match the source material (usually 23.976 or 29.97).
    3. Set Constant Framerate
    4. Set Quality = Constant Quality of RF:19
    5. Set Optimise Video = Use Advanced Tab instead
  5. In Audio:
    1. Create two audio tracks.  The first audio track will have:
      1. Source = AC3 5.1 (which is what you should have ripped from your DVD)
      2. Codec = AAC (faac)
      3. Bitrate = 160
      4. Samplerate = Auto
      5. Mixdown = Stereo
      6. DRC = 1.5
      7. Gain = 0
    2. The second audio track will have:
      1. Source = AC3 5.1
      2. Codec = AC3 Passthru
  6. In Subtitles, keep everything empty.  I do not handle subtitles.
  7. In Chapters:
    1. Ensure Create chapter markers is checked
  8. In Advanced:
    1. Set Adaptive B-Frames = Optimal
    2. Set Adaptive Direct Mode = Automatic
    3. In x264 Encoder Options, set it to b-adapt=2:rc-lookahead=50:direct=auto
    4. The rest are as default:
      1. Set Reference Frames = 3 (Default)
      2. Set Maximum B-Frames = 3 (Default)
      3. Ensure CABAC is checked
      4. Ensure 8×8 Transform is checked
      5. Ensure Weighted P-Frames is checked
      6. Set Pyramidal B-Frames = Normal (Default)
      7. Set Motion Est Method = Hexagon (Default)
      8. Set Subpixel Motion Est = 7: RD in all frames
      9. Set Motion Est Range to 16
      10. Set Partition Type = Most (Default)
      11. Set Trellis = Encode Only (Default)
      12. Set Adaptive Quant Strength = Middle
      13. Set Rate Distortion = Middle
      14. Set Psychovisual Trellis = Leftmost Setting
      15. Ensure No DCT-Decimate is unchecked
      16. Set Deblocking = 0 (Default), 0 (Default)

Instrumenting a Java Web Application with JMX

Java Management Extensions (JMX) is a technology for managing and monitoring Java applications. JMX instructs application developers how to instrument their applications to expose metrics and management mechanisms, and instructs operations teams how to collect metrics and manage Java applications. JMX is ubiquitous: there are a large number of off-the-shelf commercial and open source tools which speak JMX.

This article is intended for application developers to introduce JMX, to teach them how to create some simple metrics via JMX, and to encourage them to thoroughly instrument their applications with JMX.; This article uses only J2SE 5 and the Java Servlet specification. There may be better, more efficient, or more appropriate ways to add JMX counters to applications using other frameworks. For example, Spring developers should read http://static.springsource.org/spring/docs/2.0.x/reference/jmx.html.

Why Instrument Your Application Using JMX?

High-quality monitoring extends far beyond OS-level metrics such as CPU, memory, and disk space. Every non-trivial application has a collection of metrics which are critical to understanding how the application is behaving in production. There is no way for any off-the-shelf tool to collect these metrics, as they are internal to the application. Some examples of these metrics are:

  • Cache hits and misses for an internal, in-memory cache
  • Number of currently logged-in users
  • Authentication successes and failures

Standardizing the way applications perform instrumentation allows us to easily integrate tools such as Graphite to enable rapid data aggregation, charting, dashboarding, and eventing. For example, the below custom dashboard was built in Graphite in less than an hour:

    image

JMX for Application Instrumentation

JMX is a complicated technology which covers far more than just application instrumentation. However, this tutorial focuses on the basics of what an application developer needs to know to implement instrumentation for his application. If you would like to know more, please read the Java JMX tutorial.

To instrument an application, the developer must do the following:

  1. Write one or more Managed Beans, or MBeans, which are Java objects which contain the metrics that the application exposes
  2. Register the MBeans with the platform MBean server

A Standard MBean is defined by writing a Java interface whose name ends with MBean and a Java class which implements this interface. For example:

public interface HelloMBean
{ 
    public int getCacheSize(); 
}

public class Hello implements HelloMBean
{ 
    public int getCacheSize() { /* return cache size */ } 
}

For the MBean to be visible to management applications, it must be registered with the platform MBean server. This is typically done with code that looks like:

MBeanServer mbs = ManagementFactory.getPlatformMBeanServer(); 
ObjectName name = new ObjectName("..."); 
Hello mbean = new Hello(); 
mbs.registerMBean(mbean, name);

MBeans may also generate notifications to signal state changes, detected events, or problems. This can be used to move from a polling to a push-based update mechanism.  This is done by having the MBean class inherit from NotificationBroadcasterSupport and calling notify().

JmxCalcService – A Simple Web Application Instrumented with JMX

We will start with a simple web service called JmxCalcService which supports two URLs, add and subtract.  These URLs are implemented by two separate servlets, com.morningstar.jmxcalcservice.servlet.AddServlet and com.morningstar.jmxcalcservice.servlet.SubtractServlet.

Here is the implementation of AddServlet (SubtractServlet is virtually identical):

package com.morningstar.jmxcalcservice.servlet;

import java.io.IOException;
import java.io.PrintWriter;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

public class AddServlet extends HttpServlet
{
    public void doGet(HttpServletRequest request,
                      HttpServletResponse response)
        throws ServletException, IOException {

        int val1 = Integer.parseInt(request.getParameter("val1"));
        int val2 = Integer.parseInt(request.getParameter("val2"));
        int result = val1+val2;

        PrintWriter out = response.getWriter();
        out.println(result);
        out.flush();
        out.close();
    }
}

These servlets are bound to URLs using the WEB-INF/web.xml file:

<!DOCTYPE web-app PUBLIC
 "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN"
 "http://java.sun.com/dtd/web-app_2_3.dtd" >

<web-app>
  <display-name>jmxcalcservice - A simple calculation web service
    instrumented via JMX</display-name>

  <servlet>
    <servlet-name>add</servlet-name>
    <servlet-class>com.morningstar.jmxcalcservice.servlet.AddServlet</servlet-class>
  </servlet>
  <servlet>
    <servlet-name>subtract
    <servlet-class>com.morningstar.jmxcalcservice.servlet.SubtractServlet</servlet-class>
  </servlet>
  <servlet-mapping>
    <servlet-name>add</servlet-name>
    <url-pattern>/add</url-pattern>
  </servlet-mapping>
  <servlet-mapping>
    <servlet-name>subtract</servlet-name>
    <url-pattern>/subtract</url-pattern>
  </servlet-mapping>
</web-app>

JmxCalcService may be tested as follows:

$ mvn jetty:run 
$ curl 'http://localhost:8080/add?val1=5&val2=8'
$ curl 'http://localhost:8080/subtract?val1=9&val2=4'

View JMX Instrumentation Using JConsole

The JVM is already instrumented with JMX, and the Servlet container often is (although typically the instrumentation on the latter must be explicitly enabled). We can view this instrumentation using JConsole. Let’s start up the web service and then view its instrumentation. Be sure to start the web service first so that JConsole can find the process:

$ mvn jetty:run 
$ jconsole

Find the Jetty process and click “Connect”:

image

JConsole opens up a window which is updated in real-time with the state of the application:

image

You can view individual MBeans by clicking on the MBeans tab.  For example, we can view the value of the java.lang.ClassLoading.LoadedClassCount attribute as follows:

image

Adding Metrics to JmxCalcService

We will add two metrics to JmxCalcService: the number of times that add is called, and the number of times that subtract is called. These metrics will be implemented in a MBean whose interface is called com.morningstar.jmxcalcservice.mbean.JmxCalcServiceMBean:

package com.morningstar.jmxcalcservice.mbean;

public interface JmxCalcServiceMBean
{
    public int getNumAdds();
    public int getNumSubtracts();
}

The implementation of the MBean is in the class com.morningstar.jmxcalcservice.mbean.JmxCalcService This class also includes two additional functions, incrementNumAdds() and incrementNumSubtracts() These functions are called by the servlets each time their methods are called. These functions are not exposed through the MBean interface and are not available to JMX management applications.

JmxCalcService is implemented as a Singleton with thread-safe counters, as we only want one instance of the data and increments can happen on any thread. It's implementation as below:

package com.morningstar.jmxcalcservice.mbean;

import java.util.concurrent.atomic.AtomicInteger;

public class JmxCalcService
    implements JmxCalcServiceMBean
{
    private static JmxCalcService s_instance  = new JmxCalcService();

    public static JmxCalcService getInstance()
    {
        return s_instance;
    }

    private AtomicInteger numAdds = new AtomicInteger(0);
    private AtomicInteger numSubtracts = new AtomicInteger(0);
    private JmxCalcService() { }

    public int getNumAdds() { return this.numAdds.get(); }
    public int getNumSubtracts() { return this.numSubtracts.get(); }
    public int incrementNumAdds() { return
        this.numAdds.incrementAndGet(); }
    public int incrementNumSubtracts() { return
        this.numSubtracts.incrementAndGet(); }
}

The JmxCalcService MBean must be registered with the platform MBean server. The proper way to register MBeans varies based on application type or servlet container. For Jetty, the following technique works well:

First, create a ServletContextListener which registers the MBean at application startup time and deregisters it at application shutdown time:

package com.morningstar.jmxcalcservice;

import java.lang.management.ManagementFactory;
import javax.management.MBeanServer;
import javax.management.ObjectName;
import javax.servlet.ServletContext;
import javax.servlet.ServletContextEvent;
import javax.servlet.ServletContextListener;
import com.morningstar.jmxcalcservice.mbean.JmxCalcService;

public class JmxCalcServiceContextListener
    implements ServletContextListener
{
    private static final String mbeanName =
        "com.morningstar.jmxcalcservice:type=JmxCalcService";

    public void contextInitialized(ServletContextEvent event) {
        try {
            MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
            ObjectName objName = new ObjectName(this.mbeanName);
            JmxCalcService mbean = JmxCalcService.getInstance();
            mbs.registerMBean(mbean, objName);
        } catch (Exception ex) {
            System.err.println("Error registering JmxCalcService MBean: " +
                ex.toString());
        }
    }

    public void contextDestroyed(ServletContextEvent event) {
        try {
            MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
            ObjectName objName = new ObjectName(this.mbeanName);
            mbs.unregisterMBean(objName);
        } catch (Exception ex) {
            // Ignore unregistration failures
        }
    }
}

Second, add JmxCalcServiceContextListener to the web.xml file:

...
<web-app>
  <display-name>jmxcalcservice - A simple calculation web service
    instrumented via JMX</display-name>

  <listener>
    <listener-class>com.morningstar.jmxcalcservice.JmxCalcServiceContextListener</listener-class>
  </listener>

  <servlet>
    <servlet-name>add</servlet-name>
    <servlet-class>com.morningstar.jmxcalcservice.servlet.AddServlet</servlet-class>

We must also change AddServlet and SubtractServlet to call incrementNumAdds() or incrementNumSubtracts() at the end of the function:

public class AddServlet extends HttpServlet {
    public void doGet(HttpServletRequest request,
                      HttpServletResponse response)
        throws ServletException, IOException {

        int val1 = Integer.parseInt(request.getParameter("val1"));        
        int val2 = Integer.parseInt(request.getParameter("val2"));
        int result = val1+val2;

        PrintWriter out = response.getWriter();
        out.println(result);
        out.flush();
        out.close();

        JmxCalcService.getInstance().incrementNumAdds();
    }
}

After all this, we can start the application, hit the URLs a few times, load JConsole and see the values:

image

What’s Next?

After you instrument your application with JMX, consider hooking up your application to your centralized monitoring system or using jmxtrans to collect and send metrics to Graphite so that you can chart and trend data over time.  Of course, if Graphite is your ultimate goal, statsd might be a better choice.

Clean Code is Not Threadsafe Code?

I am currently reading Clean Code: A Handbook of Agile Software Craftsmanship by Robert C. Martin and I’ve noticed a pattern.

In Listing 2-1, Martin recommends refactoring this code:

private void printGuessStatistics(char candidate, int count)
{
    String number;
    String verb;
    String pluralModifier;
    if (count == 0) {
        number = "no";
        verb = "are";
        pluralModifier = "s";
    } else if (count == 1) {
        number = "1";
        verb = "is";
        pluralModifier = "";
    } else {
        number = Integer.toString(count);
        verb = "are";
        pluralModifier = "s";
    }
    String guessMessage = String.format(
        "There %s %s %s%s", verb, number, candidate, pluralModifier
    );
    print(guessMessage);  
}

into this:

public class GuessStatisticsMessage {
    private String number;
    private String verb;
    private String pluralModifier;

    public String make(char candidate, int count) {
        createPluralDependentMessageParts(count);
        return String.format(
            "There %s %s %s%s",
            verb, number, candidate, pluralModifier);
    }

    private void createPluralDependentMessageParts(int count) {
        if (count == 0) {
            thereAreNoLetters();
        } else if (count == 1) {
            thereIsOneLetter();
        } else {
            thereAreManyLetterscount();
        }
    }

    private void thereAreManyLetters(int count) {
        number = Integer.toString(count);
        verb = "are";
        pluralModifier = "s";
    }

    private void thereIsOneLetter() {
        number = "1";
        verb = "is";
        pluralModifier = "";
    }

    private void thereAreNoLetters() {
        number = "no";
        verb = "are";
        pluralModifier = "s";
    }
}

In listing 4-7, Martin recommends refactoring this code (comments elided):

public class GeneratePrimes
{
    public static int[] generatePrimes(int maxValue)
    {
        if (maxValue >= 2) {
            int s = maxValue + 1;
            boolean f = new boolean[s];
            int i;

            for (i = 0; i < s; i++)
                f[i] = true;

            f[0] = f[1] = false;

            int j;
            for (i = 2; i < Math.sqrt(s) + 1; i++) {
                if (f[i]) {
                    for (j = 2 * i; j < s; j += i)
                        f[j] = false;
                }
            }

            int count = 0;
            for (i = 0; i < s; i++) {
                if (f[i]) {
                    count++;
                }
            }

            int[] primes = new int[count];

            for (i = 0, j = 0; i < s; i++) {
                if (f[i])
                    primes[j++] = i;
            }

            return primes;
        } else {
            return new int[0];
        }
    }
}

into this:

public class PrimeGenerator
{
    public static boolean[] crossedOut;
    public static int[] result;

    public static int[] generatePrimes(int maxValue)
    {
        if (maxValue < 2) {
            return new int[0];
        } else {
            uncrossIntegersUpTo(maxValue);
            crossOutMultiples();
            putUncrossedIntegersIntoResult();
            return result;
        }
    }

    private static void uncrossIntegersUpTo(int maxValue)
    {
        crossedOut = new boolean[maxValue + 1];
        for (int i = 2; i < crossedOut.length; i++)
            crossedOut[i] = false;
    }

    private static void crossOutMultiples()
    {
        int limit = determineIterationLimit();
        for (int i = 2; i <= limit; i++) {
            if (notCrossed(i))
                crossOutMultiplesOf(i);
        }
    }

    private static int determineIterationLimit()
    {
        double iterationLimit = Math.sqrt(crossedOut.length);
        return (int)iterationLimit;
    }

    private static void crossOutMultiplesOf(int i)
    {
        for (int multiple = 2*i; multiple < crossedOut.length; multiple += i)
            crossedOut[multiple] = true;
    }

    private static boolean notCrossed(int i)
    {
        return crossedOut[i] == false;
    }

    private static void putUncrossedIntegersIntoResult()
    {
        result = new int[numberOfUncrossedIntegers()];
        for (int j = 0, i = 2; i < crossedOut.length; i++) {
            if (notCrossed(i))
                result[j++] = i;
        }
    }

    private static int numberOfUncrossedIntegers()
    {
        int count = 0;
        for (int i = 2; i < crossedOut.length; i++)
            if (notCrossed(i))
                count++;

        return count;
    }
}

In both cases, Martin has transformed thread-safe, nearly pure functions into thread-unsafe functions. Even worse, while the former example can be used on multiple threads by having each thread instantiate a local instance of GuessStatisticsMessage, the latter example, due to its use of static member variables, cannot be used on multiple threads without some form of locking.

While I agree that Martin’s examples are more readable, it seems to me that there may be a fundamental tension between object-oriented programming and thread safety, and this tension does not exist in functional programming.

Ffmpeg encoding for the Roku

Update 2013-12-02: I no longer use this method. See this post for my new method.

Roku2_XD_Family_610x465I have three (someday to be four) Roku players at home.  I primarily use them to stream Netflix and Amazon Instant Video and to stream MP4s from a home server using Roksbox.

My typical workflow is to rip DVDs to VOBs using Dvdfab, copy the VOBs to my home server, and then convert them in bulk from VOB to MP4 using ffmpeg.

The Roku does not support all profiles of H.264, so here is the ffmpeg 1.0.6 command I use to create Roku-compatible MP4s:

ffmpeg -i src.vob -vcodec libx264 -threads 0 -f mp4 -y -acodec libfaac -ab 128k -ac 2 -preset slow -profile:v high -level 4 -crf 20 dest.mp4

Creating a Virtual Java RPM

javaSome RPMs (e.g. jpackage.org’s tomcat7-7.0.39-1.jpp6.noarch.rpm) express their dependency upon Java by requiring a RPM that provides capability java (as opposed to, for example, depending on the existence of a file /usr/bin/java). On CentOS, this capability is normally provided by the java-*-openjdk RPM. Therefore, if you execute # yum install tomcat7 on a clean install of CentOS, yum will install OpenJDK in addition to Tomcat 7.

Some people prefer to run the Oracle JRE/JDK instead of OpenJDK. Oracle provides RPMs named jre-version-linux-x64.rpm and jdk-version-linux-x64.rpm to make installing them easier. Unfortunately, these RPMs do not provide the capability java. This means that if you already have the Oracle JRE installed, and you install a RPM which requires the capability java, the OpenJDK will be unnecessarily installed (and might even become the default!).

I solved this dilemma by creating a ‘virtual’ RPM package which provides the capability java by depending on the Oracle JRE. I named this package virtual-java.

Creating this package is quite easy. First I created a Makefile to make building the RPM easier and deal with rpmbuild’s nonsense regarding _topdir:

VERSION=1.7
RELEASE=1
RPMNAME=virtual-java-$(VERSION)-$(RELEASE).noarch.rpm

.PHONY: all
all: dist/$(RPMNAME)

.PHONY: clean
clean:
	rm -rf work
	rm -rf dist

dist/$(RPMNAME): work/RPMS/noarch/$(RPMNAME) dist
	cp work/RPMS/noarch/$(RPMNAME) dist/$(RPMNAME)

work/RPMS/noarch/$(RPMNAME): work/BUILD work/RPMS/noarch work/SPECS/virtual-java.spec
	rpmbuild -bb --define="_topdir ${PWD}/work" work/SPECS/virtual-java.spec

work/SPECS/virtual-java.spec: work/SPECS virtual-java.spec
	cat virtual-java.spec | sed -e s/%VERSION%/$(VERSION)/g | sed -e s/%RELEASE%/$(RELEASE)/g > work/SPECS/virtual-java.spec

dist:
	if [ ! -d dist ]; then mkdir -p dist; fi
	touch dist
work/BUILD:
	if [ ! -d work/BUILD ]; then mkdir -p work/BUILD; fi
	touch work/BUILD
work/RPMS/noarch:
	if [ ! -d work/RPMS/noarch ]; then mkdir -p work/RPMS/noarch; fi
	touch work/RPMS/noarch
work/SPECS:
	if [ ! -d work/SPECS ]; then mkdir -p work/SPECS; fi
	touch work/SPECS

Note how the Makefile defines VERSION and RELEASE, and provides them to the .spec file. This is done because the Makefile needs to know the name of the generated RPM file, which depends on VERSION and RELEASE.

Here is virtual-java.spec:

Name: virtual-java
Version: %VERSION%
Release: %RELEASE%
Group: ENTER GROUP HERE
Summary: Virtual package which provides java but uses the Sun/Oracle JRE
License: None
BuildArch: noarch
Provides: java
Requires: jre > %VERSION%

%description
Virtual package which provides java but uses the Sun/Oracle JRE

%prep

%build

%pre

%post

%install

%files

%changelog

Booting an Acer Aspire easyStore H340 from External USB

440x330-easystoreA couple of years ago I bought a Acer Aspire easyStore H340 to use as a home NAS device.  While the H340 came with Windows Server, I decided to replace it with Debian Linux running on an external USB stick.  Besides using it for basic file storage, I use Nginx to stream MP4s to Roku devices throughout my home using Roksbox; I use forked-daapd to stream FLACs to iTunes (albeit with limited success – foobar2000 pointing to the CIFS file share works better); and I use CrashPlan to back up my important data (e.g. pictures) to the Internet.

Two days ago, S.M.A.R.T. indicated that hard drive which originally came with the machine was failing, so I replaced it with a spare I had on-hand.  I replaced the hard drive, rebuilt the RAID5 array, and everything seemed fine.  However, after upgrading Debian to the latest kernel, the machine failed to boot.

I attached my VGA debugging cable and determined that the machine appeared to be having issues with the boot order.  Using the H340 service manual as a reference, I attached a spare jumper to JP3 and changed the boot order in the BIOS.  I heard that if you removed this jumper the machine would restore to its default boot order, so I left the jumper in and put everything back.

Mission accomplished.