Sunday, May 4, 2014

DNG 1.4 Parser

This tutorial describes how to build the Adobe DNG SDK on Linux.
It generates the dng_validate C++ program that can parse any DNG images, a bit like a "Hello world" for DNG image processing.


  1. Adobe DNG SDK 1.4


Adobe DNG SDK 1.4


The goal of this blog entry is to execute the dng_validate command built from the source.
We show how to build the XMP libraries then suggest a Makefile configuration to build a DNG SDK sample app.
All the patches were generated by the diff command.

XMP SDK


In the current directory, download the XMP SDK:
wget http://download.macromedia.com/pub/developer/xmp/sdk/XMP-Toolkit-SDK-CC-201306.zip
unzip XMP-Toolkit-SDK-CC-201306.zip
mv XMP-Toolkit-SDK-CC201306 xmp_sdk

Make sure you have at least the following packages installed:

sudo apt-get install cmake libjpeg8-dev uuid-dev

The 2 next snippets download zlib and expat sources. Instructions are explained in the ReadMe.txt files in the third-party directories.

zlib


cd xmp_sdk/third-party/zlib
wget http://zlib.net/zlib-1.2.8.tar.gz
tar xzf zlib-1.2.8.tar.gz
cp zlib-1.2.8/*.h zlib-1.2.8/*.c .

expat


cd xmp_sdk/third-party/expat
Download here the source at http://sourceforge.net/projects/expat/files/expat/2.1.0/expat-2.1.0.tar.gz/download.
tar xzf expat-2.1.0.tar.gz
cp -R expat-2.1.0/lib .

For cross-platform compatibility, to be able to support both Mac & Windows, cmake is the tool selected by the SDK authors to build the source. We go through all the build errors.

cd xmp_sdk/build

Build error 1


make
...
CMake Error at shared/SharedConfig_Common.cmake:38 (if):
  if given arguments:

    "LESS" "413"

  Unknown arguments specified
...

I use gcc 4.8.2 for the build. There seems to be a XMP_VERSIONING_GCC_VERSION variable not properly set. We just delete the version check by removing line 37 till 42 in xmp_sdk/build/shared/SharedConfig_Common.cmake:

xmp_sdk/build/shared/SharedConfig_Common.cmake
36a37,41
>               # workaround for visibility problem and gcc 4.1.x
>               if(${${COMPONENT}_VERSIONING_GCC_VERSION} LESS 413)
>                       # only remove inline hidden...
>                       string(REGEX REPLACE "-fvisibility-inlines-hidden" "" CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS})
>               endif()

Build error 2


make
...
Linking CXX shared library /home/alexis/linux/dng2/xmp_sdk/public/libraries/i80386linux_x64/release/libXMPCore.so
g++: error: /usr/lib64/gcc/x86_64-redhat-linux/4.4.4//libssp.a: No such file or directory
...

The README.txt mentions a XMP_ENABLE_SECURE_SETTINGS:

xmp_sdk/build/README.txt
... He may also want to change the parameter XMP_ENABLE_SECURE_SETTINGS as per the configured gcc.
 a) If the gcc is configured with --enable-libssp (can be checked by executing gcc -v), he has to set the variable XMP_GCC_LIBPATH inside of file /build/XMP_Linux.cmake to the path containing the static lib( libssp.a).In this case he can set the variable the XMP_ENABLE_SECURE_SETTINGS on.
b) If the gcc is configured with --disable-libssp, he has to set the variable XMP_ENABLE_SECURE_SETTINGS off.

I don't have any --enable-libssp in my gcc version:

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.8/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 4.8.2-21' --with-bugurl=file:///usr/share/doc/gcc-4.8/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.8 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.8 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-libmudflap --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.8-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --with-arch-32=i586 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.8.2 (Debian 4.8.2-21)

so I disable it in xmp_sdk/build/shared/ToolchainGCC.cmake line 37:

xmp_sdk/build/shared/ToolchainGCC.cmake
37c37
< set(XMP_ENABLE_SECURE_SETTINGS "OFF")
---
> set(XMP_ENABLE_SECURE_SETTINGS "ON")

Build error 3


/home/alexis/linux/dng2/xmp_sdk/XMPFiles/build/../../XMPFiles/source/NativeMetadataSupport/ValueObject.h:111:60: error: 'memcmp' was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
    doSet = ( memcmp( mArray, buffer, numElements*sizeof(T) ) != 0 );
                                                            ^
In file included from /home/alexis/linux/dng2/xmp_sdk/XMPFiles/build/../../XMPFiles/source/FormatSupport/TIFF_Support.hpp:17:0,
                 from /home/alexis/linux/dng2/xmp_sdk/XMPFiles/build/../../XMPFiles/source/FormatSupport/ReconcileLegacy.hpp:15,
                 from /home/alexis/linux/dng2/xmp_sdk/XMPFiles/build/../../XMPFiles/source/FormatSupport/Reconcile_Impl.hpp:15,
                 from /home/alexis/linux/dng2/xmp_sdk/XMPFiles/source/FormatSupport/WAVE/WAVEReconcile.cpp:24:
/usr/include/string.h:65:12: note: 'int memcmp(const void*, const void*, size_t)' declared here, later in the translation unit
 extern int memcmp (const void *__s1, const void *__s2, size_t __n)

This is an error relative to my new version of gcc, described in the Name lookup changes section.
We simply add the #include "string.h" in xmp_sdk/XMPFiles/source/NativeMetadataSupport/ValueObject.h, line 16:

xmp_sdk/XMPFiles/source/NativeMetadataSupport/ValueObject.h
16,17d15
< #include "string.h"

Validate the build


make
[100%] Built target XMPFilesStatic

Check that the shared and static libraries got properly generated:

$ ls xmp_sdk/public/libraries/i80386linux_x64/release
libXMPCore.so*  libXMPFiles.so*  staticXMPCore.ar  staticXMPFiles.ar

DNG SDK


In the current directory, where the XMP SDK got downloaded, download the DNG SDK:

wget http://download.adobe.com/pub/adobe/dng/dng_sdk_1_4.zip
unzip dng_sdk_1_4.zip

There are multiple ways to come up with the binary. In this case,
  • We seperate the compilation and linking steps in 2 separate targets.
  • We link with the shared libraries of the XMP Toolkit.
  • We did not install system wide in /usr/local/lib and /usr/local/include the required XMP files that were generated during the XMP build.

cd dng_sdk/source

Makefile


Create the "Makefile" build configuration file:

# Binary name
EXECUTABLE=dng_validate

# A DNG image
DNG_IMAGE=~/job/image_samples/9436b5a2336f0a575a5b0ef3adf0b25171125081.dng

# The XMP SDK build directory if we don't want to install it system-wide.
XMP_PUB_DIR=/home/alexis/linux/dng/xmp_sdk/public

INCL=-I $(XMP_PUB_DIR)/include
XMP_RELEASE=$(XMP_PUB_DIR)/libraries/i80386linux_x64/release
LIB=-ljpeg -lz -lpthread -ldl -L $(XMP_RELEASE) -lXMPCore -lXMPFiles
SOURCES:=$(shell ls *.cpp)
OBJECTS=$(SOURCES:.cpp=.o)

# Execute the binary
all: $(EXECUTABLE) $(SOURCES)
        LD_LIBRARY_PATH=$(XMP_RELEASE) ./$< -v $(DNG_IMAGE)

# Linking
$(EXECUTABLE): $(OBJECTS)
        g++ $^ $(LIB) -o $@

# Compilation
.cpp.o:
        g++ -c -Wall -g $(INCL) $^

clean:
        rm $(EXECUTABLE) *.o

Build error 1


To avoid errors like

dng_flags.h:36:28: fatal error: RawEnvironment.h: No such file or directory
 #include "RawEnvironment.h"

Create the RawEnvironment.h file containing build settings:

#define qLinux 1
#define qDNGThreadSafe 1
#define UNIX_ENV 1

Build error 2


To avoid errors like

dng_flags.h:40:2: error: #error Unable to figure out platform

A qLinux setting got created in RawEnvironment.h. The "!defined(qMacOS) || !defined(qWinOS)" kind of test line 39 in dng_flags.h does not seem effective on Linux platform even when qMacOS or qWinOS are defined. We replace it by something more standard, for example "#ifndef qLinux":

39 #ifndef qLinux
40 #error Unable to figure out platform
41 #endif


chmod +w dng_flags.h
39c39
< #ifndef qLinux
---
> #if !defined(qMacOS) || !defined(qWinOS)

Build error 3


dng_string.cpp:1792:24: error: 'isdigit' was not declared in this scope
    if (isdigit ((int) c) || c == '.' || c == '-' || c == '+' || c == 'e' || c == 'E')

We simply need to include "ctype.h" in dng_string.cpp for linux platform at line 33:

32 #if qiPhone || qAndroid || qLinux
33 #include <ctype.h> // for isdigit
34 #endif

At line 32 we add qLinux is the list of platforms on when to include ctype.h.

chmod +x dng_string.cpp
32c32
< #if qiPhone || qAndroid || qLinux
---
> #if qiPhone || qAndroid

Build error 4


/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'

The main function declared in dng_validate.cpp was not found. Enable the command build setting in line 200 in dng_flags.h:

200c200
< #define qDNGValidateTarget 1
---
> #define qDNGValidateTarget 0

Build error 5


make clean
make

We now have a linking error

dng_xmp_sdk.cpp:(.text._ZN9TXMPFilesISsE8OpenFileEP6XMP_IOjj[_ZN9TXMPFilesISsE8OpenFileEP6XMP_IOjj]+0x40): undefined reference to `WXMPFiles_OpenFile_2'
collect2: error: ld returned 1 exit status

We want to link with the shared libraries. Delete the static build setting in dng_xmp_sdk.cpp line 48:

chmod +w dng_xmp_sdk.cpp
48a49,50
> #define XMP_StaticBuild 1
> 

Validate the build


The execution of the command looks like:

make
...
LD_LIBRARY_PATH=/home/alexis/linux/dng/xmp_sdk/public/libraries/i80386linux_x64/release ./dng_validate ~/job/image_samples/9436b5a2336f0a575a5b0ef3adf0b25171125081.dng
Validating "/home/alexis/job/image_samples/9436b5a2336f0a575a5b0ef3adf0b25171125081.dng"...
Raw image read time: 0.074 sec
Linearization time: 0.051 sec
Interpolate time: 0.000 sec
Validation complete

15 comments:

  1. unfortunately, I fail compiling it so far...xmp_sdk does not eat -ldl for some reason

    http://pastebin.com/Lz49d34g

    tried

    make CFLAGS="${CFLAGS} -ldl"
    CC=gcc-4.4
    LIBS="$LIBS -lm -ldl"
    LDLIBS=-ldl make
    SET (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -ldl") in CMakeLists.txt

    no avail..

    any hints?

    ReplyDelete
  2. found it, you have to add ${CMAKE_DL_LIBS} as last item (after ${LIB_ADOBEXMP} ) in line 319 to the function target_link_libraries() in XMP-SDK/XMPFiles/build/CMakeLists.txt

    ReplyDelete
  3. Thank you, thank you, thank you, for your time and for sharing a step-by-step. Now I need no more boring reboots to develop (in Linux) and validate (in Windows). As a side note, shalamander's tip saved the day, a sunny sunday. Thank all of you guys!

    ReplyDelete
  4. The tutorial was very helpful, but I ran into a few extra snags. I was building for Ubuntu 14.04 64-bit, sourcing dng_sdk_1_3 and XMP-Toolkit-SDK-CC2014-12. I've included the diff's of the directories that I finally got to build.

    Unified diff of the XMP_SDK directory (command: diff -r XMP-SDK XMP-Toolkit-SDK-CC201412):
    http://pastebin.com/6qb8STay

    Unified diff of the dng_sdk directory (command: diff -r dng_sdk dng_sdk_original):
    http://pastebin.com/nW7n9fmS

    ReplyDelete

  5. That is very interesting; you are a very skilled blogger. I have shared your website in my social networks..!

    Digital Marketing Services in Chennai

    Digital Marketing Company in Chennai

    ReplyDelete
  6. keep sharing your information regularly for my future reference. This content creates a new hope and inspiration with in me

    Content marketing services in chennai


    ReplyDelete
  7. This idea is mind blowing. I think everyone should know such information like you have described on this post. Thank you for sharing this explanation.Your final conclusion was good. We are sowing seeds and need to be patiently wait till it blossoms.
    Software Testing Training

    ReplyDelete
  8. wow amazing post.The key points you mentioned here related to maintenance of car is really awesome.Checking all fluid levels,changing oil and of course the regular service of the car which is necessary to maintain our vehicle.Thank you for the information.

    bike spa services in mumbai
    house cleaning services in mumbai
    car wash services in mumbai

    ReplyDelete

  9. yes good description about xmp it will be remember to me,but i want more information from this topic.
    Back to original

    ReplyDelete
  10. Superb. I really enjoyed very much with this article here. Really it is an amazing article I had ever read. I hope it will help a lot for all. Thank you so much for this amazing posts and please keep update like this excellent article.thank you for sharing such a great blog with us. expecting for your.
    Digital Marketing Company in India

    ReplyDelete
  11. This was so useful and informative. The article helped me to learn something new. Android App Development Company in India

    ReplyDelete
  12. Very nice post here thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.

    Salesforce Training in Chennai

    ReplyDelete
  13. I just see the post i am so happy the post of information's.So I have really enjoyed and reading your blogs for these posts.Any way I’ll be subscribing to your feed and I hope you post again soon.
    Skilled Manpower Services in Chennai

    ReplyDelete
  14. It's interesting that many of the bloggers to helped clarify a few things for me as well as giving.Most of ideas can be nice content.The people to give them a good shake to get your point and across the command.

    Digital Marketing Company in Chennai

    ReplyDelete