Workflow (3) – quick RAW converting batch

Welcome back!

After covering the general concept of workflow and talking about the RAW vs JPEG dispute, we realize that after a day shooting, we will probably end up with:

  1. a mix of RAW and JPEG files
  2. the need to process these images quickly for review

Now this is not about fine-tuned post-processing of 4 images, this is about “I have 500 files in a directory, of which 150 are RAW and the rest JPEG. I want to browse through these files and check which ones I keep and which ones I throw away”.

My camera does it this way: JPEG files are numbered _igpXXXX.jpg and RAW fils are numbered imgpXXXX.pef. PEF is the Pentax RAW file extension and XXXX is the file number. So I would need a file viewer that allows me to view PEF and JPG files and sort them by time taken (in the EXIF datas). Or, Linux being what it is, I would need a trick.

The trick

The idea here is to quickly convert all the RAW files to JPEG and rename them to fit the JPEG pattern – with _r.jpg at the end so we know we have a RAW version of this file available. Then we can review all our files in a standard image viewer (my favorite is Gthumb):

  • they are all JPEG
  • they all follow the same naming pattern (so order by name also means order by order the pictures were taken)
  • we can identify (_igpXXXX_r.jpg) the files for which we have a RAW version.

In order to achieve this, we need: a copy of ufraw and a batch script.

Ufraw a RAW converter for Linux…

We will talk more in detail about Ufraw (web, wiki) later, but let’s just say that it is a Linux RAW converter that offers

  • one picture conversion / editing user interface
  • a command line interface.

It is the command line interface that we are interested in here. So entering something like:

ufraw-batch mypicture.pef –output mypicture.jpg

on a terminal from the directory where mypicture.pef is would convert my RAW file and spit out mypicture.jpg (white keeping the original file, obviously). You can specify quite a few parameters.

… and a batch file

Now consider this batch file:

#!/bin/bash

if [ ! -d ./processed_images ]; then mkdir ./processed_images; fi;

# processes raw files
for f in *.pef;
do
	echo "Processing $f"
	ufraw-batch \
		--wb=camera \
		--exposure=auto \
		--out-type=jpeg \
		--compression=96 \
		--out-path=./processed_images \
		$f
done

cd ./processed_images

# change the image names
for i in *.jpg;
do
	mv "$i" "${i/.jpg}"_r.jpg;
done
for i in *.jpg;
do
	mv "$i" "${i/imgp/_igp}";
done

What it does is take each *.pef file (my RAW files) and convert them to JPEG, using the in camera white balance, auto exposure and a 96% JPEG compression (roughly equivalent to the size I get from the camera). It saves the image in a subfolder (processed_images) and then changes the image names to fit the JPEG pattern (except for the _r marker at the end).

In order to use this thing, just copy and paste it to a text file. Save the file as raw-processing.sh and make it executable (usually there is a choice in your file manager). Then copy and paste the file to the directory where your images are and execute it – either by clicking on it, or by opening a terminal in the directory and typing ./raw-processing.sh at the command prompt. Obviously, this only works for Pentax PEF files and you need to have a working copy of Ufraw.

So what is that all about?

What we made today is just an equivalent to the in-camera processing from RAW to JPEG. That allows us to consider all our files as standard JPEG (which we can easily browse, review and sort out – ie, throw away…). But we keep our RAW files and their post-processing power at hand, so if we stumble upon a portfolio-grade image (and that we shoot that one in RAW), we can go back to it for fine-tuned post-processing.

And the interest of that is that it requires no post-processing time. Just start your batch script, make yourself a cup of coffee (drink it, depending on how many files you need to process…) and you can start reviewing your files. No attention needed to fine-tune each file (not interesting at that time). Just a quick, easy and standard way to review all your files.

I think Linux for photography not there yet (otherwise, there wouldn’t be a need for this blog🙂 ). However, this forces us to manage and find our way – and all in all, I think tricks like this one are not just workarounds: to me, this perfectly fits the bill for easily dealing with a big number RAW files when back from a day of shooting.

Here is one more picture which I like a lot – it was pitch dark (wedding dancing time) and almost no chance to take good pictures, but then the accordionist / singer just moved a step forward in the light, time to franticly shoot a couple of pictures before she moved again… and guess what? It is based on a JPEG file.

accordionist.jpg

46 Responses to Workflow (3) – quick RAW converting batch

  1. Ethan says:

    I like your blog. It’s really informative. I have been following it. Keep it up!

  2. jcornuz says:

    Hi Ethan,

    Thanks for your comment – I’ll do my best🙂

    Take care,

    Joel

  3. Halgeir says:

    Hi! Thank you for the script. I think Linux Photo tools these days almost have everything. I uses a Pentax K100D camera with Linux and it works great. I basically creates pictures for web pages and home use. Tools like DigiKam, Hugin Panorama Creator, Gimp and Krita (for 16-bit) is good enough. And in addition there is speed and stability for the Linux OS system. My favorite tool is the Aspect Ratio Crop & Composition Guide in DigiKam.
    I have one problem that is raw import of pef files to 16-bits. These gets to dark when I use GUI. Have solved it with a scripting and have used your script as a basic.

  4. jcornuz says:

    Hello Halgeir,

    Thanks for passing by (always good to see a fellow Pentaxian🙂 ). I am in the process of having another go at Krita, so I’ll say a word about that when I have a chance.

    Regarding the dark ouput from ufraw, I have had the same issue. It goes away when I chose sRGB instead of AdobeRGB as profile for the photo – it is not ideal since the image is REALLY in Adobe, but…

    Happy to see you could do something useful from stuff on this blog.

    Take care,

    Joel

  5. sune says:

    Hi,

    regarding the colour space the raw image isn’t in any colour space actually. So if you have set the camera to AdobeRGB that is only for jpg:s produced by the camera.

    So the raw image you have in ufraw dosn’t have a coulor profile. It’s the output from ufraw that is the coulor profile you set it too. So if you output a jpeg it will be in the colour space you have choosen in ufraw.

  6. jcornuz says:

    Hi Sune,

    Actually, Ufraw allows to have an input profile (which I chose to default to my camera’s profile) since this is what seems most logical to me. However, you are right that the AdobeRGB / sRGB is irrelevant to RAW files.

    I would be interested to learn more about it, if you have more info (especially what the “color matrix” tick does…)

    Thanks for your comment and take care,

    Joel

  7. sune says:

    Hi,

    yes it’s true that you can set the input profile in ufraw. It isn’t the profile that you can set in the camera which is for jpg:s but instead a profile specific for that cameras raw files.

    There seems to be some info about it on http://ufraw.sourceforge.net/Colors.html

    There you can find input profiles for Nikon cameras and there is a link of how to do it for Canon.

    I am just trying to figure things out and I am quit new to working with raw files on linux but I realy wan’t to get it to work so I don’t have to use mac or windows for raw processing.

    //sune

  8. Mark Lennox says:

    Thanks for the batch file! Worked a treat getting some files batch converted for a friend of mine and saved me hours of manual conversion

  9. jcornuz says:

    You are welcome. That is what it is here for🙂

  10. Michal says:

    hi

    i’m not sure how it works in pentax world but in canon there is embeded small (1536×1024) jpg in the raw file. Extracting the jpg from raw is very quick and the jpg is good enough to judge which pictures to keep.

    Funny is comparing the jpg with postprocessed raw – just in case you not sure if doing raw is worth the effort😉

    the command is:
    dcraw -e *.cr2

    .cr2 is canon extension for raw files

    however i don’t know if this works for pentax

    ps.
    great web site
    thanks

  11. jcornuz says:

    Hi Michal,

    Thanks for your comment. I think there is a jpg thumbnail in Pentax as well – but I never thought of extracting it for a quick review… Thanks for the tip.

    It is true that in most case, direct JPEG is good enough. My problem is that more often than not the interesting picture is in the “few other” cases. So if I can, I shoot RAW to get the extra tweaking possibilities🙂

    Take care,

    Joel

  12. jandrade says:

    Hi jcornuz,

    Very good blog about digital photography and Linux (Opensue 10.3).
    I’me a nikonian (NikonD50) and I always shot in raw because I’ve the possibility of better post processing.
    To choose which raw files are going to post processing with ufraw I just use Picasa for Linux. This will not only allow you to see the raw files without any pre-conversion but will also allow you to make some processing (crop, saturation, sharpness, light, etc) to see if it is really worth to keep the photo.

    Have nice clicks

  13. jcornuz says:

    Hi Jandrade,

    That’s a good idea for the “quick sorting” step. Aside Picasa, I think Digikam also offers RAW files view. gThumb (my image viewer of choice) only has thumbnails… less useful🙂
    As a RAW processing / batch solution, RawStudio is very promising, but development is not very quick…

    Take care,

    Joel

  14. Mortimer says:

    Hello,

    for quick review of the RAW images, I use:
    ufraw-batch –embedded-image *.CR2
    that extracts the thumbnails out of all my RAWs (CR2 is canon raw file) to filename.embedded.jpg, I then rename them to remove the .embedded so that the thumbnails are easilly “attached” to the original RAW. It’s a lot faster than using the RAW conversion and is sufficient for quick review of a bunch of files.

  15. Benno says:

    Hi,

    I really recommend digikam (www.digikam.org). It’s fine for organising and sorting out pictures (you can even assign tags and rate them) and it’s able to display raw images. So no need to batch convert these in the first place. It supports color management and GREYCstoration denoising.

    Besides, great blog, great photography (I really love the above picture)

  16. Chuck says:

    Hi,

    Just thought I would let you know that I used your batch programme tonight and with a slight mod (my pef ext is caps) it worked a dream. Glad I stumbled across your blog.

    A fellow Pentax user (although through Samsung :))

  17. Jeroen Sijpkens says:

    Hello all,

    I use the same mod with capital PEF, because My Pentax K100D makes this extension by default. Someone here told us the picasa program, which i like a lot, is able to preview raw. That could be, but at least not for Pentax raw. I use the latest version, and it doesn’t even see the files! Nothing with pef is imported, and if imported manually, it skips the file as if they aren’t there at all.
    I prefer Gnome, but i will have a look at Digicam as well, since i’m still interested in a plain (thumb)viewer for PEF.

    Keep the luck in your mind fooks….

  18. Jeroen Sijpkens says:

    By the Way, i discovered this PEF and other raw handling beauty today!
    http://bluemarine.tidalwave.it/
    it’s a beta, but it sure does the job well.\
    This is a real raw suite, and quite fancy too.
    I like it allready, it’s free and nagless.
    Deb version for my Ubuntu makes it a one click install. (~30MB)
    or type: dpkg -i blueMarine-x.x.x_all.deb.
    Goodbye LightRoom.

  19. jcornuz says:

    I Jeroen,

    I have tried Bluemarine a couple of time with mixed results. Maybe it is time to try again with the last version. Good to see that there is a deb ready to go.

    Take care,

    Joel

  20. Try it if you like, it’s improving but it’s not still for production level (not yet for everbody, I mean). Any feedback is appreciated.

  21. Chris says:

    Hi Joel
    Thanks for the script. It has saved me a huge amount of time. I wrote a script in Scheme to coonect UFRaw and GIMP to convert PEF files, but yours is so much better (I wrote my script a few years ago, and now I can’t even understand it!). There was one change that I thought you might like to make. My DS produces PEF files, not pef files and the script didn’t work for me until I changed it. So the following line:

    for f in *.pef;

    should be

    for f in *.PEF;

    I’m glad to come across another Pentaxian who uses Linux. It’s been really lonely trying to figure out a Linux workflow for my Pentax, untul I came across your website!
    Thanks
    Chris

  22. NewMikey says:

    Usage: from Konqueror service menu (select raw files & right-click)

    You will be prompted for whitebalance settings (auto|camera|preset), the ufraw ID file to take the standard settings from, sRGB final conversion
    (yes|no) if you have selected a gamma-1 working profile in UFRaw and thumbnails+framing

    Make sure you have started the GUI version of UFRaw at least once and have generated a ufraw ID file that contains the settings for auto-exposure(or
    not), whitebalance and colorprofiles, as well as curves and adjustments. After starting ufrb-kde you will be prompted for this ID file.

    Best combined with a file in /usr/share/apps/konqueror/servicemenus so ufrb-kde becomes a right-click menu action in konqueror
    See comment section at the bottom of this file for the servicemenu (cut and paste in new document, then save to /usr/share/apps/konqueror/servicemenus

  23. NewMikey says:

    I seem to not be able to post a link, but it is on the DFPE website which you get to by clicking on my name in this post. Top left you will see a link to the forumposting where the actual downloadlink is attached. (“ufrb-kde script and servicemenu for RAW image conversion”)

    Sorry for that people, I don’t want to mess up Joel’s security on this blog.

  24. jcornuz says:

    Hi Mike,

    I rely on wordpress.com for security – so there is not much I can do to change that. Thanks for your perseverance and sharing your work here.

    Take care,

    Joel

  25. Harry Lime says:

    Great post, man. I added a size=800 attribute to only output smallish versions

  26. ptecza says:

    Hello!

    Really great and very helpful post for me! It’s exactly what I have been searching. ufraw-batch rox!😉 Thank you very much!

    Best regards,
    Pawel

  27. ptecza says:

    I also discovered that ufraw-batch can resize (scale) images😀 You only need to set maximum width or height of image and ufraw-batch make a hard job automagically for you. Please believe, but it’s really fast operation.

    I know I can use dcdraw or mogrify or another Linux console utilities for converting RAWs to JPGs. But ufraw-batch is perfect and very easy to usage tool for me and I’m glad like little child that it exists🙂

    So, bye-bye SHQ JPG images on my digital camera!😉 RAW files rules!😉

  28. Alex says:

    Hello,

    Another way to do it is looping the conversion. I usually use Ufraw to develop raw files, but to actually convert them into jpeg I use the convert command with a loop.

    for i in *; do convert $i -quality 100 $(echo $(basename “$i” “.CR2”).jpg); done

    This is a variant of a snippet I found elsewhere, it uses Canon’s raw format and leaves 100% quality.

  29. Bigtuna says:

    Here is another script made by pure, just right click on raw files and convert in ubuntu

    http://github.com/purex01/imgproc_g-script

  30. […] imagemagick convert的简单用法 batch script example […]

  31. mk says:

    Awesome script man. I took 4GB worth of pictures today, don’t have photoshop installed on my work computer (but have my Linux VM🙂, and I wanted to share pics with my family and friends overseas. As a final step you can use imagemagick to downsize before sending the pics off.

    Thanks again for the wonderful, wonderful script.

  32. DopeAddict says:

    Great blog. I hope you’re still monitoring comments. If not, I’ll stick around to read more!

    dcraw -e *.cr2

    I’m having a problem batch processing. I know the “dcraw -e” command, but I can’t remember the exact command to include each ORF rather than doing it one at a time. I’ve tried everything I can remember, but it’s been a while & I’ve forgotten how to do it. If you can help I’ll be grateful, if not I’ll keep reading your blog regardless. Keep up the great work!

  33. Dcdjfnce says:

    Will I have to work shifts? cp pthc
    56878

  34. Rqopeqcn says:

    We’d like to offer you the job swimsuit models
    wcz

  35. Sqjkbcok says:

    One moment, please cp sites
    bzkoen

  36. Eazuoyzw says:

    No, I’m not particularly sporty Hooters Teen Models
    %PPP

  37. canon powershot sx10…

    […]Workflow (3) – quick RAW converting batch « Linux Photography[…]…

  38. […] name has a great tutorial on using this utility through a bash script, my only recommendation is converting to PNG but it is entirely up to personal preference. […]

  39. 手機收購 says:

    手機收購…

    […]Workflow (3) – quick RAW converting batch « Linux Photography[…]…

  40. […] modified the ufraw-batch script here. I named the script as raw2jpg. # […]

  41. Lmao at this list. Yeah interaction is essential, and from talking
    about what we both want I’ve discovered that each one person I’ve ever slept with has appreciated rough passionate sex and half of them beloved
    anal. So carry out some freakin’ homework on what folks truly want. Repetitive bland missionary is vanilla sex leftover from Victorian period romance novels. Welcome towards the twenty first century. Get naked, get wild, get with it.

  42. Let us know when you’re prepared to go and we are going to do what we are able to to support you!

  43. Now that you just have read tthis article, you’ve got a considerably better understanding
    of three typical over the counter snoring cures, every of these
    solving a diverse difficulty that can be the result in of one’s snoring.
    There’s the nasal strips that open up the airways inside
    your nose, which can close up as a result of getting a cold, or
    perhaps even allergies, they are what I use when I’ve a cold and unable to
    make use of my cpap. Then you can find the throat sprays which assist avoid the soft tissue inside your mouth and throat
    from vibrating to lead to the snoring sounds. After which needless to say you will find
    the sprcific mouth guards that pull your lower jjaw forward tto open any airways,
    and avoid further blockage.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: