Sunday, February 20, 2011

Increase your Swap partition

Build error with Mahout


Interestingly enough, with as low as 1GB for the RAM and Swap sizes, you should run into the exact same issue during the build of Mahout project from source than the one described in this previous post, HBase memory issue.

$ svn co http://svn.apache.org/repos/asf/mahout/trunk mahout
$ cd mahout
$ mvn

From core/target/surefire-reports/org.apache.mahout.cf.taste.hadoop.item.RecommenderJobTest.txt, I had the same error:


-------------------------------------------------------------------------------
Test set: org.apache.mahout.cf.taste.hadoop.item.RecommenderJobTest
-------------------------------------------------------------------------------
Tests run: 21, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 33.882 sec <<< FAILURE!
testCompleteJobBoolean(org.apache.mahout.cf.taste.hadoop.item.RecommenderJobTest)  Time elapsed: 15.717 sec  <<< ERROR!
java.io.IOException: Cannot run program "chmod": java.io.IOException: error=12, Cannot allocate memory
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:149)
        at org.apache.hadoop.util.Shell.run(Shell.java:134)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:286)
        at org.apache.hadoop.util.Shell.execCommand(Shell.java:354)
        at org.apache.hadoop.util.Shell.execCommand(Shell.java:337)
        at org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:481)
        at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:473)
        at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:280)
        at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:266)
        at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:573)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:447)
        at org.apache.mahout.cf.taste.hadoop.item.RecommenderJob.run(RecommenderJob.java:234)



As mentioned here, you should increase the size of your swap partition. The swap partition was not big enough for the system to fork the java process while simultaneously keeping the memory pages required by the other running applications.

Increase your Swap partition



To get more swap, you can create a swap file. A cleaner way is to change your partition table, shrink one partition to get some space and then increase the swap one. You want to know what you are doing here. Read carefully the Partition guide and back-up your personal files before changing anything. In my case, I got lucky as I was able to shrink an unimportant primary partition (/dev/sda2) to give additional space to the swap (the logical partition /dev/sda8) located in a different primary partition (/dev/sda1). Following is the former partition table that had the same amount of swap than RAM, 1GB. The newer now has twice as much as RAM, 2GB. This was the old partition table:


# fdisk /dev/sda

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): p

Disk /dev/sda: 58.5 GB, 58506416640 bytes
255 heads, 63 sectors/track, 7113 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xcfce28df

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1        3649    29310561    5  Extended
/dev/sda2   *        3650        7112    27816547+   7  HPFS/NTFS
/dev/sda5               1         973     7815559+  83  Linux
/dev/sda6             974        1095      979933+  83  Linux
/dev/sda7            1096        3527    19535008+  83  Linux
/dev/sda8            3528        3649      979933+  82  Linux swap / Solaris


/dev/sda8 got assigned the cylinders from 3528 to 3649. Its size is 121 * 8225280 / 1024 = 971 932 kB ( ~ 1 GB)


To perform the modification, I followed these steps:

  1. Count the number of additionnal cylinders required for the swap partition (121)
  2. Run fdisk /dev/sda
  3. Shrink primary partition /dev/sda2 : delete it (d), add it again (n) with less cylinders (starting cylinder is now 3771 instead of 3650) and write the partition table to disk (w).
  4. Run kpartx /dev/sda
  5. Expand extended partition /dev/sda1 : (d) then (n), with more cylinders (ending cylinder is 3770 instead of 3649). Recreate logical partitions /dev/sda5, /dev/sda6, /dev/sda7 ((n) 3 times with the same corresponding start/end cylinders than before) and finally /dev/sda8 with more cylinders (ending cylinder is now 3770 instead of 3649). Assign 82 as system id to /dev/sda8 to tag it as "Linux swap / Solaris" (t). Then (w).
  6. Run kpartx /dev/sda again
  7. Reformat the modified partitions:

Format /dev/sda2 partition with ext3 filesytem. You will loose everything.

# mke2fs -j /dev/sda2

Format /dev/sda8 as swap:

# swapoff /dev/sda8
# mkswap -f /dev/sda8
# swapon /dev/sda8


This is the new partition table:


   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1        3770    30282493+   5  Extended
/dev/sda2            3771        7113    26852647+  83  Linux
/dev/sda5               1         973     7815559+  83  Linux
/dev/sda6             974        1095      979933+  83  Linux
/dev/sda7            1096        3527    19535008+  83  Linux
/dev/sda8            3528        3770     1951866   82  Linux swap / Solaris




With the top command, you should see the new total amount of swap available.


top - 09:45:16 up  1:48,  6 users,  load average: 0.01, 0.04, 0.01
Tasks: 139 total,   1 running, 138 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.0%us,  1.0%sy,  0.0%ni, 96.8%id,  0.0%wa,  0.7%hi,  0.5%si,  0.0%st
Mem:   1026468k total,   668960k used,   357508k free,    22036k buffers
Swap:  1951856k total,   168876k used,  1782980k free,   438632k cached


Conclusion


Run the build in Mahout directory with mvn command. It should now be successful.

1 comment: