Build error with Mahout
Interestingly enough, with as low as 1GB for the RAM and Swap sizes, you should run into the exact same issue during the build of Mahout project from source than the one described in this previous post, HBase memory issue.
$ svn co http://svn.apache.org/repos/asf/mahout/trunk mahout $ cd mahout $ mvn
From core/target/surefire-reports/org.apache.mahout.cf.taste.hadoop.item.RecommenderJobTest.txt, I had the same error:
-------------------------------------------------------------------------------
Test set: org.apache.mahout.cf.taste.hadoop.item.RecommenderJobTest
-------------------------------------------------------------------------------
Tests run: 21, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 33.882 sec <<< FAILURE!
testCompleteJobBoolean(org.apache.mahout.cf.taste.hadoop.item.RecommenderJobTest) Time elapsed: 15.717 sec <<< ERROR!
java.io.IOException: Cannot run program "chmod": java.io.IOException: error=12, Cannot allocate memory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:149)
at org.apache.hadoop.util.Shell.run(Shell.java:134)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:286)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:354)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:337)
at org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:481)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:473)
at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:280)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:266)
at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:573)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:447)
at org.apache.mahout.cf.taste.hadoop.item.RecommenderJob.run(RecommenderJob.java:234)
As mentioned here, you should increase the size of your swap partition. The swap partition was not big enough for the system to fork the java process while simultaneously keeping the memory pages required by the other running applications.
Increase your Swap partition
To get more swap, you can create a swap file. A cleaner way is to change your partition table, shrink one partition to get some space and then increase the swap one. You want to know what you are doing here. Read carefully the Partition guide and back-up your personal files before changing anything. In my case, I got lucky as I was able to shrink an unimportant primary partition (/dev/sda2) to give additional space to the swap (the logical partition /dev/sda8) located in a different primary partition (/dev/sda1). Following is the former partition table that had the same amount of swap than RAM, 1GB. The newer now has twice as much as RAM, 2GB. This was the old partition table:
# fdisk /dev/sda
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): p
Disk /dev/sda: 58.5 GB, 58506416640 bytes
255 heads, 63 sectors/track, 7113 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xcfce28df
Device Boot Start End Blocks Id System
/dev/sda1 1 3649 29310561 5 Extended
/dev/sda2 * 3650 7112 27816547+ 7 HPFS/NTFS
/dev/sda5 1 973 7815559+ 83 Linux
/dev/sda6 974 1095 979933+ 83 Linux
/dev/sda7 1096 3527 19535008+ 83 Linux
/dev/sda8 3528 3649 979933+ 82 Linux swap / Solaris
/dev/sda8 got assigned the cylinders from 3528 to 3649. Its size is 121 * 8225280 / 1024 = 971 932 kB ( ~ 1 GB)
To perform the modification, I followed these steps:
- Count the number of additionnal cylinders required for the swap partition (121)
- Run fdisk /dev/sda
- Shrink primary partition /dev/sda2 : delete it (d), add it again (n) with less cylinders (starting cylinder is now 3771 instead of 3650) and write the partition table to disk (w).
- Run kpartx /dev/sda
- Expand extended partition /dev/sda1 : (d) then (n), with more cylinders (ending cylinder is 3770 instead of 3649). Recreate logical partitions /dev/sda5, /dev/sda6, /dev/sda7 ((n) 3 times with the same corresponding start/end cylinders than before) and finally /dev/sda8 with more cylinders (ending cylinder is now 3770 instead of 3649). Assign 82 as system id to /dev/sda8 to tag it as "Linux swap / Solaris" (t). Then (w).
- Run kpartx /dev/sda again
- Reformat the modified partitions:
Format /dev/sda2 partition with ext3 filesytem. You will loose everything.
# mke2fs -j /dev/sda2
Format /dev/sda8 as swap:
# swapoff /dev/sda8 # mkswap -f /dev/sda8 # swapon /dev/sda8
This is the new partition table:
Device Boot Start End Blocks Id System
/dev/sda1 1 3770 30282493+ 5 Extended
/dev/sda2 3771 7113 26852647+ 83 Linux
/dev/sda5 1 973 7815559+ 83 Linux
/dev/sda6 974 1095 979933+ 83 Linux
/dev/sda7 1096 3527 19535008+ 83 Linux
/dev/sda8 3528 3770 1951866 82 Linux swap / Solaris
With the top command, you should see the new total amount of swap available.
top - 09:45:16 up 1:48, 6 users, load average: 0.01, 0.04, 0.01
Tasks: 139 total, 1 running, 138 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.0%us, 1.0%sy, 0.0%ni, 96.8%id, 0.0%wa, 0.7%hi, 0.5%si, 0.0%st
Mem: 1026468k total, 668960k used, 357508k free, 22036k buffers
Swap: 1951856k total, 168876k used, 1782980k free, 438632k cached
Conclusion
Run the build in Mahout directory with mvn command. It should now be successful.
