blog.bloke.com

Maybe I’m spoiled from the CVS integration into eclipse.  But certain things about subclipse are terrible (sorry I can’t compare to subversive since I didn’t try it yet).  But when I do team -> replace with -> latest from repository, then I expect it to say “are you sure” and then just do it.  Not complain, blah .. try cleanup .. blah .. it’s not a working copy.  Make it one. argghh…

(on the flip side, the command line interface seems since, but I like the graphical compare).

Okay .. one more gripe:

The best thing about svn so far is the ability to delete (which is *so* annoying in CVS), but, you have to mark it in svn ignore (which you can’t do at commit time .. annoying) *and* you have to do an update after the delete else it’s not a current version.  This is true when you either delete a file, or you add one to svnignore.

I had once considered moving from CVS to SVN for the previous business.  We taught many people how to use CVS through eclipse (perhaps 40-50 people, many not very technical).  But the things that go wrong with eclipse and SVN are such a pain, I wouldn’t recommend such a transition (and I was to do it again, I would have to think long about actually just starting with SVN).

okay, another gripe:

Need the feature: ignored resources (global ignore) inside eclipse (not sure if it exists outside svn).

The easy way to run a job every 5 minutes is to use */5 for the minute. So to run a cleanup script every 5 minutes use: (crontab -e)

*/5 * * * * /usr/local/bin/cleanup.sh

The format of the cron is:

#min hour dayofmonth month dayofweek commandtorun
*/10 * * * * /usr/local/bin/cleanup.sh

Would run it every 10 minutes. Alternatively, you could use:

0,10,20,30,40,50 * * * * /usr/local/bin/cleanup.sh

Sometimes finding an unbalanced quote can be a real pain, so here is a quick hack to find them (it doesn’t work in all cases, hence ‘it’s a quick hack’

take the file (lets call it /tmp/broken.html)

 cat /tmp/broken.html | sed -e 's/[^"]//g' | sed -e 's/""/xx/g' > /tmp/blah

now vi the file /tmp/blah, and look for “, if you find one, there is an odd number of “s on that line, so note the line number and look in /tmp/broken.html on that line to see what the problem might be.

so just breaking down the two sed statements.

sed -e 's/[^"]//g'

This looks for all the characters that are not ” and replaces them with nothing (ie removes them). The g at the end means do it more than once per line. So after this point, we have just the ” from each line. An example of the output at this point would be

""
""""""

"""

Now we want to remove the even “s and so we pair them up and turn them into x’s. leaving the ” all by itself.

 sed -e 's/""/xx/g'

This means when you find two quotes (“”) replace it with xx, and do this a number of times on the line (the g at the end).
So the result would then look like:

xx
xxxxxx

xx"

And we can see on the last line there is an unbalanced “.

This would work equallt well for single quotes, or anything else, as long as you escape it correctly, and that they appear on the same line.

I’ve done this so many times, but can still never remember the exact steps, and some of the things that can go wrong.

Of course, I recently tried it and it failed. Hence the last line (chmod). I found this by checking the log file on the remote machine

which said:

So I ran the following on the remote machine, and it worked just fine!

If you are going to use this for some automated script, don’t forget to log in once manually, so that you already have the host in your known_hosts file.
As for the security implications, we’ll leave that as an exercise for the reader

Links

• http://www.bloke.com/linux/ssh/

G’Day,
I run my own template engine using JSP’s. After I added some dynamic templates (I generate the JSP’s with a JSP ) the number of JSP’s exploded and tomcat started to run slowly. So easy fix was just to allocate some more memory. I edited startup.sh:

exec "$PRGDIR"/"$EXECUTABLE" start "$@" -Xmx4096m

but the proper fix will be to unload the servlet after they are used (since they are only used once).

Also note, I’m running 64bit, so I can say 4096m It’s nice when 64bit actually helps rather than causes you pain.

Interestingly enough, the problem can be traced to tomcat, and someone recently submitted a patch for exactly the behaviour I’m seeing!

• https://issues.apache.org/bugzilla/show_bug.cgi?id=48358

And (Since I didn’t apply the patch), but the current last line in my startup.sh is:

exec "$PRGDIR"/"$EXECUTABLE" start "$@" -Xms4000m -Xmx5096m -XX:-UseGCOverheadLimit

okay, so that all didn’t work that great, and so I built apache with the above patch. Works great.

I started reading about parallel make since I have a build process that takes quite a while. What a pain, I quickly (within a minute) discarded the idea as a waste of time, and figured I could do it in bash.

The main problem is that I have a slow script (that I can’t be bothered rewriting) that is run over and over again to generate different parts of my website.

So the target used to look like this:

shapehtml-target: force
        ./genshapehtml 96 58 4 arrowleft 0 0 18
        ./genshapehtml 96 58 4 arrowright 0 0 18
        ./genshapehtml 96 39 4 capsule 0 0 18
        ./genshapehtml 156 39 4 capsule-wide 0 0 18
        ./genshapehtml 96 58 4 fun 0 0 18
        ./genshapehtml 103 93 4 heart 0 -7 18
        ./genshapehtml 96 96 4 octagon 0 0 18
        ./genshapehtml 96 58 4 oval 0 0 18
        ./genshapehtml 96 57 4 rounded 0 0 18
        ./genshapehtml 156 57 4 rounded-wide 0 0 24
        ./genshapehtml 104 58 4 shard 0 -2 18
        ./genshapehtml 102 41 4 trapezoid 0 0 18

So when I time this part it takes 2m 23 seconds, a complete eternity.

[cameron@mars buttons]$ time make shapehtml-target
./genshapehtml 96 58 4 arrowleft 0 0 18
...
./genshapehtml 102 41 4 trapezoid 0 0 18

real    2m23.236s
user    0m44.110s
sys     1m43.508s

but I just run all the scripts in the background, and added a wait at the end. So now the target looks like this:

shapehtml-target: force
        ./genshapehtml 96 58 4 arrowleft 0 0 18 &\
        ./genshapehtml 96 58 4 arrowright 0 0 18 &\
        ./genshapehtml 96 39 4 capsule 0 0 18 &\
        ./genshapehtml 156 39 4 capsule-wide 0 0 18 &\
        ./genshapehtml 96 58 4 fun 0 0 18 &\
        ./genshapehtml 103 93 4 heart 0 -7 18 &\
        ./genshapehtml 96 96 4 octagon 0 0 18 &\
        ./genshapehtml 96 58 4 oval 0 0 18 &\
        ./genshapehtml 96 57 4 rounded 0 0 18 &\
        ./genshapehtml 156 57 4 rounded-wide 0 0 24 &\
        ./genshapehtml 104 58 4 shard 0 -2 18 &\
        ./genshapehtml 102 41 4 trapezoid 0 0 18 &\
        wait

and the result of running this parallel target: 17 seconds. Quite acceptable. (around 1/10th of the time)

[cameron@mars buttons]$ time make shapehtml-target
./genshapehtml 96 58 4 arrowleft 0 0 18 &\
 ./genshapehtml 96 58 4 arrowright 0 0 18 &\
...
./genshapehtml 102 41 4 trapezoid 0 0 18 &\
 wait

real    0m17.413s
user    0m35.692s
sys     1m32.897s
[cameron@mars buttons]$

Of course, your mileage might vary, I have I7 quad core with hyperthreading and a relatively fast SSD. And while the script is running, you notice much better CPU ultization.

If you notice the CPU is pegged, then just don’t run as many in parallel, and add wait after a couple. eg:

shapehtml-target: force
        ./genshapehtml 96 58 4 arrowleft 0 0 18 &\
        ./genshapehtml 96 58 4 arrowright 0 0 18 &\
        ./genshapehtml 96 39 4 capsule 0 0 18 &\
        ./genshapehtml 156 39 4 capsule-wide 0 0 18 &\
        wait
        ./genshapehtml 96 58 4 fun 0 0 18 &\
        ./genshapehtml 103 93 4 heart 0 -7 18 &\
        ./genshapehtml 96 96 4 octagon 0 0 18 &\
        ./genshapehtml 96 58 4 oval 0 0 18 &\
        wait
        ./genshapehtml 96 57 4 rounded 0 0 18 &\
        ./genshapehtml 156 57 4 rounded-wide 0 0 24 &\
        ./genshapehtml 104 58 4 shard 0 -2 18 &\
        ./genshapehtml 102 41 4 trapezoid 0 0 18 &\
        wait

And similarly you might want to check the hard drive usage to make sure each process is not contending with each other.

The big drawback here is that if genshapehtml had failed with an exit code, the make will continue on.  In my case, this never happens (I always exit 0), but I could add a command after the wait to ensure that it succeeded: eg each program  writes success/fail to a file, and the testprogram checks that they all succeeded.

I want to use it in my own application, so I use:

#include <syslog.h>
int
main(int argc, char *argv[])
{
  char *me = argv[0];
  openlog(me, LOG_NDELAY, LOG_LOCAL0);
  syslog(LOG_NOTICE,"started.");
}

and then when I want to log something:

  syslog(LOG_ERR,"Unable to connect to %s",remote_host);

So now if I want to log those messages to a separate file, edit /etc/syslog.conf

*.info;mail.none;authpriv.none;cron.none;local0.none    /var/log/messages

# Log myapp stuff (local0)
local0.*                                                /var/log/myapp

If you do a lot of logging, then adding a dash before the filename will not flush after every write, which is more efficient, but runs the risk of loosing information if the plug is pulled.

# Log myapp stuff (local0)
local0.*                                                -/var/log/myapp


And restart with

/etc/rc.d/init.d/syslogd restart

If you want to get syslog to send you an email when something happens, then as root:

mkdir -p /etc/syslog/
mkfifo /etc/syslog/mail.pipe

change /etc/syslog.conf (and restart after with /etc/rc.d/init.d/syslogd restart)

#log myapp to named pipe (logs crit and error messages)
local0.crit   |/etc/syslog/mail.pipe
local0.error   |/etc/syslog/mail.pipe
local.* /var/log/myapp

add this to /usr/sbin/syslogMail (and “chmod 755 /usr/sbin/syslogMail”)

#!/bin/bash

# syslogMail < /etc/syslog/mail.pipe
# run from cron every 5 minutes
# */5 * * * * /usr/sbin/syslogMail < /etc/syslog/mail.pipe

to="me@example.com"
TMOUT=1   #set timeout for read line to 1 second

FILE=/tmp/syslogMail.$$

while read line
do
 echo ${line} | grep -v "message repeated" >> $FILE
done

lines=`wc -l $FILE | awk '{print $1}' `
if [ $lines > 0 ];  then
 cat $FILE | mailx -s "syslog mail" $to
fi
/bin/rm -f $FILE

and then set up a cron job to run every 5 minutes (crontab -e)

*/5 * * * * /usr/sbin/syslogMail < /etc/syslog/mail.pipe

G’Day,

After some significant pain, I setup linux dual boot for the mac, but what I really want is to sometimes boot into linux, and sometimes just access the linux from my macos.  So you think no worries, Parallels supports Boot Camp, just use that. Wrong.  For Parallels to use a file system it needs to be mountable by the mac. CentOS uses ext3, and by default the mac doesn’t support ext2/ext3.  So

• Step 1. Install CentOS using Boot Camp.  (see here…) TODO
• Step 2. Reboot into MacOS, and install Mac Fuse
• Step 3. Install fuse-ext2 (see Mount Ext2/Ext3 file system from a mac)
• Step 4. Create a new Parallels install using the custom settings and boot camp partition. Here are the step-by-step screen shots.

Create New Virtual Machine

Choose Boot Camp Partition, click Continue

Select Version: More Linux -> CentOS Linux

Click Continue

Virutal Machine Type: Click Custom

CPU and Memory Options: I like 1G of main memory, click continue

Hard Disk Options: choose Boot Camp Partition, click Continue

I only have one disk, but pick the right one, click Continue

I like Shared Networking (Recommended), click Continue

Optimize for Virtual machine (Recommended) seems like a good choice, after all if you want to optimize for the mac, then don’t run the virtual machine.  You can always change this later.

Pick a name and a location for the Virtual machine, click Create.

When you start up the vm .. done

And soon enough you see the CentOS linux running inside my mac.

I needed to mount an ext3 file system which is on my dual boot mac (I’m doing that to allow parallels to run from “boot camp”). So my idea is to just mount the drive.  To do that:

• I installed macfuse, then fuse-ext2 from:

http://sourceforge.net/projects/fuse-ext2/

Once downloaded and extracted, then

Segmentation fault

damn, there is a bug in fuse-ext2.c, first line! edit fuse-ext2/fuse-ext2.c, comment out this (line 324)

And I see there is a MACOSX.build file, so lets do this instead:

then double click on the installer from the finder: fuse-ext2-0.0.7.dmg

magic.

Note that it is mounted as ext2 (and ext3 is backward compatible with ext2), and the net effect is that you lose the journal.  That sounds bad I hear the non-techies (and some techies) say. Well, the best thing about ext3 is the journal, and what it does is provide a fast way to recover the file system if (and when) it is not shut down cleanly (like power out). The downside of not having the journal is that after an unclean shutdown, it can take a while to go through all the files and make the file system clean again.

I want to get linux running natively, as well as from parallels. So the first step is to install linux direct to the hard drive. This one would think is straight forward, but the grub bootloader fails with CentOS install (silently) so you have to go back in and do this yourself.

The Setup:

• on a 15″ 2.8Ghz MacBook Pro that I bought in late 2009 (from HongKong if it matters).
• 4G of memory (2x2G)
• 320G, 7200 RPM drive.
• MacOS 10.6.2,
• Parallels 5.0 build 9310
• BootCamp 3.0.1 (uh oh, maybe that was a source of problems .. 3.1 is available)

• Install MacFuse and fuse-ext. This will save you a lot of pain.
• Partition your drive. Strangely, just don’t use BootCamp, use disk util.
• Parition your drive. Strangely, just don’t use BootCamp, there is no need.

Using diskutil to partition:

This is certainly the recommended way.  You can resize, and repartition all in the one go.

diskutil blah blah .. TODO

Using Bootcamp to partition:

• applications -> utilities -> boot camp assistant, click continue

insert pic

• drag size to be 200GB for windows .. 97 gb for windows, click partition

insert pic

• If you have problems partitioning, kill the parallels processes (which are prl_.. and prl_…)
• Click Quit and Install Later

Now for the CentOS install.

• put centos 5.4 CD into the drive and reboot (apple -> Restart…)
• hold down the ‘c’ key. This will boot from the CD.
• hit Enter – (see other article on the CentOS setup)

• When you get to the partitioning part, chose a custom layout.  You don’t need a swap (although you can have one). You just need /. Which is what I did.
• Delete the /dev/sda3 vfat parttion  (click on it, then click delete, then click delete)
• Click “New”
• Set the mount point to /
• Set the size to maximum allowable size (or pick a size)
• Click OK
• Click Next
• ignore swap partition (click yes).
• You can install the bootloaded on sda3 if you want, but it’s going to fail anyway. So pick no bootloader.
• Click Next (install takes 12-15 minutes once started on my 320Gb 7200 pm drive).
• click reboot after CD comes out.
  • hmm .. again the linux is hung there, so 5 seconds pressing the power button to power it down, then press again to turn it back on.

Now if we reboot into linux (using refit, or holding down the option key on startup), we probably see that it fails with “Can’t find operating system” or similar.  This is because grub failed to install, so we need to reinstall it.

• Place the CentOS 5.4 CD (mine is 64bit) in to the CD, reboot, and hold down ‘c’.
• At the prompt type linux rescue.
• Choose all the default options (no need for network) and finally get to a shell prompt

# chroot /mnt/sysimage
# grub-install /dev/sda3
Installation finished. No error reported.
This is the contents of the device map /boot/grub/device.map.
Check if this is corrrect or not. If any of the lines is incorrect,
fix it and re-run the script `grub-install'.

# this device map was generated by anaconda
(hd0)     /dev/sda
#

Which was infact correct. I many times did get the error that it couldn’t load the stage1 file.