Joe Mocker's Weblog How'd you know I was blogging at you if you weren't blogging at me? en-us Copyright 2014 Wed, 22 Jan 2014 17:26:26 +0000 Apache Roller BLOGS401ORA6 (20130904125427) Setting up Subversion over SSH on Solaris mock Wed, 26 Sep 2007 13:29:15 +0000 Sun <p> Before I forget how I did it, I figured I should probably document some steps on how I set up a Subversion repository on Solaris so that it can be accessed over SSH, using svn+ssh:// URLs. </p> <p> The <a href="">Tunneling over SSH</a> section in the <b>Version Control with Subversion</b> book actually does a pretty good write up of the basics. The main tweak that I did with Solaris was to beef security slightly by creating a Solaris Role on the repository server which can only execute a limited number of subversion commands. Here's what I did. </p> <p> First off, I created a RBAC profile called "Source Code Shell" which can run only the <tt>svnserve</tt> command, and runs it as the source code management user I had previously created called <tt>scm</tt> </p> <pre> cat >> /etc/security/exec_attr Source Code Shell:suser:cmd:::/opt/svn-1.4.0/bin/svnserve:uid=scm Source Code Shell:suser:cmd:::/opt/svn/bin/svnserve:uid=scm cat >> /etc/security/prof_attr Source Code Shell:::Access Subversion Only: </pre> <p> Ok that was easy enough. There are probably some more appropriate commands to use than just adding to the files directly. I should probably look into that. </p> <p> Next, I need to create a role to assign this new profile to. The user name will be <tt>src</tt> </p> <pre> useradd -d /export/home/src -m -c "Source Code User" -s /usr/bin/pfsh -g 100 -u 242 src passwd -N src </pre> <p> And assign the RBAC Profile to the <tt>src</tt> user. </p> <pre> usermod -P "Source Code Shell" src </pre> <p> Now, the rest of this is pretty much all described in the "Tunneling over SSH" section mentioned earlier. Essentially what is done is to use the public-key authentication mechanisms in SSH to identify the incoming user and automatically start up the <tt>svnserve</tt> command in tunnel mode. </p> <p> This is accomplished by adding lines to the <tt>src</tt> user's <tt>authorized_keys</tt> file for each user who will be accessing the repository. The one thing you need for each user is their public-key file(s), typically <tt></tt> or <tt></tt> The format of the lines in <tt>authorized_keys</tt> is </p> <pre> command="/path/to/svnserve -t OPTIONS" KEY-TYPE KEY KEY-COMMENT </pre> <p> The <tt>-t</tt> option puts <tt>svnserve</tt> in tunnel-mode. There are a bunch of options you can pass to <tt>svnserv</tt>, but the most common ones are <tt>--tunnel-user</tt> to specify the username of the remote user and <tt>-r</tt> to specify a virtual root for the repository. The former allows subversion to recognize each user as their real username instead of the <tt>src</tt> user, so things like permissions work. The later allows you to hide the real path to the repository which can help shorten URLs and provide some abstration from the physical location in case you ever want to move it. </p> <p> In my case, lines in <tt>authorized_keys</tt> are looking like </p> <pre> command="/opt/svn/bin/svnserve -t -r /app/repos/svn --tunnel-user=mock" ssh-dss KEY mock@watt </pre> <p> Not very painful at all. Actually what is the more painful part is trying to get NetBeans on Windows to access Subversion over SSH. I'll write that up soon. </p> Tagsoup is Super! mock Thu, 13 Sep 2007 12:48:53 +0000 Sun <p> I've never been much of a fan of screen-scraping but I seem to be doing a fair amount in my spare time recently. </p> <p> For example, I put together an application to run a private <a href="">NASCAR</a> "fantasy" pool with some friends and family. One of the features I have is a live update mechanism to see where everyone stands during the race. </p> <p> Only problem, I couldn't find a simple, easy data feed to get the current race information. Ok, so, the next best thing, look at one of the sports sites like the <a href="">Yahoo NASCAR update</a> and try to extract information out of the HTML. </p> <p> Well, the data looks relatively well formatted, but as we all know browsers are pretty lenient about what they accept as HTML, what with missing close tags and so on, and I wanted to use an XML parser, and even XSLT to extract the data. So I needed a way to fix the HTML before passing it to a parser. </p> <p> Enter <a href="">Tagsoup</a>. A SAX-compliant HTML parser that spits out well-formatted XML. Ah that sounds like just the ticket. And even better, the maintainers include a modified version of Saxon - <a href="">TSaxon</a> - to process XSLT. </p> <p>So with TSaxon, in hand, it made easy work of converting something like an <a href="">ESPN Qualifying Grid</a> into SQL that I can load into Derby with an XSL like <pre> &lt;xsl:stylesheet xmlns:xsl="" xmlns:html="" version="1.0"&gt; &lt;xsl:param name="race"/&gt; &lt;xsl:param name="season"/&gt; &lt;xsl:param name="table"/&gt; &lt;xsl:param name="type"/&gt; &lt;xsl:template match="/"&gt; &lt;xsl:apply-templates select="//html:table[html:tr[html:td[@colspan='5']]]"/&gt; &lt;/xsl:template&gt; &lt;xsl:template match="html:table"&gt; &lt;xsl:text&gt;delete from &lt;/xsl:text&gt; &lt;xsl:value-of select="$table"/&gt; &lt;xsl:text&gt; where season = &lt;/xsl:text&gt; &lt;xsl:value-of select="$season"/&gt; &lt;xsl:text&gt; and race = &lt;/xsl:text&gt; &lt;xsl:value-of select="$race"/&gt; &lt;xsl:text&gt;;&amp;#10;&lt;/xsl:text&gt; &lt;xsl:apply-templates select="html:tr[@class = 'evenrow' or @class='oddrow']"/&gt; &lt;xsl:text&gt;commit;&amp;#10;&lt;/xsl:text&gt; &lt;/xsl:template&gt; &lt;xsl:template match="html:tr"&gt; &lt;xsl:variable name="pos" select="html:td[1]"/&gt; &lt;xsl:variable name="car" select="normalize-space(html:td[3])"/&gt; &lt;xsl:variable name="spd" select="html:td[5]"/&gt; &lt;xsl:text&gt;insert into &lt;/xsl:text&gt; &lt;xsl:value-of select="$table"/&gt; &lt;xsl:text&gt; values (&lt;/xsl:text&gt; &lt;xsl:value-of select="$season"/&gt; &lt;xsl:text&gt;, &lt;/xsl:text&gt; &lt;xsl:value-of select="$race"/&gt; &lt;xsl:text&gt;, &lt;/xsl:text&gt; &lt;xsl:value-of select="$pos"/&gt; &lt;xsl:text&gt;, '&lt;/xsl:text&gt; &lt;xsl:value-of select="$car"/&gt; &lt;xsl:text&gt;', '&lt;/xsl:text&gt; &lt;xsl:value-of select="$spd"/&gt; &lt;xsl:text&gt;', &lt;/xsl:text&gt; &lt;xsl:value-of select="$type"/&gt; &lt;xsl:text&gt;);&amp;#10;&lt;/xsl:text&gt; &lt;/xsl:template&gt; &lt;/xsl:stylesheet&gt; </pre> <p> By passing it to TSaxon like </p> <pre> java -jar lib/saxon.jar -H "$1" xsl/grid-to-sql.xsl race=$2 season=$3 table=grid type=$4 </pre> <p> Ok, so that's pretty cool by itself. But now with the NFL season just beginning, and another private "fantasy" pool among relatives, I found that I wanted to do a similar "live tracker" to see everyone's points in the pool. This time I wanted to do it a little bit differently. </p> <p> I wanted to programmatically call Tagsoup to parse a page and pass it through an XSLT. So why not try to use all the JAXP facilities in Java. And it turns out to be pretty easy. This easy <pre> // Create an instance of Tagsoup SAXBuilder builder = new SAXBuilder("org.ccil.cowan.tagsoup.Parser"); // Parse my (HTML) URL into a well-formed document Document doc = URL(poolurl)); JDOMResult result = new JDOMResult(); JDOMSource source = new JDOMSource(doc); // Get a JAXP Factory TransformerFactory factory = TransformerFactory.newInstance(); // Get a Transformer with the XSL loaded. StreamSource sheet = new StreamSource(sheetpath); Transformer transformer = factory.newTransformer(sheet); // Transform the page transformer.transform(source, result); // Spit out the result. XMLOutputter outputter = new XMLOutputter(Format.getPrettyFormat()); outputter.output(result.getDocument(), out); </pre> <p> The middle section is really the part where JAXP comes into play. But as you can see its quite simple. </p> <p> Oh, and say you want to just programmatically select nodes with XPath. That's pretty easy too. Here's an example that gets the Title of the page </p> <pre> JDOMXPath titlePath = new JDOMXPath("/h:html/h:head/h:title"); titlePath.addNamespace("h", ""); String title = ((Element) titlePath.selectSingleNode(doc)).getText(); out.println("Title is " + title); </pre> <p> Voila. </p> Updated JVM Options List mock Tue, 28 Aug 2007 15:13:33 +0000 Sun <p> Its been quite a while, but I finally spent some time to update my <a href="">List of JVM Options</a> to include everything in Java 6. This is just the first pass as Java 6, so most of the new Java 6 options have no description for them. I still need to go out and search for references for any/all of them. </p> <p> Hope people find this useful. </p> performance mock Mon, 16 Apr 2007 15:05:33 +0000 Sun <p> A couple of months ago, we went through a tuning exercise for our primary web site <a href=""></a>. A few execs noticed some strange pauses during page rendering (of primary the home page.) So a few of us started to look closer at the problem. </p> <p> Before I get into it a bit, I just took a look at the daily reports again for today, and I'm happy to report that performance is still looking pretty good. <p> <p> Although I'm not currently on the engineering team for the site, the I was one of the primary "architects" for the framework that serves the site, and I still have soft spot for it. I take offense when someone challenges the performance of the site. So, I decided to get involved, mainly to help gather some statistics for how the site is performing. </p> <p> So what did performance look like when we looked into it? Here are a couple of charts showing what it looked like. The first chart shows the average response time to retrieve all the home page components. The second shows each component in detail. </p> <center> <table border="0" width="60%"> <tr> <td align="center"> <a href="/mock/resource/tuning-1/avg-dur-2007-02-05-sun.png"><img src="/mock/resource/tuning-1/avg-dur-2007-02-05-sun-sm.png"></a> <br> Averages </td> <td align="center"> <a href="/mock/resource/tuning-1/dur-2007-02-05-sun.png"><img src="/mock/resource/tuning-1/dur-2007-02-05-sun-sm.png"></a> <br> Details </td> </tr> </table> </center> <p> The charts don't look all that great. Very erratic, and pretty poor average response time. Now, the home page has about 70 components totaling about 396K. Making a very rudimentary guess at a theoretical minimum download time of all the components over say a 1.5Mbit DSL line, we're looking at about 2 seconds [ ((396 \* 8 / 1024) / 1.5) = 2 seconds ]. Now lets compare that against what we see on the average duration chart. Assuming a sequential download of components, even a 100 millisecond response time would take 7 seconds to download all the comonents [ 100 \* 70 / 1000 = 7 ]. Which wasn't far from the truth. </p> <p> One thing to note, through our analysis, there is actually some parallelism going on to retrieving the page components. There is a great Firefox Extension that will show this visually called <a href="">Firebug</a>. So that helps reduce some overall download time. </p> <p> Eventually we found the culprit. Having narrowed the problem down to some thing with Sun Java System Web Server, <a href="">Chris Elving</a> with Web Server Engineering helped us resolve the problem. As it turned out, it ended up being <a href="">Reverse Proxy Plugin Bug # 6435723</a>, which, fortunately had a fix available for it. </p> <p> So, how are we looking today? Well, here's the charts as of Friday. Not much fluctuation in response times throughout the day. </p> <center> <table border="0" width="60%"> <tr> <td align="center"> <a href="/mock/resource/tuning-1/avg-dur-2007-04-13-sun.png"><img src="/mock/resource/tuning-1/avg-dur-2007-04-13-sun-sm.png"></a> <br> Averages </td> <td align="center"> <a href="/mock/resource/tuning-1/dur-2007-04-13-sun.png"><img src="/mock/resource/tuning-1/dur-2007-04-13-sun-sm.png"></a> <br> Details </td> </tr> </table> </center> <p> I'm sure you notice the occasional blips of response times in the over 1.5 second range. Well this is due to occasional cache flushes and page rendering due to content changes. Our rendering framework makes extensive use of XML and J2EE technologies to dynamically render the web site. I'd say in light of that, the platform is doing an incredible job at serving content. </p> <p> For comparison... Here's similar charts for one of our competitors from Friday as well. Their performance hasn't changed much over the last two months. I'm not gonna tell them. </p> <center> <table border="0" width="60%"> <tr> <td align="center"> <a href="/mock/resource/tuning-1/avg-dur-2007-04-13-comp.png"><img src="/mock/resource/tuning-1/avg-dur-2007-04-13-comp-sm.png"></a> <br> Averages </td> <td align="center"> <a href="/mock/resource/tuning-1/dur-2007-04-13-comp.png"><img src="/mock/resource/tuning-1/dur-2007-04-13-comp-sm.png"></a> <br> Details </td> </tr> </table> </center> And now Chicken of the VNC tunneled through SSH on OS X mock Thu, 22 Mar 2007 15:19:17 +0000 Sun <p> After I posted yesterday about the <a href="">VNC Over SSH Startup Script</a> I use with TightVNC on Solaris, I started thinking about how I could do the same thing with my MacBook Pro and <a href="">Chicken of the VNC</a>. </p> <p> And I have just completed the first version. Its amazing what you can fumble through with an internet search engine. So here we go... </p> <p> <b>Password Please</b> -- One thing you don't get with CotVNC is the <tt>vncpasswd</tt> command. So we need a copy. I just did it the quick and dirty way by grabbing the TightVNC source and compiling only the <tt>vncpasswd</tt> command: </p> <pre> wget bzip2 -cd tightvnc-1.2.9_unixsrc.tar.bz2 | tar xvf - cd vnc_unixsrc gcc -I include -I libvncauth -o vncpasswd/vncpasswd libvncauth/\*.c vncpasswd/\*.c </pre> <p> Now squirrel off the <tt>vncpasswd/vncpasswd</tt> command in your favorite <tt>bin/</tt> directory. </p> <p> <b>Passphrase Please</b> -- If you look back at my previous post, I make use of <tt>ssh-agent</tt> and <tt>ssh-add</tt> to allow tunneling without entering a password. Only problem on OS X is that there is no <tt>gnome-ssh-askpass</tt> command available. So I had to hack one up myself. Its an odd combination of Bourne Shell and AppleScript, but it appears to work. Save a copy as <tt>macos-askpass</tt>. </p> <pre> #! /bin/sh # # An SSH_ASKPASS command for MacOS X # # Author: Joseph Mocker, Sun Microsystems # # To use this script: # setenv SSH_ASKPASS "macos-askpass" # setenv DISPLAY ":0" # TITLE=${MACOS_ASKPASS_TITLE:-"SSH"} DIALOG="display dialog \\"$@\\" default answer \\"\\" with title \\"$TITLE\\"" DIALOG="$DIALOG with icon caution with hidden answer" result=`osascript -e 'tell application "Finder"' -e "activate" \\ -e "$DIALOG" -e 'end tell'` if [ "$result" = "" ]; then exit 1 else echo "$result" | sed -e 's/\^text returned://' -e 's/, button returned:.\*$//' exit 0 fi </pre> <p> <b>Now lets put it all together</b> - I'm skipping over a bunch of stuff, assuming you have read my previous entry. But here's the final script that manages asking for all the information via dialogs, creating the SSH tunnel and starting up CotVNC with the appropriate arguments. <p> One thing to note here is to make sure the <tt>macos-askpass</tt> and <tt>vncpasswd</tt> commands are in the <tt>PATH</tt>. I would suggest you modify the <tt>PATH</tt> command in the following script to ensure this. </p> <p> Oh, also, change <tt>COTVNC_HOME</tt> to the location of your Chicken of the VNC application. </p> <pre> #! /bin/sh # # Script to put a GUI front end around Chicken of the VNC over SSH for MacOS X # # Author: Joseph Mocker, Sun Microsystems. # # This script works only with Chicken of the VNC but needs the # "vncpasswd" command from TightVNC. # COTVNC_HOME="/Applications/DarwinPorts/Chicken of the" COTVNC_CMD="$COTVNC_HOME/Contents/MacOS/Chicken of the VNC" PATH=$HOME/bin:$PATH export PATH # Define a general Prompt routine prompt () { TITLE="$PROMPT_TITLE" DIALOG="display dialog \\"$@\\" default answer \\"\\" with title \\"$TITLE\\"" DIALOG="$DIALOG with icon caution" result=`osascript -e 'tell application "Finder"' -e "activate" \\ -e "$DIALOG" -e 'end tell'` if [ "$result" = "" ]; then return 1 else echo "$result" | sed -e 's/\^text returned://' -e 's/, button returned:.\*$//' return 0 fi } # Prompt for VNC Server PROMPT_TITLE="VNC Connection" export PROMPT server=`prompt "Connect to server (host:port)"` if [ "x$server" = "x" ]; then exit 0 fi # Prompt for VNC Password MACOS_ASKPASS_TITLE="VNC Password" export MACOS_ASKPASS_TITLE passwd=`macos-askpass "Password for server ${server}:"` host=`echo $server | cut -d: -f1` port=`echo $server | cut -d: -f2` port=`expr $port + 5900` touch /tmp/tmpvnc.$$ chmod 600 /tmp/tmpvnc.$$ echo $passwd | vncpasswd -f > /tmp/tmpvnc.$$ # Start up a "private" SSH agent eval `ssh-agent -s` # Register SSH Identities with the agent DISPLAY=:0 export DISPLAY SSH_ASKPASS=macos-askpass export SSH_ASKPASS unset MACOS_ASKPASS_TITLE ssh-add < /dev/null # Now lets find an open port for tunnelling tunnel=41235 while [ "`telnet localhost $tunnel < /dev/null 2>&1 |grep refused`" == "" ]; do tunnel=`expr $tunnel + 1` done # Start up an SSH with a Local tunnel. ssh -L $tunnel:$host:$port $host "sleep 60" & # Wait for the tunnel to activate while [ "`telnet localhost $tunnel < /dev/null 2>&1 |grep refused`" != "" ]; do sleep 1 done "$COTVNC_CMD" localhost:$tunnel --PasswordFile /tmp/tmpvnc.$$ & # Give it a little bit to make the connection sleep 10 # Kill the "private" SSH Agent eval `ssh-agent -k` # Cleanup rm /tmp/tmpvnc.$$ </pre> <p> So you can now run the script in a Terminal window and have it do all the wonderful magic to burrow VNC through SSH. But that's not really in the point and click nature of MacOS so... </p> <p> <b>One tiny invoker</b> -- What would really be the icing is a little AppleScript to invoke this whole thing from, say, the Scripts menu - you do have the Scripts menu enabled don't you? Well then here you go. I call mine <i>VNC Over SSH</i> and shoved it in <tt>~/Library/Scripts</tt> </p> <pre> do shell script "$HOME/bin/macos-vncv > /tmp/vncv.$$ 2>&1 &" </pre> <p> Ahh... I don't know about you, but I feel a lot better about using VNC on the Mac now. </p> A startup GUI for running TightVNC over SSH mock Wed, 21 Mar 2007 13:21:34 +0000 Sun <p> Here's a little utility I put together to make it a little easier to startup TightVNC to run over SSH. Although TightVNC supports SSH itself, I wanted to be able to start it up without the need to always start it in a terminal window. So I got to thinking how I could use <tt>zenity</tt> with some scripts to do this. And here's what I came up with. </p> <p> Essentially, what this utility does is use the <tt>ssh-agent</tt> utility to register your public SSH identities - using <tt>ssh-add</tt> - so that you can start up an SSH session without having to enter a password. The key piece to making this whole thing work as a GUI is by use of the <tt>gnome-ssh-askpass</tt> utility included with OpenSSH. </p> <p> <b>First off, set some passphrases</b> -- So the first thing I would suggest is that you set a passphrase on all your SSH identities, otherwise someone could grab them, and gain entry to your account in various ways. The easiest way to do this is with <tt>ssh-keygen</tt> </p> <pre> ssh-keygen -p -f ~/.ssh/id_dsa ssh-keygen -p -f ~/.ssh/id_rsa ssh-keygen -p -f ~/.ssh/identity </pre> <p> Whew. I feel more secure already. </p> <p> <b>Up next, configure some authorized keys</b> -- Ok, so now that you have some passphrases set, you need to tell SSH that they are authorized to allow connections with. The key here is that you want to do this on the system you are running the VNC server on. What you need to do is grab the <tt></tt> and <tt></tt> files and put them in the <tt>authorized_keys</tt> file on the target. </p> <pre> source% scp ~/.ssh/ target:. source% scp ~/.ssh/ target:. target% cat ~/ >> ~/.ssh/authorized_keys target% cat ~/ >> ~/.ssh/authorized_keys </pre> <p> This should be enough to enable to to <tt>ssh</tt> to the target machine without a passowrd, it will however require you to enter a passphrase instead. This is good. </p> <p> <b>Configuring a password popup</b> -- One nice little feature of <tt>ssh-add</tt> is that it can defer to an optional (GUI) utility to ask for passphrases. And in fact if you look in the <tt>contrib/</tt> folder of the OpenSSH source, you will see a couple utilities for GNOME, the most relevant one now days being <tt>gnome-ssh-askpass2.c</tt>. </p> <p> You can get <tt>ssh-add</tt> to invoke this by setting the environment variable <tt>SSH_ASKPASS</tt> to the path of the utility and invoking a "<tt>ssh-add < /dev/null</tt>". </p> <p> Now, this is all well and good, but I wanted to generalize this a little bit so I can use it to ask for passwords in other contexts. First and foremost, TightVNC requires a password. And the <tt>gnome-ssh-askpass</tt> utility spits out some verbage specific to OpenSSH. I wanted to remove that. So a quick hack a the source and <tt>gnome-ssh-askpass</tt> became <tt>gnome-askpass</tt>. Here's the <tt>diff</tt>, which you can use with <tt>patch</tt> to enable my mods. </p> <pre> 33,35c33,35 < \* "GNOME_SSH_ASKPASS_GRAB_SERVER=true" then gnome-ssh-askpass will grab < \* the X server. If you set "GNOME_SSH_ASKPASS_GRAB_POINTER=true", then the < \* pointer will be grabbed too. These may have some benefit to security if --- > \* "GNOME_ASKPASS_GRAB_SERVER=true" then gnome-ssh-askpass will grab > \* the X server. If you set "GNOME_ASKPASS_GRAB_POINTER=true", then the > \* pointer will be grabbed too. These may have some benefit to security if 88a89 > char \*title; 95,96c96,97 < grab_server = (getenv("GNOME_SSH_ASKPASS_GRAB_SERVER") != NULL); < grab_pointer = (getenv("GNOME_SSH_ASKPASS_GRAB_POINTER") != NULL); --- > grab_server = (getenv("GNOME_ASKPASS_GRAB_SERVER") != NULL); > grab_pointer = (getenv("GNOME_ASKPASS_GRAB_POINTER") != NULL); 111,112c112,115 < < gtk_window_set_title(GTK_WINDOW(dialog), "OpenSSH"); --- > > title = getenv("GNOME_ASKPASS_TITLE"); > if (title == NULL) title = "OpenSSH"; > gtk_window_set_title(GTK_WINDOW(dialog), title); </pre> <p> With these mods, you can set a <tt>GNOME_ASKPASS_TITLE</tt> environment variable to change the title. </p> <p> <b>Finally, putting it all together</b> -- Are you still with me? Now here is the final script that I use to fire up TightVNC. When you use the script it prompts for three things <ol> <li>First, it asks which VNC Server (host:port) to connect to</li> <li>Next, it will ask for the VNC Server password</li> <li>Next, it will prompt you to enter your SSH Identity passphrase(s)</li> </ol> </p> <p> And now, without further adieu... </p> <pre> #! /bin/sh # # Script to put a GUI front end around TightVNC over SSH # # Author: Joseph Mocker, Sun Microsystems. # # This script works only with TightVNC # # Prompt for VNC Server server=`zenity --entry --title="VNC Server" --text="Connect to server (host:port):"` if [ "x$server" = "x" ]; then exit 0 fi # Prompt for VNC Password GNOME_ASKPASS_TITLE="VNC Server" export GNOME_ASKPASS_TITLE passwd=`gnome-askpass Password for server ${server}:"` host=`echo $server | cut -d: -f1` port=`echo $server | cut -d: -f2` echo $passwd | vncpasswd -f > /tmp/tmpvnc.$$ # Start up a "private" SSH agent eval `ssh-agent -s` # Register SSH Identities with the agent SSH_ASKPASS=gnome-askpass export SSH_ASKPASS unset GNOME_ASKPASS_TITLE ssh-add < /dev/null vncviewer -passwd /tmp/tmpvnc.$$ -encodings "hextile zlib raw" -via $host localhost:$port & # Give it a little bit to make the connection sleep 10 # Kill the "private" SSH Agent eval `ssh-agent -k` # Cleanup rm /tmp/tmpvnc.$$ </pre> Compiling UW imapd with SSL on Solaris 10 mock Wed, 14 Mar 2007 11:34:59 +0000 Sun <p> I've been running IMAP over SSL for a while on Solaris, but until recently I've used <a href="">STunnel</a> to provide the SSL support in front of a plain IMAP daemon. I've known that you could compile SSL into imap for a while but never really looked into it until <a href="/rama">Rama</a> figured out the magic certificate generation piece. </p> <p> But what Rama did was to just install the <a href="">Sunfreeware</a> version of imapd. I have a love/hate relationship with those types of distributions, so I decided to look at compiling it myself. Heck Solaris includes OpenSSL so it should be easy. </p> <p> Well, actually, I couldn't get it to build with the version of OpenSSL that ships with Solaris. Looking at <tt>syslog</tt> I'd see messages like: </p> <pre> Mar 14 10:23:24 watt imapd[5834]: [ID 853321 mail.error] SSL error status: error:140D308A:SSL routines:TLS1_SETUP_KEY_BLOCK:cipher or hash unavailable </pre> <p> And looking at the imapd binary I saw a missing <tt>libcrypto_extra</tt>. Searching the net I saw a bunch of people talking about it. It appears that this is no longer needed with Solaris 10, but others say that you need to install {{SUNWcry}} package. Well, I must be a loser because I could not find enough info to make it work. </p> <p> So I decided to just compile up a fresh copy of OpenSSL to use to compile imapd. So here's what I did. </p> <p> <b>Compiling OpenSSL</b> -- its pretty trivial to do, in this day and age, however my first attempt compiled it 64bit, and imapd had issues with that. There are a few extra configuration parameters to force it to 32 bit. Here's the <tt>Configure</tt> line. </p> <pre> Configure --prefix=/opt/openssl-0.9.8e 386 shared solaris-x86-gcc </pre> <p> After that.. Compile and Install... </p> <pre> gmake gmake install </pre> <p> <b>Compiling Imapd</b> -- The instructions in <tt>docs/SSLBUILD</tt> go over the basics. But there were a few additional changes I needed to make. The main change was to make sure imap was built with my OpenSSL instead of the Solaris version. All these changes were to <tt>src/osdep/unix/Makefile</tt>: </p> <p> Fist I set the <tt>SSLDIR</tt> and <tt>SSLCERTS</tt> variables to where I wanted them: </p> <pre> SSLDIR=/opt/openssl SSLCERTS=/etc/sfw/openssl/certs </pre> <p> Next, I forced it to use the static version of libcrypto.a by changing <tt>SSLCRYPTO</tt>: </p> <pre> SSLCRYPTO=$(SSLLIB)/libcrypto.a </pre> <p> Finally, I need to force it to use my static version of libssl.a. </p> <pre> SSLLDFLAGS= -L$(SSLLIB) $(SSLLIB)/libssl.a $(SSLCRYPTO) $(SSLRSA) </pre> <p> After that. Simply compile it up, and install it where ever you want: </p> <pre> gmake gso mkdir /opt/bin cp imapd/imapd /opt/bin </pre> <p> <b>Configuring the imapd certificate</b> -- Thanks to <a href="/rama">Rama</a> on the magic OpenSSL command. All that you really do is create a PEM certificate called <tt>imapd.pem</tt> in the OpenSSL <tt>certs</tt> folder: </p> <pre> cd /opt/sfw/openssl/certs openssl req -new -x509 -nodes -out imapd.pem -keyout imapd.pem -days 3650 </pre> <p> <b>Starting imapd from inetd</b> -- Ok well now with Solaris 10 this is done though SMF, but inetd has a conversion utility to do this. I put the following line in <tt>/etc/inetd.conf</tt> </p> <pre> imaps stream tcp nowait root /opt/bin/imapd imapd </pre> <p> Then added a line to <tt>/etc/services</tt> </p> <pre> imaps 1143/tcp imap2 # Internet Mail Access Protocol v2 </pre> <p> Then just run <tt>inetconv</tt> per instructions in <tt>inetd.conf</tt> and bob's your uncle. </p> One of my favorite Firefox extensions mock Tue, 13 Mar 2007 22:27:56 +0000 Sun <p> I use several computers on a daily basis. I've done this for quite a while. At work I have my workstation, at home I have a PC and lots of times I just use my laptop. The problem is I like to try to keep my desktop environment in sync across all those machines as much as possible. </p> <p> With Firefox, for example, I want all my bookmarks up to date everywhere. And fortunately, for Firefox, I have found an extension quite some time ago which helps do this. </p> <p> Its currently called <a href="">Bookmark Sync and Sort</a>, but previously it was called <i>Bookmark Synchronizer [2,3]</i>. </p> <p> What this extension allows you to do is to store your bookmarks on server somewhere, and load them in multiple Firefox instances. It does some cool things like allowing you to merge bookmarks on your server with any changes you've made locally. And it provides multiple protocols for saving and loading the bookmarks, including FTP and WebDAV. </p> <p> What I did was set up Sun Java System Web Server 7.0 on a server I can access from anywhere I am, and configured a WebDAV collection for <i>Bookmark Sync and Sort</i>. What about security? Well a couple of things. My SJSWS instance is configured for SSL with a self-signed certificate. And I have restricted the WebDAV collection to authenticated users. <i>Bookmark Sync and Sort</i> handles all this great. </p> <p> So how does one configure WebDAV with SJSWS 7.0? Its actually not that difficult. A good reference is <a href="">Meena Vyas' SJSWS WebDAV entry</a>. <p> <p> Specifically, here's what I did. First enable WebDAV with wadm </p> <pre> wadm&gt; enable-webdav --config=&lt;server-instance&gt; </pre> <p> Then create a WebDAV collection </p> <pre> wadm&gt; create-dav-collection --config=&lt;server-instance&gt; --vs=&lt;virtual-server&gt; --uri=/davx/marks --source-uri=/marks </pre> <p> This will create a WebDAV collection that you access via DAV at <tt>/marks</tt>, which is a little opposite of usual with WebDAV but we want to make sure the web server doesn't mess with the bookmarks file when it serves it. </p> <p> Now, that is enough to set up an open WebDAV collection. But it would be good to control access to the collection. So, from the Admin Console, Set up access control. Go to the <i>Access Control</i> tab of <i>Configurations -> server-instance</i>. </p> <p> Then select the <i>Users</i> tab and create or edit the users and passwords. </p> <p> Next, go to the <i>Access Control Lists (ACL)</i> tab. Edit the <b>default</b> ACL, and add an entry with the settings </p> <ul> <li> Access: Allow <li> Users & Groups: All in the authentication database <li> Rights: All Access Rights </ul> <p> Next, Edit the <b>dav-src</b> ACL, add a similar entry with the settings <p> <ul> <li> Access: Allow <li> Users & Groups: All in the authentication database <li> Rights: All Access Rights </ul> <p> Now, Deploy the configuration and you are done. </p> Force SSL Web Server Hack mock Thu, 23 Nov 2006 10:31:01 +0000 Sun <p> I've finally gotten around to upgrading my personal website from Sun Java System Web Server 6.1 to 7.0 - as well as a refresh of <a href="">JSPWiki</a> <p> <p> As I have been doing the migration, one of the things I ran across was a method I have for forcing a URLs to the SSL/HTTPS port of the web server. For example, I have a <a href="">WebMail</a> application installed that I occasionally use. The web server is configured to serve both Plain HTTP and SSL HTTP requests. But in this case I don't want to be able to access the WebMail application over plain HTTP. How do I force SJSWS to redirect any plain HTTP requests to the SSL? Its really pretty easy. </p> <p> This works for both 6.1 and 7.0. All you really do is edit the <tt>obj.conf</tt> (or with 7.0, it might be <tt>node-obj.conf</tt>) and add a <tt>Client</tt> tag to the beginning of the "default" <tt>Object</tt> of the form: </p> <pre> &lt;Client match="all" security="false"&gt; NameTrans fn="redirect" from="/webmail" url-prefix="" NameTrans fn="redirect" from="/webmail/\*" url-prefix="" &lt;/Client&gt; </pre> <p> What this is saying is, for any request that is over plain HTTP (security="false") evaluate, the included <tt>NameTrans</tt> parameters. In this case, any request starting with <tt>webmail</tt> will be redirected to the SSL port. </p> <p> The <tt>Client</tt> tag can perform much more complex evaluations. For example, one thing that can be handy is to limit the evaluation of the inner directives to a specific virtual host. To do that, Simply add a <tt>urlhost</tt> attribute: </p> <pre> &lt;Client match="all" security="false" urlhost=""&gt; NameTrans fn="redirect" from="/webmail" url-prefix="" NameTrans fn="redirect" from="/webmail/\*" url-prefix="" &lt;/Client&gt; </pre> Deploying JSPWiki on Sun Java System Web Server 7 mock Mon, 25 Sep 2006 17:13:15 +0000 Sun <p> Sriram and Marina's article on <a href="">Deploying Wikis to Sun Java System Web Server 7.0, Part 1: JSPWiki</a> reminded me that I wanted to write a few things about my experiences with deploying JSPWiki on SJSWS 7.0 </p> <p> The article does a good job at explaining the basics of installation and deployment. SJSWS 7 really makes it pretty easy to deploy web applications. </p> <p> The tougher part is how to configure the JAAS components in JSPWiki with SJSWS. Actually once you get your bearing with SJSWS, its not really that difficult at all. There are two components to JAAS that you need to deal with. These are the <i>JAAS login configuration</i> and the <i>Java 2 security policy</i>. And here is how to do it. </p> <h3>JAAS login configuration</h3> <p> SJSWS comes with a login configuration already installed. The trick is to find the configuration file and add the JSPWiki configuration to it. The file is called <tt>login.conf</tt> and it is located in the <tt>config/</tt> folder of the web server instance. The contents look something like </p> <pre> fileRealm { required; }; ldapRealm { required; }; solarisRealm { required; }; nativeRealm { required; }; </pre> <p> What needs to be done is simply add the JSPWiki configuration to the end of the file. The JSPWiki configuration typically looks like </p> <pre> JSPWiki-container { com.ecyrd.jspwiki.auth.login.WebContainerLoginModule SUFFICIENT; com.ecyrd.jspwiki.auth.login.CookieAssertionLoginModule SUFFICIENT; com.ecyrd.jspwiki.auth.login.AnonymousLoginModule SUFFICIENT; }; JSPWiki-custom { com.ecyrd.jspwiki.auth.login.UserDatabaseLoginModule REQUIRED; }; </pre> <p> Just add those lines to the end of <tt>login.conf</tt> and you are done. </p> <h3>Java 2 security policy</h3> <p> JSPWiki uses a standard Java 2 security policy to control access to one or more JSPWiki instances within a servlet container. The policy file comes with JSPWiki is <tt>WEB-INF/jspwiki.policy</tt>. </p> <p> Now, in my observation, JSPWiki will find this file without doing anything, so this part is somewhat optional. However, without doing any explicit configration you will generally see the error messages: </p> <pre> WARN com.ecyrd.jspwiki.auth.PolicyLoader - You have set your '' to point at '/opt/sun/SUNwbsvr7/ https-node/config/file:/opt/sun/SUNwbsvr7/https-node/web-app/node/wiki/WEB-INF/jspwiki.policy', but that file d oes not seem to exist. I'll continue anyway, since this may be something specific to your servlet container. Just c onsider yourself warned. WARN com.ecyrd.jspwiki.auth.PolicyLoader - I could not locate the JSPWiki keystore ('jspwiki.jks') in the same direc tory as your jspwiki.policy file. On many servlet containers, such as Tomcat, this needs to be done. If you keep hav ing access right permissions, please try copying your WEB-INF/jspwiki.jks to /opt/sun/SUNwbsvr7/https-node/config/fil e:/opt/sun/SUNwbsvr7/https-node/web-app/node/wiki/WEB-INF </pre> <p> The bit about "Consider yourself warned" is a little dubious. And the fix is pretty easy. So why not just do it. </p> <p> What needs to be done is to explicitly set the property <tt></tt> for the container to the location of the <tt>jspwiki.policy</tt> file. This can be done easily in the administration console. Simply log in to the administration console and drill down into <i>Configurations -> node -> Java</i>, then click on <i>JVM Settings</i>. </p> <p> The first section you will see on this page is a <i>JVM Options</i> table, which will include, among other things, a reference to the <tt>login.conf</tt> file that we tweaked earlier. Simply click on <i>New</i> to add another value, and enter a value like </p> <pre> </pre> <p> Then click <i>Save</i> and Deploy your configuartion as described in Sun Developer Network article and you are golden. </p> Upgrading Bit Torrent mock Thu, 13 Jul 2006 16:02:36 +0000 Sun <p> The week before the break, Alon R at Azureus Inc. came in to help us redo our Bit Torrent servers. It all went pretty well. Working together it took us only a couple of days to implement a more simplified seeder and tracker network at Sun. </p> <p>The main goal that we were trying to accomplish was to simplify authoring/management of torrents. I have described the <a href="">original implementation</a> a while ago. The authoring piece was a mess, because it required remote displaying Azureus from a remote data center. All in all the process was perhaps 15 steps. Not something that could be handed off to a support team very easily. </p> <p>The other major requirements of the environment included</p> <ul> <li>High Availability - both the seeders and more importantly the tracker needed to run on multiple machines and fail over automatically.</li> <li>Geographical Location Screening - (aka rDNS) we needed to make sure that people from embargoed countries were denied access.</li> <li>Simple remote administration - the most preferable would be the web</li> </ul> <p> All in all it went together pretty quickly. I threw a couple of curves at Alon, most notably, being able to fail over the tracker to a secondary box, but he was able to get patches from the developers overnight to allow for it. </p> <p> We did encounter a strange problem when we attempted to fire up Azureus on the first server, we received a ton of Exceptions with stacktraces of the form </p> <pre> Invalid argument at Method) at at at at at </pre> <p> We we're both perplexed. I know I had run Azureus on Solaris 10 before and not seen the problem. And it was a new one to Alon as well. I decided to look on <a href=""></a> for lack of anything else to try. Low and behold, I found the problem. The bug <a href=""><tt>6322825</tt></a> <tt>Selector fails with invalid argument on Solaris 10</tt>. And the bug had a work around. Just set the maximum number of file descriptors on the process over 8K. Once I did that, it fired up like a charm. </p> <p> After that, the only other major task was to hook into our rDNS provider. From my description, Alon thought we could write a plugin, and again, talked to the developers who provide us a shell of a plugin. All I needed to do was add about 10-20 lines of code which actually did the screening query with the provider. </p> <p> In the end, the network consists of three servers, a Master Tracker and two Seeders. </p> <p> The Master Tracker is configured to watch a particular folder for new and removed files. When it finds changes, it automatically creates torrents for the files and starts seeding them. It also copies the <tt>.torrent</tt> files to a separate folder within the web servers docroot for customers to access. </p> <p> The two Seeders, watch an RSS feed provided by the Master Tracker. The RSS feed contains a list of all the torrents the Master is seeding. When the seeders detect a change to the RSS feed, they act appropriately. If its a new torrent, they download it from the Master and start seeding it. If if a torrent is removed, they remove it from themselves as well. </p> <p> Now, authoring a torrent is a easy a rsync'ing, scp'ing or sftp'ing a file over to the Master Seeder and voila. </p> Slightly painful migration to ZFS mock Thu, 13 Jul 2006 15:17:36 +0000 Sun <p> Fresh back from the break I decided to give and upgrade to Solaris 10_U2 a go, and migrate my data to ZFS (now included w/ U2). </p> <p> The ZFS migration was slightly painful so I figured I'd post my experience, in case anyone else might want to attempt it. </p> <p> Actually migrating the data itself to a ZFS partition was simple, however I wanted to mirror the data as I have been doing with UFS/SVM which caused the problem. Apparently logged as a known bug: </p> <pre> 6355416 zpool scrubbing consumes all memory, system hung </pre> <p> Even though the system was unresponsive, I decided to let it do its thing overnight and it did eventually finish resyncing/mirroring, and now everything is fine. </p> <P>So for what its worth, here it is. The system involved was a SunBlade 2000 with 3G of RAM. <p>The goal was to combine three separate UFS partitions (<tt>/app, /work, /extra</tt>) into a single ZFS pool which would then host all those partitions again. The partitions would then use portions of the pool as need be, precluding the need to set arbitrary sizes as has been needed with UFS. </p> <p>Also, the partitions were mirrored, and I wanted to continue mirroring with ZFS. </p> <p>The three partitions are </p> <pre> /dev/md/dsk/d4 90030867 30328334 58802225 35% /extra /dev/md/dsk/d5 24795240 23208057 1339231 95% /app /dev/md/dsk/d6 20646121 16648727 3790933 82% /work </pre> <p>(Notice that I was bumping up against some partition limits for a couple of the filesystems. The primary reason I wanted to convert to ZFS was to just have a big pool of space which could be used by whoever needed it.) <p>With the following SVM meta partitions assigned </p> <pre> d4 had submirrors d40 (c1t1d0s4), d41 (c1t2d0s4) d5 had submirrors d50 (c1t1d0s5), d51 (c1t2d0s5) d6 had submirrors d60 (c1t1d0s6), d61 (c1t2d0s6) </pre> <p>The method I went through was </p> <ul> <li>break the mirrors on one disk </li> <li>combine the partitions </li> <li>create a ZFS pool from the combined partition </li> <li>migrate the UFS/SVM data to the ZFS pool </li> <li>delete the remaining UFS/SVM partitions and combine them </li> <li>attach the combined partition as the second side of the mirror </li> </ul> <p>Here's the procedure. </p> <p><b>Break the mirrors on one disk</b> </p> <pre> metadetach d4 d41 metaclear d41 metadetach d5 d51 metaclear d51 metadetach d6 d61 metaclear d61 </pre> <p><b>Remove the meta db from the disk</b> </p> <pre> metadb -d /dev/dsk/c1t2d0s7 </pre> <p><b>Combine the disk partitions</b> </p> <p>I used the <tt>partition</tt> commands in the <tt>format</tt> utility to change the c1t2d0 disk. </p> <p><b>Create a ZFS pool called <tt>storage</tt> and filesystems within it</b> </p> <pre> zpool create storage c1t2d0s4 zfs create storage/extra zfs create storage/work zfs create storage/app </pre> <p><b>Migrate the UFS/SVM data to the ZFS filesystems</b> </p> <pre> cd /work find . -depth -print | cpio -pdmv /storage/work cd /app find . -depth -print | cpio -pdmv /storage/app cd /extra find . -depth -print | cpio -pdmv /storage/extra </pre> <p><b>Unmount the UFS/SVM partitions</b> </p> <pre> unshareall umount /app umount /work umount /extra </pre> <p><b>Remount the ZFS partitions where I expect them</b> </p> <pre> zfs set mountpoint=/work storage/work zfs set mountpoint=/extra storage/extra zfs set mountpoint=/app storage/app shareall </pre> <p><b>Remove the remaining SVM meta partitions</b> </p> <pre> metaclear d4 d40 metaclear d5 d50 metaclear d6 d60 </pre> <p><b>Remove the meta db from the disk</b> </p> <pre> metadb -d /dev/dsk/c1t1d0s7 </pre> <p><b>Combine the disk partitions</b> </p> <p>I used the <tt>partition</tt> commands in the <tt>format</tt> utility to change the c1t1d0 disk. </p> <p><b>Finally, add the combined partition as a ZFS mirror</b> </p> <p>This was the painful step. Apparently ZFS decided to take all available CPU and memory in the system. The system became unresponsive. Apparently ZFS is a little to aggressive with its resyncing (they call it scrubbing/resilvering). </p> <p>I found bug <a class="external" href="">6355416</a><img class="outlink" src="images/out.png" alt="" /> which appears to describe the issue. </p> <p>I would recommend before issuing this last command, that you boot the system into single user mode, and kill and processes that you don't need. </p> <pre> zpool attach storage c1t2d0s4 c1t1d0s4 </pre> <p>The system will probably become unresponsive within a few minutes. If so, just walk away and let it do its thing. You can use the <tt>zpool status</tt> command to check the progress of the resilver. </p> Remote Monitoring hack for JMX mock Fri, 26 May 2006 17:07:20 +0000 Sun <p> Quite some time ago when a team of us worked on rewriting the code which ran the Java Developer Connection, a co-worker came up with the idea of a simple instrumentation interface.&nbsp; It was almost a pre-cursor to JMX. I'm sure other groups inside and outside of Sun had done the same thing. The nice thing about the framework we developed is that it was extremely simple, 'course this was around the time of JDK 1.2 so keeping the code simple was a good thing. </p> <p> The instrumentation framework was invaluable for us as we worked through code iterations and daily support on the JDC. It provided a multi-level view of all of the components of the system, and helped us monitor things such as how many sessions and users are logged into the JDC, as well as help us resolve problems such as the culprit of a JDBC Connection Pool leaks. </p> <p> Since then, I have always been keen on pushing fellow engineers to instrument their applications. Of course, now days I've been pushing JMX as the preferred instrumentation solution. It provides many of the same features that our home grown solution provided. And what's better is that because its the blessed Monitoring and Management standard for Java, more and more applications are being instrumented with JMX, and even the newer JDKs. </p> <p> The only problem that had been unresolved for us is remote monitoring of JMX enabled applications. When I first looked into it a year or two ago, the stage was sort of dim. Although JMX does have remote monitoring capabilities, its based on RMI, which was unfortunate. </p> <p> Its unfortunate because RMI is somewhat of a dynamic protocol, similar to SunRPC, where ports which services reside are dynamically allocated at runtime by the RMI Registry. This poses a problem when trying to connect to a remote data center, with umpteen firewalls to get through. Security didn't appear to be of much concern to the original RMI developers. </p> <p> So I decided to abandon the use of RMI back then, and settled on a plain old HTTP based console embedded as part of the application. That sounds like a lot, however, looking around JMX implementations I found <a href="" target="_self">MX4J</a> had this built into their framework. And in fact, the HTTP interface was quite reasonable. And that's what we went with. </p> <p> Recently, I took another look at the whole problem of remote management. After some poking around I found a method of using RMI with JMX in a more "secure" way. That is, configuring RMI and JMX to use well defined ports for communication, which makes me happy, and my Operations people (who manage the firewalls) happy as well. </p> <p> So here's the deal. There's really only two ports involved. The RMI Registry, which is already on a well-known port, typically 1099, and the JMX RMI Connector, which is the problem. </p> <p> The solution lies in the fact that when creating a JMXServiceURL to define the connection parameters, you can, in fact, specify a port number for the service. This is not clear at all from the <a href="" target="_self">documentation</a>. Here's an example </p> <pre> String svc = "service:jmx:rmi://apphost:9994/jndi/rmi://apphost:1099/connector"; JMXServiceURL url = new JMXServiceURL(svc); RMIConnectorServer rmiServer = new RMIConnectorServer(url, null, mbeanServer); rmiServer.start(); </pre> <p> The key is really the first two lines. What its saying is, the use RMI Registry is on&nbsp; "apphost:1099" and register a service called "connector" which is located at "apphost:9994".Now, All I really have to do is tell my Operations folks to open the firewall to ports 1099 and 9994. After that's done, remotely monitoring the application is as simple as running the jconsole command: </p> <pre> jconsole service:jmx:rmi://apphost:9994/jndi/rmi://apphost:1099/connector </pre> <p> How cool is that. And all of a sudden this becomes just the tip of the iceberg. If I don't want to use the standalone RMI Registry, or I want to run an embedded registry within the application, well then all I have to do is instantiate one with the desired port number: </p> <pre> rmiRegistry = LocateRegistry.createRegistry(4099); </pre> <p> One slightly more complete final example to wrap it all up. The nice thing about JMX is that you can start out simple. For example, at the very least, just enabling JMX to monitor what the JVM is doing.&nbsp; That can be accomplished with just a couple more lines of code. The key is getting a handle to the Platform MBean Server: </p> <pre> Registry rmiRegistry = LocateRegistry.createRegistry(4099); MBeanServer mbeanServer = ManagementFactory.getPlatformMBeanServer(); String svc = "service:jmx:rmi://apphost:9994/jndi/rmi://apphost:4099/connector"; JMXServiceURL url = new JMXServiceURL(svc); RMIConnectorServer rmiServer = new RMIConnectorServer(url, null, mbeanServer); rmiServer.start(); </pre> <p> Doesn't get much easier than this. </p> Bit Torrent file distribution at Sun mock Wed, 26 Apr 2006 17:21:54 +0000 Sun A while ago I put together a Bit Torrent solution for the OpenSolaris folks. They wanted as many download options as possible for the OpenSolaris bits. As is typical, we only had a couple of weeks to select technologies and deploy a solution. Not a problem, I thought, I had some familiarity with Bit Torrent so it seemed like it would be pretty straight forward.<br><br>What we eventually decided on was a solution using a tracker called <a target="_self" href="">BNBT</a>, and the <a target="_self" href="">Azureus</a> bit torrent client. It all went together pretty easily however there were a few twists that we had to acaccommodatethe biggest of which was being able to restrict access to the torrents to users in non-embargoed countries. <br><br>For that, we decided on using a product called <a target="_self" href="">NetAcuity</a>. The product has multiple language bindings including C and Java (which is great because BNBT is C++ and Azureus is written in Java). The software itself is fast and stable. The key was finding the right place in both BNBT and Azureus to interpose an IP Address check and permit or deny access.<br><br>The deployment environment presented a bit of a challenge as well. Anyone who's used Azureus knows that the main interface is a GUI, but GUIs aren't permitted in our hosting environment.. Its a pretty secure environment, Solaris is trimmed down considerably, including the removal of all of the X/Openwin/GNOME software. <br><br>However, after some plugging around on the Azureus Wiki, I found that an alternate TTY &amp; Telnet interface was being worked on. And I had already known about the more established Swing WebUI. The TTY interface was fairly rudimentary at the time, but it was enough to allow it to be started and stopped without the need of an X server somewhere. <br><br>So that only left a model for torrent authoring. We devised a method that, although somewhat twisted, worked for low volume authoring. The method involved starting up a master or uber-seeder torrent client, Azureus, and remotely displaying it back to a desktop using the port forwarding capabilities of SSH. Within this master, the torrent file was created and the master started seeding the thtorrentThe torrent file would then get loaded by hand using the Swing WebUI into the seeders that we set up, there are four of those. After the torrent was downloaded and being seeded by all of our seeders, the master/uber-seeder could then be shut down until the next new torrent.<br><br>At the end of the day, it all has been working pretty well. At least for a first phase. But recently we started to look at how we could improve and streamline the framework. Particularly around torrent authoring.<br><br>So last week, we got together with a few folks from <a target="_self" href="">Aelitis</a>, the main developer of Azureus. We described our environment and asked how it could be improved. They came back with a wealth of information on how to automate the process much better. There are some new features and UIs available that would help us greatly, including both an HTML WebUI and an RSS feeds. The method they suggested was to still create an master seeder, but enable a shared folder feature of Azureus which provides a way for it to automatically pick up new files in the shared folder and create torrents out of them. From there, the you would then point all the regular seeders to the RSS feed on the master seeder. They would periodically read the RSS feed, identify new torrents and automatically download and start seeding them.<br><br>Great stuff, that will go a long way to simplifying our torrent offering and provide a more scalable solution. <br> +1 for Synergy mock Sun, 29 Jan 2006 12:26:07 +0000 Sun Ok so I know its been blogged about a least a couple of times on <a target="_self" href="../../"></a>. Both <a target="_self" href="../page/yongsun?entry=software_dual_head">Yong Sun</a> and <a target="_self" href="../page/richb?entry=synergy">Rich Burridge</a> have commented about it, but this is cool enough I have to repeat it just again.<br /><br />But I finally got around to trying <a target="_self" href="">Synergy</a> after hearing about it from co-worker Kenny by way of <a target="_self" href="../../rama">Rama</a>, and the virtual keyboard/mouse software is pretty dang cool. I've got a Asus desktop running Windows XP connected to Acer Ferrari laptop running Solaris Nevada 29, and I'm able to control the laptop from the PC. <br /><br />Incredibly cool. Now I don't have to hunch over anymore while I work on the laptop. <br /><br /> Tinkering with JFreeChart mock Wed, 4 Jan 2006 11:37:17 +0000 Sun Recently I've been putting together some scripts to extract performance data from various log files on a particular cluster of <a target="_self" href=""></a> servers - there have been intermittent performance issues and I have been tried to zone in on the source.<br /><br />At this point, I have scripts that will record vmstat data, web page accesses and errors, as well as detailed GC data from the application server JVM and load them into a database.<br /><br />What I had been lacking was a way to turn that data into some pretty graphs. For a few days, I had just been building a CSV file from the data and loading it into StarOffice to produce a graph. But that quickly became pretty tedious. So I started looking at packages out there which might help.<br /><br />All I really wanted to do was graph some simple time-series charts for various data points, for&nbsp; example, CPU usage at 15 minute intervals for each day. After a little bit of searching I came across <a target="_self" href="">JFreeChart</a>, and after looking at the demo page,&nbsp; JFreeChart seemed like it would fit the bill, so I decided to take a closer look. <br /><br />Unfortunately finding information on how to use the API was a little challenging. The JFreeChart developer's have an interesting business model where the API is free, but you have to buy their book to understand how to use it. That's fine, I can go ahead and&nbsp; buy the book, but I want to start tinkering NOW.<br /><br />I did find a couple of articles which got me started in the right direction. A Javaworld article titled <a target="_self" href=""> Chart a new course with JFreeChart</a> which provided some snippets of code on how to create various types of charts. And I also found a DevChannel article titled <a target="_self" href="">Data driven graphics with JFreeChart</a> which provided a complete sample program.<br /><br />From these articles I was able to pretty quickly put together some utilities that read data directly from the database I have my profile data in and graph them out in various ways.&nbsp; Here's an <a href="../resources/mock/gc-time-star-fep4-2005-12-10.png" target="_self">example chart</a> from one of the more interesting days when the garbage collector got a little jiggy with it - still trying to figure out what when on there, the JVM did seem to recover the next day which is a little interesting.<br /><br />Anyways, anyone looking for some graphing/charting software for Java, I'd definitely recommend taking a look at <a target="_self" href="">JFreeChart</a>.