Software Archaeology Techniques

by James Shaw 6. November 2009 04:42

Listened to a good podcast on techniques reading other's code.  

Reminds of my last project of adding a feature to a partner company's C code with lots of dependencies on libraries tied to the OS (Linux), no tests, no debugger working, and no documentation :).

Key take-aways for me from the podcast:

1. Start small and look at 5-20K lines of code for practice (e.g., open source projects). 

2. Know the environment (e.g., read the source code of the Ruby interpreter if you're programming Ruby).  Beware of evil wizards (e.g., understand the code generated by IDE wizards).

3. Use version control to modify the code you're reading to play around with it.  Use VM for playing with code that's tied to the OS (like the project I had). 

Tags:

Tech

C# is more advanced than Java, but...

by James Shaw 7. October 2009 23:49
FYI: My comment to the blog post C# now a better language than Java:
 
Interesting discussion here.  I've had about 7 yrs programming in Windows/.Net and 3 yrs in Java experience, and I agree that C# as a language has more than caught up with Java, and become more advanced and convenient to use, esp Linq.  After using Linq for everyday tasks like filtering collections of objects or parsing XML, it's hard to go back to the "old" way of doing things; it's just more expressive and better for the VM too as language becomes more declarative for the CLR to optimize in the future (parallelize behind the scenes for example).  New features coming out in C# 4.0 like writing dynamic language directly in C# and integrating with the static part just shows that the MSFT folks are constantly improving the langauge given the trends.

 

On the other hand I also agree that JVM is much more portable and in use than CLR, and I hope that MSFT realizes the conflicting goals of having the most popular developer platform and tying companies to Windows.  After dabbling in both open source tools and MSFT tools, MSFT developer tools are indeed the best (Visual Studio beats out Eclipse by wide margin); it'll be a shame for MSFT to continue to keep those superior tools and languages behind the walls of Windows platform.  Visual Studio should've beeen cross-platform like MonoDevelop.  It would be interesting if in the future the Mono implementation becomes more popular than MSFT's implementation.  I totally salute the efforts of the Mono folks and think that MSFT should've been the one pushing for the effort and making the entire .Net more open and license free (not just the run-time and the language but all the frameworks as well).  

 

The macro trend is for technologies to be more open, so that businesses collaborate with a fraction of the effort on the technologies (e.g., Linux kernel) than if they were to implement the whole thing themselves (like what MSFT is doing), and also benefit from having a stake and say in the direction of the technologies rather than being dictated and tied to a specific vendor.  That's why Unix/Linux is so popular with businesses, from the small device (iPhone, Android, Palm Pre) to desktop (like Mac OS and Ubuntu) to server.

 

I have to check out Scala given your high opinion :).  I heard twitter rewrote their Ruby code to Scala for performance.  One thing I wonder about these other JVM language is performance, whether the community is able to make it keep up with Java (even though all compile down to Java byte code, I assume the compilation itself could have different efficiencies).  

Tags: , , , , ,

Tech

Running VirtualBox on a Windows Vista bootcamp partition in Mac OS X

by James Shaw 26. August 2009 11:51

VirtualBox is an open-source virtualization software from Sun much like VMWare and Parallels, but free for personal use!  I just tried it out on my Mac OS X to run a Windows Vista VM on a physical Windows Vista partition created through bootcamp, and thought I'd share my experience (and to write down for myself for future reference :)).  

I had googled and found someone doing very similar thing on a Ubuntu machine.  Most of the info is pretty relevant on a Mac OS X as well, except that I couldn't use the solution of assigning my user account to the "disk" group; there's no "disk" group on Mac OS X.  The other solution mentioned of changing permission works however.  Here's exactly what I did:

0. Read the relevant part of the VirtualBox User Manual regarding using raw partition ("9.10.2 Access to individual physical hard disk partitions").

1. Locate the device and partition holding your Windows Vista partition (e.g, using "df -k" command):

jshawsworlds-macbook:~ jshawsworld$ df -k
Filesystem    1024-blocks     Used Available Capacity  Mounted on
/dev/disk0s2     73269248 31999364  41013884    44%    /
devfs                 110      110         0   100%    /dev
fdesc                   1        1         0   100%    /dev
map -hosts              0        0         0   100%    /net
map auto_home           0        0         0   100%    /home
/dev/disk0s3     82685740 56175456  26510284    68%    /Volumes/Untitled
/dev/disk1s1        15632      144     15488     1%    /Volumes/CANON_DC
/dev/disk2s1    334784000 46614707 288169292    14%    /Volumes/My Book
/dev/disk2s2    153600000 29766376 123833624    20%    /Volumes/Mac OS Partition 2

For me, for example, I know the Windows partition is /dev/disk0s3.

2. Back up your existing Windows Vista partition, to a external drive for example, since you might damage the Windows paritition in this exercise and render it unusable!  One free tool I find pretty reliable in backing up and restoring Windows partition is winclone.

3. Change to root to do the next few commands:

sudo su

You may need to enable root user by going to the "Directory Utility" app, click on the lock icon, and then Edit->Enable Root User.

4. Unmount the Vista partition in Mac OS X if you currently have it mounted, since we don't want VirtualBox accessing it at the same time that Mac OS is accessing it:

umount /dev/disk0s3

and make sure it has been unmounted (i.e.,  the directory /Volumes/Untitled is empty in my case).

5. Change the device to read/write, which is needed by VirtualBox as stated in the user manual:

chmod 666 /dev/disk0*

Note that you have to change permission for all of disk0, not just the Windows parition (hence the "disk0*"), since createrawvmdk needs to access disk0 as well as disk0s3.

6. Run the command

VBoxManage internalcommands createrawvmdk -filename /Users/jshawsworld/WinVista.vmdk -rawdisk /dev/dev0 -partitions 3 -relative

7. Finally change the owner to your user account for the vmdk file you just created as root, so you can access it as yourself when running VirtualBox later:

chown jshawsworld /Users/jshawsworld/*.vmdk

8. Run VirtualBox from Finder, and browse to the vmdk file. 

9. Make sure that "IO APIC" and "VT-x/AMD-V" in your Virtual Machine Settings are both on, and then start the VM.  It should start.

Even though I got it working, it was painfully slow..., even after giving it 2 GB of memory.  I had tried VMWare Fusion to do the same thing before, and remember it wasn't any faster.  To really work efficiently in Windows I think booting to it via bootcamp is still the way to go.  XP might be faster; who knows maybe Windows 7 will be much faster.  For light-weight OS's (like Linux) running it as a VM is definitely the way to go.

Thought on the economy

by James Shaw 31. March 2009 16:33

Read the following news:

http://www.bloomberg.com/apps/news?pid=newsarchive&sid=atJq2AOruM1M

 

Couple of thoughts on the article:

 

- Larry Summers said now's NOT the time to have "less demand in America...There’s no place that should be reducing its contribution to global demand right now." I'm no economist but it makes me wonder how you can ever get out of a hole by digging yourself deeper.  It's illogical to me that as indebted as we are, we can just spend more and more and prosperity will come back. Like a family that already has a pile of debt, how can you get out of debt by getting into more debt? To get out of debt you cut costs and save, and the time to do that is immediately.  

 

- Countries like China that have saved are the ones that can afford to spend, not the US. China's being criticized for not spending enough, esp on its social security programs (75% of its stimulus is on infrastructures). To me they're being very careful with their money. Rather than bailing out failing companies they're spending money on stuff that will help them increase their productivity once the world economy gets better and demand is back, which is smart.

 

I'm also a little skeptical of government-implemented social security program; we've got one here in the U.S., but my generation can totally not depend on it. We still have to save for retirement ourselves. Maybe rather than counting on the government for retirement and other entitlements (other than protection again the unforeseen like health-care and unemployment), China's (granted it's a little too light on the entitlements) way is better in that it keeps citizens on their toes to save, and those big savings in the banks in turn generate the credit that fuels the economy. That's a healthy economy, not one based on borrowing and consumption.

 

Obama talks about the need to to get credit flowing again in this country, but true credit comes from savings, not from the central banks' printing money... So rather than spending, instead, as indebted as we are, we should save our way back to a sound economy. The alternative (borrowing more from foreigners and printing money, which is the action taken right now), will just keep us in this phantom economy that'll only last as long as the dollar doesn't collapse..., and there are already signs of it.

   

 

Beautiful Code

by James Shaw 13. March 2009 22:07
When coding recently, I found myself greatly appreciating C#'s rich features that reduce the number of lines of code written.  Let me elaborate.

There has been so many times when I find myself wanting to access a list of objects randomly, based on some field of that object.  To do it efficiently I'd want to use a dictionary to to access these objects, with the key being the field I want to find the object by.  This need happens so often with lists of different types of object (say from 3rd party API's you use) that it's best to write a generic helper function that can wrap a list of any type of object with a dictionary and returns that dictionary, rather than writing a helper function for list of each type of object.  The following is such code, using generics and reflection in C#:
 
    public class Util
    {
        ...
        /// <summary>
        /// generic method that wraps the given IEnumerable into a IDictionary with the given
        /// field (of the objects in IEnumerable) as the key.
        /// </summary>
        /// <typeparam name="KeyType"></typeparam>
        /// <typeparam name="ValType"></typeparam>
        /// <param name="src"></param>
        /// <param name="keyFieldName"></param>
        /// <returns></returns>
        public static IDictionary<KeyType, ValType> ToDictionary<KeyType, ValType>(IEnumerable<ValType> src,
                                                                                   string keyFieldName) 
            where KeyType : class
            where ValType : class
        {
            var ret = new Dictionary<KeyType, ValType>(src.Count() );

            foreach (ValType ele in src)
            {
                var key = typeof(ValType).GetProperty(keyFieldName, typeof(KeyType) ).GetValue(ele, null) as KeyType;
                ret[key] = ele;    
            }

            return ret;
        }

    }
 
With the above you can access a list of any objects like this:
 
            var attendeesById = Util.ToDictionary<string, Attendee>(attendees, "Id");
 
Here for example it assumes Attendee class contains an "Id" string property.  You can use Util.ToDictionary with IEnumerable of any other types of objects. 

Another good use of generics to reduce code I had was when parsing XML; when parsing XML frequently you want to parse an element's value to a .Net type, using int.parse() for example, if the element is supposed to contain an int.  Well I would just use int.parse() to do the job except that sometimes the element can be optional, so it's again nice to have a generic helper method that takes in an XML element and returns the appropriate type after parsing, or null if it's optional.  For example, when using the following code to parse XML,
 
    WebRequest httpRequest = HttpWebRequest.Create(url);
    using (var httpResponseStream = httpRequest.GetResponse().GetResponseStream())
    {
        var xmlRoot = XElement.Load(XmlReader.Create(httpResponseStream));
        var xmlNamespace = XNamespace.None;
        ret = new List<Event>(from result in xmlRoot.Descendants(xmlNamespace + "item")
                                    select new Event()
                                    {
                                        Id = XmlUtil.GetXElementValue(result.Element("id"), true),
                                        Name = XmlUtil.GetXElementValue(result.Element(xmlNamespace + "name"), true),
                                        Description = XmlUtil.GetXElementValue(result.Element(xmlNamespace + "description"), false),
                                        VenueName = XmlUtil.GetXElementValue(result.Element(xmlNamespace + "venue_name"), false),
                                        VenueLatitude = Util.ParseOptional<double>(XmlUtil.GetXElementValue(result.Element(xmlNamespace + "venue_lat"), false)),
                                        VenueLongitude = Util.ParseOptional<double>(XmlUtil.GetXElementValue(result.Element(xmlNamespace + "venue_lon"), false)),

                                        ... 
                                    });
 
The following helper methods in Util come in handly, esp Util.ParseOptional, which is shared code to parse any type that I want to parse out of XML and that has a parse method.  The more shared code (i.e., the less code a developer writes) the less chance for bugs :).
 
    public class XmlUtil
    {
        ...
        public static string GetXElementValue(XElement xe, bool required)
        {
            string ret = null;

            if (xe != null)
            {

                if (xe.Value != null)
                {
                    ret = xe.Value.Trim();
                }
                else if (required)
                {
                    throw new Exception(xe.Name + " should contain a value");
                }
            }
            else if (required)
            {
                throw new Exception("required XML element is missing");
            }

            return ret;
        }

    } // XmlUtil class


    public class Util
    {
        /// <summary>
        /// generic method that parses the given string to the given type, taking into account
        /// that the given string might be null.
        /// </summary>
        /// <typeparam name="T"></typeparam>
        /// <param name="arg"></param>
        /// <returns></returns>
        public static Nullable<T> ParseOptional<T>(string arg) where T : struct
        {
            Nullable<T> ret = null;

            if (!string.IsNullOrEmpty(arg) )
            {
                var methodInfo = typeof(T).GetMethod("Parse",
                                                     new Type[] { typeof(string) });
                ret = methodInfo.Invoke(null, new object[] { arg }) as Nullable<T>;

            }

            return ret;
        }
 
 In summary, I find .Net's generics and reflection features very helpful in abstracting functionality further and reducing even more lines of code written.  The other features demonstrated in the snippets like Linq to Xml, variable type inference (the "var" keyword), and object initializer (the ability to initialize properties in constructor) also help a lot in writing less code, and really speed up development.  I recently had to switch coding between C# and Java, and what a difference it was without them: the Java code is a lot more lines (and thus more chance for bugs) and you just end up typing more.  
 
Java was a revolutionary language (and I learned it before C#) but it seems like in the evolution of languages C# has surpassed it (at least in terms of the core language features).  

Short ETF's a better way to short?

by James Shaw 20. January 2009 21:22

Shorting a stock can be a risky thing to do: unlike with a normal purchase of a stock where the most you lose is the original purchase price of the stock, the loss from shorting a stock can be unlimited should the market move against your prediction, during which time you'd suffer the additional agony of margin calls from your broker to cover your shorts, making it more expensive and psychologically debilitating to wait out the market to move in your direction.  Yet shorting can make very good sense at times, like the financial meltdown that occurred late last year which affected tons of the financial stocks; it would have been very rewarding to have shorted those stocks.  How do you play shorts while minimizing the drawbacks of shorting? 

 

I found that the short ETF's can be a good way to minimize the drawbacks of shorting.  ETF's (Exchange Traded Funds) are a financial instrument that holds other assets (like a mutual fund) to accomplish a very explicit, stated financial objective, like tracking an index (how well an ETF meets its stated objective depends on how well it's managed).  Short financial ETF's like SKF, for example, buys other financial instruments (e.g., derivatives like options) to seek twice the inverse of the daily performance of the Dow Jones Financial Index.  When you short using a short ETF, you buy it like a stock so your loss is not unlimited, and you have the ability to wait out market fluctation against your prediction, without the worry of margin calls :).

 

During November last year I bought and sold SKF at quite a big profit, for example; I had guessed the market right then (it was really obvious back then too with the financial stocks :)).  Later I bought SKF again, and this time I guessed wrong and it fell almost half, but because it was an ETF and not a pure short I didn't panic and was able to wait out the down-turn and sold when it came back up, and actually made a little profit too.  BTW I found technical analysis not very applicable to short ETF's (but again value investors probably don't think technical analysis works at all), so I wouldn't enter or exit based on it (exit when the profit is enough).

 

Normally I don't play shorts unless it's blatently obvious a market for shorts, and my conclusion is, if you're going to play shorts, it's safer to do it through the short ETFs; the drawback is that you can't short as granularly (you play by sector, like the financial stocks in this case, rather than a specific stock), but to me that's actually safer. 

 

P.S. This blog entry is just my personal opinion; please don't hold me liable for any financial decisions you make. 

Developing Facebook app on your local machine

by James Shaw 12. January 2009 18:23

Facebook used to let a developer specify the Callback URL to be localhost (e.g., http://localhost:56570/MyApp/) for an IFrame-canvas app such that as you develop your app, you can just test it on your local machine before pushing it out to your hoster.  That's no longer the case: you cannot specify a callback URL containing "localhost" anymore.  So how do you keep using localhost for debugging purposes during development?

Here's what I did:

1. You need to have another Facebook app, say "My App Dev", registered for development purpose (you'll need one anyways once you've released your app and need to develop incremental features). 

 

2. In this app have settings that parallel the production Facebook app except the "Post-Authorize Redirect URL".  Have the "Post-Authorize Redirect URL" point to a small HTML file (say "TestPage.htm") on your hoster that redirects back to localhost :).  Create TestPage.htm like the following:

<html xmlns="http://www.w3.org/1999/xhtml">

<head>

<title>Untitled Page</title>

</head>

<body>

<script type="text/javascript">

var newWindowLocation = "http://localhost:56570/ThePageIWantPostAuthorize.aspx" +

                                    window.location.search; // get the query string that FB tags on, including "?"

window.location = newWindowLocation; // redirect to localhost

</script>

</body>

</html> 

So your "Post-Authorize Redirect URL" would be http://apps.facebook.com/myapp_dev/TestPage.htm", for example.

 

3. Make sure in your code use the right Facebook APIKey and Secret for the right release, since now that we have 2 "Facebook apps" for the same app.  With .NET for example, you can have the following in web.config:

 

<add key="APIKey" value ="My App's key"/>

<add key="Secret" value ="My App's secret"/>

<
add key="APIKey.debug" value ="My App Dev's key"/>

<add key="Secret.debug" value ="My App Dev's secret"/>

and use compile-time macro to determine which set of values to use in your code:

#if(DEBUG)

var apiKey = WebConfigurationManager.AppSettings["APIKey.debug"];

var secret = WebConfigurationManager.AppSettings["Secret.debug"];

#else

var apiKey = WebConfigurationManager.AppSettings["APIKey"];

var secret = WebConfigurationManager.AppSettings["Secret"];

#endif

The same compile-time macro can help you pick the right set of other properties like db connection strings, since you'll probably want to use a local database instead of the one on your hoster during development. 

With the above trick, with just one additional HTML file (TestPage.htm) uploaded to your hoster and an additional Facebook app registration, you are back to your happy self debugging on your local machine :).

P.S. Obviously this trick won't work for FBML-based canvas.

Tags:

Tech

How the current (and past) financial crises are built into the system

by James Shaw 18. December 2008 07:28

My sister sent me a great video on how our current monetary system works that I encourage all of you to see (watch all 5 parts).

It explains this question of how current and past financial crises are built into the system.  The reason is that nowadays money is backed by debt, e.g., every dollar banks create is backed by someone taking out loan from the bank, at interest (and to be more precise, every dolloar is only backed by a fraction of a loaned dollar, thanks to our fractional reserve system). The fact is that the system assumes in the future there will be more debt and therefore more money to pay off not only the loans but also the interest on them; it's basically assuming an ever-growing society, in terms of  productivity, economic activities, etc, that drives the need for more loans and thereby create the bigger money pool needed to fully back existing money pool (otherwise they are just paper) plus interest.

This assumption of an ever-growing society is not sustainable (because businesses do go bust) and can be downright dangerous, for the world to keep wanting to grow, depleting our natural resources on the way, and trying to achieve growth by perhaps taking excessive risks (like the mortgages and their packaged securities in this crisis). The central banks' (the Federal Reserves in the US) tinkering with the system by printing money without backing is not really solving the problem, and actually adds fuel to the fire by adding more debt to the citizens (inflation is really another form of tax) that assumes even bigger productivity (fingers crossed) in the future to repay it.
 

This begs the question that perhaps we should change our out-dated monetary system to not back our money by debt, or make our money an interest-free instrument. 

 

China Trip 2008

by James Shaw 6. December 2008 20:04

I better blog about my trip to China in September before I forget too much Tongue out.

I went to China for 3 weeks with GAP Adventures, a travel company that I highly recommend, esp if you're single Smile.  If you travel alone, they will pair you up with another single traveller to share a room in the hotel, unlike most other travel companies I found that force single travellers to pay more to stay in a single room.  This company's got trips to everywhere, even space!  The particular one I went on is called "China Experience".  Here's the people I travel with:

The places we covered include (in the order visited): Shanghai, Wuzhen, Hangzhou, Yangshou, Kunming, Chengdu, Lhasa, Xian and Beijing.  It was a lot of ground, and we not only experienced China as a tourist, but also as people livingthere, e.g., riding on the train in the same crowded section asordinary Chinese, visiting the country-side as well as the urban areas,etc. It was more physically engaging than just walking around (biked alot).  You can check out my photos with comments on Facebook; my fellow traveller Lars also took some pretty artistic pictures of the trip.

My favorite city out of the whole trip is Kunming in Southwestern China.  It's not a top tier city like Beijing or Shanghai, but the weather is a lot nicer, and it also seems less polluted.  It's in fact much like San Diego: dry and not humid, unlike most other Chinese cities.  Tibet also is also an interesting experience, simply because it's so different from the rest of China.  My overall impression is that some places of China are naturally gorgeous, some monuments are quite grand (see Leshan Buddha and the Great Wall in the pictures), some places are quite modern, but some places are also quite rural.  It's really a place of contrasts.

There are also cultural differences that took some getting used, even for me who came from Taiwan.  People spitting is common.  Everything is negotiable like the price of the stuff that stores/vendors sell.  At first I was quite embarassed to bargain when I buy stuff, but later I was told by a local that I should start bargaining at 30% of the initial price, and pay no more than 50%, otherwise I overpay.  It's really a good experience to bargain, since you'll need to learn to negotiate with people in life sooner or later, might as well start with the small stuff. 

It was funny that in Tiananmen Square I was approached by a lady that tried to sell me a hat for 15 yuan.  I said I'd only pay 5 yuan for it.  She said it cost her 8 yuan, and walked away pretty offended.  A while later I was about to leave, she came back herself, and sold it to me for 5 yuan, but still seemed pretty offended and kept saying I got a steal. 

Also people there can be quite pushy sometimes.  I was annoyed by a masseuse that kept trying to upsell me services, and a KTV bar that was pretty unscrupulous on their billing.   You just have to be very careful.

One thing I can get used to is that tipping is not customary at most places Tongue out.  I like the idea that good customer service is just a natural expectation of a business, rather than something you pay for; in most countries in Asia tipping is not customary.  I was also impressed by how they in general really work it to earn their money, e.g., store clerks doing their best to sell you stuff vs in the US those clerks just stand there.

China seems very dynamic with people hustling and bustling all the time to better their lives.  I think Jim Rogers is right when he said that "If you were smart in 1807 you moved to London, if you were smart in1907 you moved to New York City, and if you are smart in 2007 you moveto Asia."

 

McCain and Obama's tax policies

by James Shaw 27. October 2008 10:26

One obvious big factor in deciding which Presidential candidate to vote for is their tax policies, i.e., how will they affect my income.  The following is what I gathered from a tax newsletter that I subscribe to:

John McCain
• Repeal AMT for individuals but not businesses (corporations).
• Double the personal exemptions from $3,500 per person to $7,000.
• Raise the estate tax exemption to $5 Million and cut the estate tax to 15%, but not repeal the estate tax.
• Would permit portability of estate tax exemption for spouses (same as Obama). This would allow each spouse to make full use of his or her estate tax exemption and effectively double McCain’s estate tax exemption to $10 Million for married couples. This is a change from current law in that today if one spouse dies without using his or her full exemption, anything not used is effectively wasted.
• Maintain current tax rates (maximum 35%).
• Maintain 15% rate for long-term capital gains and many dividends.
Would reduce top corporate tax rate to 25%. It is currently about 39% for some corporate (C corporation) taxpayers.
• Maintain current Section 179 amounts for expensing.
• Raising taxes by Congress would require a 3/5 majority vote.

 

Barrack Obama
• Payroll tax credit of $500 per person or $1,000 per family to offset some payroll taxes.
• Reform the AMT to eliminate it from the middle class.
• Child and Dependent care credits, along with the savers credit, would increase.
• Would raise the estate tax exemption to $3.5 Million and lower the top tax rate to 45%.
• Would permit portability of estate tax exemption for spouses (same as McCain). This would allow each spouse to make full use of his or her estate tax exemption and effectively double Obama’s estate tax exemption to $7 Million for married couples.
• Would reduce the top corporate tax bracket to 30.5%.
• The top income tax bracket would go to 39.6%, a 4.6% increase and would apply to those who make $350K per year or more. All prior tax cuts under the Bush Administration would remain for those making $250K per year or less.
• An overhaul of the college tax savings and using a $4,000 tax credit per year per student to assist with educational costs. This would be a large tax savings to families with college students over the current law.
• The 15% long-term capital gains tax would increase to 20%.
• Taxes on dividends would be taxed as ordinary income and would lose its favorable current tax status.
• Would end the cap on the number of hybrid vehicles which qualify for favorable tax savings.
• Would help small businesses by offering a 20% tax credit (for investments up to $50K) for investments made to small businesses.
• 0% capital gains rates for start-up businesses to encourage investments in small businesses.

I've highlighted important distinctions (to me at least) between the two. 

What do people think?  Anything inaccurate?

Powered by BlogEngine.NET 1.4.5.0
Theme by Extensive SEO