In late 2006 the Netflix Prize was inaugurated and as of this post, is still going. The premise is simple: write software that predicts how a Netflix subscriber will rate a movie. To win, the predictions have to be 10% better than Netflix. At stake: $1,000,000.

Coding, Not Competing

I have programming talent but this challenge is different; it demands specialized skills. Database optimization and machine learning are a few areas that come to mind. My expertise is not in either area. Why bother to compete if my chances of winning are so small? To put it simply, I wanted to see how far I could get.

• I could barely remember how to write an SQL query. The last time I touched a database was in college when I helped design a web-enabled, automated horse feeder. I used MySQL and PHP. This was the perfect excuse to re-learn those skills.
• I was still using .NET 1.1 at work. The .NET frameworks kept evolving, so it was time to try 2.0. I could have written the software in any language but C# was most relevant to my everyday work.
• Writing interfaces is something that takes practice. Let me rephrase that: writing good interfaces is something that takes practice.
• Why not make this an exercise in GUI design? I love drawing buttons.

Database

The challenge supplies you with a large amount of sample data: 2GB. Unless you’re a seasoned database administrator, this can be intimidating. Where do you start? Is a database even the right direction for such a project? The problems mounted and addressing the actual challenge seemed far away.

I stayed up until 1AM several nights a week trying different ideas. I waited for hours, even a day, as my INSERT statement went through the sample data line-by-line and placed it into the database tables. Finally, a breakthrough: I formatted the sample data into flat files that MySQL could import via LOAD INFILE. Importing was reduced to minutes.

GUI

The problem is to know what tasks are frequently repeated. In the beginning, formatting and importing data took the most time. Thus, three of the four tabs are for that purpose. I also created some basic logging features and began to think multi-threaded. When importing data or making a prediction, such tasks would require lots of time for processing. The worker threads provide feedback on how long they’ve taken to execute.

Cooking Metaphor

I estimate 10% of my time was spent on the core problem, which is generating rating predictions. I spent so much time on the database, caching, and GUI, when I was ready to implement the algorithm to address the challenge, burnout was imminent. Since the previous 90% of my effort addressed 100% of my goals stated above, I felt a warm sense of accomplishment.

Like cooking, most of my time went into preparation, not the actual cooking. If you observe the leader board for several weeks, you’ll see the cooking metaphor in action. You’ll see the same teams making incremental improvements…adding a dash of salt, oregano, etc. It’s trial and error to make the original recipe better. The same goes for prediction algorithms and getting the RMSE just slightly lower.

Conclusion

My FlixLogix source is available for download (63KB). I created the project and solution using Microsoft Visual C# 2005 Express Edition. If you don’t have an ADO.NET driver for MySQL, you need to reference one in order to build. Using this project as-is will yield an RMSE of 1.269. Make my recipe your own. Enjoy!

The Women’s Studies Program at UCF was looking for a logo to represent Women’s History Month 2000. I won the logo contest and got a $150 gift certificate to Barnes and Noble. That bought approximately 1.5 college textbooks for the next semester.

What Works?

The year 2000 is the ultimate “new moon” of my generation, so I represented this transition with very unique zeros that play on the lunar connection. The woman’s side profile looks upward towards the future; hair would have disrupted the clean lines. The profile of the woman running towards you is crossing the finish line on an epic journey.

What Failed?

The ‘2′ in 2000 doesn’t quite fit and draws attention away from the logo’s center. Also, there’s not much contrast between the black and blue colors.

I entered this logo contest because the UCF College of Engineering merged with the College of Computer Science. They wanted a logo to represent the new entity and put out a call for submissions. I’m not sure how many were submitted but none were accepted. Did they bother to advertise to the art students?

What Works?

The logo represents a variety of engineering disciplines: aeronautical, mechanical, and computer. Blue is a vastly accepted color and is easily reproduced on a variety of media. The cube conveys structure and symmetry. What’s left on the other three sides of the cube are left to the imagination.

What Failed?

The white and blue contrast too much. Perhaps adding another color would soften the edges. Also, the logo tries to be too inclusive. Three engineering disciplines are represented but what about the rest? Environmental and electrical fell by the wayside. The cube’s structure tends to draw the eye downward to the “of” text.

This competition comes from new, highly educated labor markets. Ben will likely lose his job for now, but not due to automation or labor union disputes. Something had created a vacuum in job security. Now, following the era of corporate downsizing comes the era of off shore job outsourcing.

Webster would be envious of all these linguistic tricks. Downsizing, as a word, didn’t exist 20 years ago in relation to labor. Trends in labor seem to invent their own vocabulary and solutions, once again with outsourcing. Whereas downsizing sought a long-term growth strategy, the outsourcing strategy seeks lower costs. According to Ben, outsourcing added the indignity of retraining his counterpart lest he forfeit his severance benefits. Is the maxim “what’s good for business is good for the country” still valid? As more workers become disenfranchised and angered by outsourcing, it may prove false.

Outsourcing to foreign countries could be the epitaph for corporate downsizing, if it proves reliable. Instead of doing more with less labor, companies can do more with cheaper labor. Because the focus for white collar labor is now shifting overseas, the U.S. government is positioning itself to rein in the powers of business labor practices. One example: according to the National Foundation for American Policy, 37 U.S. states are now considering legislation that would restrict contracts awarded to offshore businesses.ⁱ For Ben, it quells some tension and prides him to know “America is fighting back,” but will it get him a job?

Whereas immigration created the manpower to do jobs for lesser wages, outsourcing seeks the same result, proactively. Americans are loath to have huddled masses, the poor, or the uneducated, immigrating into our country. In the global economy, we see a shift of U.S.-based manufacturing to countries like Mexico, Korea, and China. It’s natural evolution for the production of high volume products to shift to developing nations. As Americans, we see this and expect it to happen. Sometimes, for a lark, we look for the “Made in China” label on our Wal-Mart products.

The economic base established by governments of developing nations enables their businesses to ascend to world competitor levels, such as India. But for outsourcing to deny our right to do the same exact job, at our pay scales, is what insults an American. U.S. businesses share a philosophy similar to Genghis Khan: it is not sufficient that we succeed (at manufacturing, software design, etc), all others must fail. Unfortunately, they excelled. Because of our hubris the market share of U.S. manufacturers declined. It declined because business leaders failed to see this economic ladder of ascendancy. The net effect is lesser job security for the American worker.

The preeminence of U.S. manufacturing for such a long period and its eventual decline created a huge proportion of blue collar jobs lost to downsizing and/or competition with foreign countries. The opposite of the spectrum is the outsourcing movement of white collar jobs. But, if downsizing affected manufacturing and job security so dramatically, why is the acid debate over such a small percentage of jobs lost to outsourcing? It arises from the common job losses these social and economic classes now share. The critical mixture of these two working classes and their diminished job security to foreign countries has finally bonded our mutual needs for job security, job satisfaction, and…more pay.

Ben has worked part-time before. He knows what it’s like to work without health insurance or medical benefits of any kind. Why, then, should a foreigner get his full-time job and take away his healthcare? After calling the credit card company to dispute his balance, he was reminded by the accented voice on the phone just how closely this whole affair is affecting his life. He can’t even name the president/prime minister/whatever, whose country “stole” his job, but he sure knows what they sound like.

Most troubling to Americans is partaking in a global society where everything seems to be a patchwork of products and services devoid of personality. Our feelings are an inseparable part of our work and our jobs. We’re also patriots. We would pause in disbelief to see a sticker assuring our purchase is “Proudly Made in China.” Such gusto is the norm for American workers who revel in seeing their patriotism (in sticker form) advertised, and with overwhelming pride. Outsourcing has the potential to alienate Americans like Ben, or worse, exorcise the soul of American labor.

Outsourcing provides the workers—the cheap labor—to create an output or something of value. It doesn’t matter who or what is within the outsourced unit of business, just so it provides the intended results. It’s a black box of labor. Without the hassle of unions, individual wants and needs, and immigration sponsorship, the same (or near-same) work is provided. There are serious questions about the black box model of labor management when used on a massive scale. We aren’t there yet, but surely as this trend continues the U.S. government and others will have to assess in what ways job security, worker health and welfare, and training are being curtailed by outsourcing.

But can a black box want for health care, life insurance, or retirement plan? A black box has no soul, no dignity, nor any loyalties. It simply performs. That, above pay or anything else, troubles the American. He wants to be an individual and feel his contribution, not be unitized.

Some Americans see their jobs going overseas as an act of piracy, abetted by trade agreements such as NAFTA. The trade agreements are merely an attempt at controlling an inevitable fact of life, which is the presence of foreign labor, trade, and competition.

The folly is that U.S. businesses have failed to maintain manufacturing supremacy and could not mitigate technology costs to maintain skilled white collar jobs. Those jobs are now being reshuffled between the U.S. working classes and foreign labor markets. Ben never suspected he would take work laying cable in the ground, nor did he suspect a wide swath of immigrants was not only capable of doing these jobs, but they were content to accept the low wages.

Unskilled workers coming into America, jobs going out, and a stagnant job market looked to Washington as if the country had raised a white flag. Fortunately for Ben, the USA Jobs Protection Act, in all its patriotism, says that no U.S. firm will hire a foreign counterpart when a U.S. worker can perform the same job. But pending legislation isn’t stopping the immigrants from coming in droves, nor from jobs going overseas. What, then, are they seeking if outsourcing has contributed to this situation?

For a fee, and no benefits, these individuals fill the gaps that full-timers and part-timers cannot (or will not). Downsizing made these individuals popular choices for employers seeking a lower-cost option for non-essential work. But who’ll represent these people and take the lead for some measure of job security? Unless we are all to be contracted, leased, or manipulated like a black box, there must be oversight and representation for our rights, our health, and our safety.

Americans feel that a right has been stripped when their means to compete have been taken away. Perhaps the USA Jobs Protection Act is as much a statement that Americans can do any job, as opposed to should do any job. Immigrants humble themselves to our country and seek the benefits, opportunities, and freedoms. When outsourcing struck the emotions of the middle class, white collar American worker, he understood emotions immigrants have known since first coming to America: helplessness and indifference to another culture’s ability to do the same job.

The promises made by outsourcing must not out pace the ability to maintain the dignity of workers, globally. Some questions still remain about how disruptive off shore outsourcing is to the U.S. economy. Is the slower upturn in our economic recovery foreshadowing the future? Immigration is skewing the balance of skilled-to-unskilled workers, yet the jobs needed to support those workers have not been created. The U.S. might be more globally integrated, labor wise, but does the amalgam of unemployed labor classes lessen our power to recoup, retrain, and win the next battle for job security?

If this ad hoc workforce is the way of the future, then the government must take a larger role as a surrogate, who’ll define the benefits for the individual and shore up some measure of job security. If not now, then soon, so we can move on and feel better about ourselves and our jobs. As for Ben, his patriotism feels good but it alone won’t pay the bills.

i. National Foundation for American Policy. 5 Aug. 2004. Summary List of States With Proposed Outsourcing Restrictions. http://www.nfap.net/researchactivities/globalsourcing/.

Regardless of how long the workday lasts, an engineer should enjoy his work. Aristotle once said, “Pleasure in the job puts perfection in the work.” Many engineers have created their inventions with the soul purpose of creation itself. Myself included, the feeling of creating something, however mundane, overpowers any notions of necessity. Bringing this spirit to the workplace is essential to a productive engineering career. This is how ideas for novel inventions are seeded.

For all the work put into product development–research, market sampling, trends, and general analysis–sometimes an original idea is only worth the value marketing research says it’s worth. The roots of engineering start at home (even work) and sometimes the best ideas are kept down because of stifling market analysis. Like nature, engineering relies desperately upon randomness. No amount of research can predict the value of an engineer’s idea.

A case in point is the work of Linus Torvalds, the creator of the operating system (OS) Linux. Not only did he recreate an existing idea (a spin-off to the operating system UNIX), he did something unprecedented. He gave away his work for free with only a few stipulations: he is the arbiter to any changes made the OS and changes can never be resold. Linux will always be free. The engineering aspects are shadowed by the method which he delivered his idea to the world. He took all his creative talents, put them into this project, and released it into an unknown void for all to use and modify. Necessity will be the final judgment of the Linux OS.

Creative ideas in engineering are not limited to tangible products. In the case of Torvalds the novelty was in giving away his idea to the world. Today, Linux has an enormous following not limited to independent engineers. Large corporations now see the value of Linux and “open source”–revealing the inner workings of an idea–for all to see and improve upon. The creative process is put into the hands of the world’s engineers. Ideas now evolve on a global scale.

Another example of a creative process gone global is the wireless standard called Bluetooth. It is named after the Norsk explorer, Bluetooth, and a product of years of development by the Ericsson Corporation. The famous producer of cellular phones, Ericsson, is gaining recognition not only for the originality of the idea but the freely distributed standards on which Bluetooth is based.

Even the midnight engineers working in their garages and basements have an equal opportunity for presenting their ideas to the world. The Internet has broken down old communication barriers regarding information sharing. One man’s ambition can become a statewide, national, or even global endeavor. With so many links forming between communities of engineers, a creative product can quickly develop. But equally possible, a novel idea can be dismissed and never seen again.

The Internet has brought this creative process to a head and given new meaning to an evolving idea. The Renaissance of modern times will not be published on the printing press. It will be cut, copied, pasted, and deleted on a Web site for the whole world to see. This will happen not over a period of months or years, rather hours and days. It’s a natural selection of creative ideas taken to an extreme degree. The minutiae of one man’s creativity can be visible to the entire Internet if he chooses.

The deep feeling of satisfaction and pride when an idea comes to fruition is hard to put on a Web page. No matter how well an idea is accepted, the fatherly (or motherly) feeling the creator has for his idea is hard to duplicate. This is the value and limitation of one person’s idea turned into a global project. The creative, emotional value of an engineering project can get lost in our global Web. One person’s project, into which he poured his heart and soul and risked possible failure, can lose its significance when it trades hands multiples times through a corporate engineering team or faceless names on the Internet.

The time-to-market factor is a big strain on the creative process. Wasting corporate time on a project which is (unknowingly) destined for the scrap heap is a horrible prospect many engineers face. As the project begins to take shape, market details or shifting technology trends might affect whether the project is completed. Inventing and creating a novel idea takes time. In many cases the first try might not be sufficient for success. Thomas Edison had many successes but his failures far outnumbered them. It took time for his ideas to evolve and he didn’t let failure cut his ambitions short.

The dedicated engineer cannot let failure be a deterrent. The creative process falls apart when we, as engineers, worry too much about logistical details. Sometimes the randomness and unpredictability of our ideas form a practical solution to a problem. Other times creativity can yield a wonderful solution to no one’s problem. This is the prospect engineers must face each day. A creative idea can be a windfall one month and the next, an obsolete token of technology. The engineer is left to contemplate whether such a time investment is really worthwhile.

With reckless regard for necessity we can act like children, eager to investigate and explore new ideas. As we grow older, failure becomes an overwhelming fear within us. As engineers we have to set aside this fear. The only failure for an engineer is the idea that is never acted upon or given a chance to succeed or evolve globally. This is the evolution of creativity in engineering.