Hadoop – Latency = Google Dremel = Apache Drill???

Hadoop is one of the current IT buzzwords of the day and for good reason – it allows an organization to get meaning and actionable analysis out of “big data” that was previously unusable because it was too big (size constraints). This technology certainly solves a lot of problems but………

What happens if your problem doesn’t easily fit into the the Hadoop framework ?

Most of the work that we do in the financial sector falls into this category. It just doesn’t make sense to re-write existing code to fit into the Hadoop paradigm. Example case study here and blog post here.

As in any business, new ideas lose their ‘edge’ as they sit on the shelf or due to delays in the idea execution stage – primarily because of opportunity costs and increased chances of a competitor creating a product around the idea. The faster a concept can be brought to market, the larger the advantage to be had by the creator. This is especially true in the financial trading tech sector where advancements are measured in minutes/hours/days vs. weeks to months. Because of this, we’re always looking for new and creative ways to solve data and “big data” problems quicker.

Enter Apache Drill

One of the more interesting articles we came across recently focused on a new Apache project that aims to reduce the time to get answers out of a large data set. The project is named Apache Drill and here is a quick overview slide deck.

The Apache Drill project aims to create a tool similar to Google’s Dremel to facilitate faster queries across large datasets. Here is another take on the announcement from Wired. We’re excited about this because of the direct way this will impact our work and specifically the workloads that require real-time or near real-time answers.

Apache Drill Video Overview

[gravityform id=”4″ name=”Subscribe to our Blog” description=”false” ajax=”false”]

Software Development Life Cycle (SDLC) Case Study – Result = $440M Loss

Share this: [wpsr_retweet] [wpsr_plusone][wpsr_linkedin]

 

Software Development Life Cycle (SDLC)Importance

Solid Logic Technology’s foundershave experience across the financial industry and specifically in the development of quantative trading and investment systems. Many of the things we’ve learned along the way impact the way we develop software for clients across other industries. Most notably we’ve learned that software quality is extremely important and ‘software bugs’ cost lots of money. The study below shows how important in-depth software development, testing and launch management is for a company.

As early as August 1st, 2012 reports came out that Knight Capital Group,  a prominent electronic market-making firm specializing in NYSE equities, lost an estimated $440 million dollars due to a ‘software bug’. The news spread across financial news networks like Bloomberg, NY Times, CNBC and The Wall Street Journal.  Knight and other similar firms, trade US equities electronically using sophisticated computer algorithms with little to no human involvement in the process. While we will probably never hear the full story behind the ‘software bug’, it is suspected that a software coding error that was not quickly identified caused the loss. The loss is approximately 4 times their 2011 net income of $115m. It appears to have pretty much decimated the firm and at this point it looks like the firm will be bought or end up in bankruptcy.

While unfortunate, this example has some implications across any software project.

So what can we learn and take away from this incident?

  1. Software is not perfect, especially right after it is released
  2. A more comprehensive Software Development Life Cycle (SDLC) process and launch plan probably would have reduced the loss to a more reasonable amount.
  3. Always have a contingency plan for a new launch
  4. If new software is ‘acting funny’ then it probably has a problem and needs to be pulled from production and fixed
  5. When possible, conduct a series of small ‘pilots’ or ‘beta’ test along the way in a lower impact way
  6. If you cannot fully test the changes, then implement them slowly to minimize the potential errors in the beginning
  7. Have a ‘kill switch’ and know how to use it
  8. Have a formal SDLC process and follow it for all revisions
  9. Use source control for all software changes
  10. Have a defined launch process
  11. Have a way to quickly revert the changes implemented back to the previous version.

These are basic best practices that all software development firms should follow in order to consistently develop high quality software. Its unfortunate that there is a case study like this but these type of incidents are more common (but not to this scale) than most people imagine. I’m sure the group at Knight completed many of the above items, but something got away from them.

We put a huge amount of thought and effort into the process of software development and the consistent high level of quality that a solid process brings. We’re currently working on publishing a set of Software Development best practices – please contact us for a pre-release version.

More posts by Eric

[gravityform id=”4″ name=”Subscribe to our Blog” description=”false” ajax=”false”]

Share this: [wpsr_retweet] [wpsr_plusone][wpsr_linkedin]

When will R overtake SPSS and SAS?

Summary:

R (formally ‘The R Project for Statistical Computing’) is growing quickly in usage and the user base. Based on some recently published forecasts R will overtake SAS & SPSS around 2015.

SLTI Commentary:

We use R for many different items and just got back from the R in Finance 2012 conference last week. Check out our post about thoughts on it here. The findings about R possibly overtaking more commercial languages are interesting, but its a bit of an apples to oranges comparison – R is viewed, in a general sense, as free vs. SAS & SPSS which are commercial packages.  I would be more interested to see some kind of market share or other metrics to show the change occurring in larger commercial installations. We met some of the people from Revolution Analytics last week and are very impressed with their solution – They should do a lot to increase the use of R within the enterprise.

Any ideas on if or when R will overtake either SAS or SPSS in commercial use?

Source:

http://r4stats.com/2012/05/09/beginning-of-the-end/ & http://blog Discover More.revolutionanalytics.com/2012/05/how-long-before-r-overtakes-sas-and-spss.html

Reduce Support Costs by not supporting IE

We’ve all been there – spending down a ton of time chasing some bug that “only happens in Internet Explorer (IE)”. More and more start-ups are dropping IE support to reduce cost and allow the development team to focus on making great software, not covering holes in functionality in IE.

Source: https://paydirtapp.com/blog/we-dont-support-ie/

Freelance Designer Job Opportunity

We’re looking for freelance Graphics Designers!!!!!

Solid Logic Technology Inc. (SLTI) is a ‘virtual’ web and mobile application development company founded in early 2011. We’re based in Birmingham, MI but work out of remote offices across the globe. Our projects are cloud-based and ‘social’ in some way and integrate with Facebook, Twitter, Tumblr, 500px, Instagram, Flickr or other Web 2.0 sites. We’re growing and want to expand our artistic and graphics design team. We’re currently looking for freelance designers with experience in web graphical design, mobile app graphical design & print design.

Benefits for you:

  • Very competitive pay rates
  • Flexible work hours & times
  • Relaxed work environment – Work from your home office/coffee shop/wherever – no office to drive to
  • Use new/emerging technologies and techniques
  • Minimal bureaucracy, red tape or busywork

Requirements of you:

  • Undergraduate degree in Graphics Design, Web Design, Art, or similar experience
  • Experience and knowledge in various platforms of design including web, mobile and print
  • Be current and up to date with web & mobile UI/UX design trends, techniques, etc.
  • A current copy of your portfolio available online or in PDF form
  • Must be an effective communicator. Must be able to communicate design concepts, ideas and reasons for choices to SLTI management and clients
  • Ability and means to effectively present work in person or remotely (i.e. TeamViewer6, Webex, Skype, etc.)
  • Self-Motivated and able to meet deadlines
  • Personal laptop computer (current Mac or PC) and high-speed internet connection
  • Adobe Creative Suite CS4 or higher (Photoshop, InDesign, Illustrator, Dreamweaver, etc.)

How to Apply:

  • Email us at artcareers@solidlogic.com and we’ll be in touch with you. Include the following items with your email:
  • A current résumé and copy of your portfolio (in PDF form or link to it online).
  • Your time availability (hours per week, general days of the week and time of day)
  • Anything else on why you would be the ideal candidate for the position

Solid Logic Technology Inc. Company Overview & Background:

  • Solid Logic Technology Inc. (SLTI) is a ‘virtual’ web and mobile application development company founded in early 2011 by two experienced IT professionals.
  • SLTI partners have a combined 25+ years of software development experience across numerous industries
  • We use a distributed development model – We have no physical offices. All employees work remotely
  • We embrace lean start-up concepts and focus on constant learning and improvement.
  • Most of our employees (outside of managing partners) are freelance contractors
  • We develop both client projects (contract development) and internal properties that we then launch and manage
  • Our clients include established names and brands as well as new start-ups
  • We are very selective in the clients and client projects we take on – we don’t need to take a project in order to ‘pay the bills’
  • We love to build applications using emerging technology, design methods and functionality
  • Our value to clients is that we can provide enterprise-level software design and development skills, lean IT & startup expertise with unmatched flexibility and expertise
  • All of our IT infrastructure is in the cloud – SLTI has no physical data center to manage. All client and internal projects and information is securely stored in the cloud