Monday, June 18, 2012

Time Slicing Organized Labor

Flex has supported multiple pricing models for some time, and the ability to make one pricing model a multiple of another, but as we started taking a second look at Flex's labor functionality we realized we'd need to introduce more flexible ways of calculating pricing without requiring customers to use the overly-technical embedded rules engine.

We needed something with the flexibility and power of a full blown rules engine, but simple enough that non-technical users could reasonably understand and maintain the pricing logic.  We came up with something we call Tiered Pricing - where the pricing logic can be broken into any number of independent tiers.  Each tier has rules that determine when it will be in force for a given calculation and what values will be used in the pricing calculation.

The screen shot below shows a sample pricing tier:


On the left side are the rules that govern when this tier is matched or selected and on the right the rules for how the tier is calculated.

This example shows a tier that, as part of a set of labor pricing tiers, calculates holiday overtime.

The matching criteria determine what counts as holiday overtime and the formula criteria determine what the pricing engine will do once it determines the matching criteria have been met.  If you look at the Multiplier on the right hand side, the formula criteria tells us that holiday overtime will be charged at double time and the Time Quantity Offset tells the system not to count the first eight hours.

(What happens with the first 8 hours?  That's a different tier: Holiday Straight Time.)

Less Abstract, Please

It might be easier to understand pricing tiers if we consider a few examples of how they might be used.  Since the impetus behind designing pricing tiers was handling the complex labor pricing rules we often see in the entertainment and production industry, let's start with those.

For example, it's common for local labor unions like IATSE to have rules about minimum hours, penalties for early or late call times and the higher rates for working on holidays or weekends.

Suppose a local stagehand's union has a four hour minimum call time and charges time and a half for working before 6 AM.  Then assume you need stagehands for a load in scheduled from 5AM to 8 AM.  This is only three hours, so it should trigger the minimum requirement and the call time is before 6 AM, so that first hour should be charged at time and a half.  For this example to work, you'd need two tiers: one matching the after hours times and one matching normal working hours.  On the calculation side, both tiers would probably need a four hour minimum.

Using our tiered system, when you add this call to a quote, you'd see the following:


In this example, you can see that the first hour is billed time and a half and the next two hours are billed at straight time.  The system also detected the four hour minimum and added an extra hour to the last tier.

Time Slicing

As it turned out, getting this simple example to behave as expected was quite a challenge.  We designed our pricing tier system around matching the call time information to a pricing tier.  Time based matching in the default mode is based on the start time of the call.  Without some extra elbow grease, the previous example would have matched the after hours tier since the call starts in that tier's matching criteria - and not the straight time tier.  This would have changed our separate After Hours and Normal tiers to just one instance of the After Hours tier - with all four hours placed in that tier and billed at that rate.  Some union rules are based on when the call starts and this would have been the appropriate way to handle it.  Most situations, however, would require the system to transition to a different set of pricing tiers once the call shifts into normal working hours.  To implement this functionality, we created an alternate mode of matching pricing tiers called time slicing.

The diagram above shows an example of a very long call typical (and perhaps a little optimistic) for stagehands working all day for a tour load-in through load-out, from when the riggers show up early in the morning to when the show is back on the trucks.  To properly price the labor for a long call like this, we're looking at four different pricing tiers: After hours for the early call, straight time, normal overtime, and after hours overtime.

To make this work, our time slicing code has to figure out what times of day trigger changes in the pricing rules, divide the call up into slices corresponding with those times, and run the matching logic separately for each slice.  It has to somehow keep track of the cumulative time while doing all this so that overtime rules get triggered correctly, even if the hours that contribute to triggering overtime come from different time slices.

The algorithm we ended up implementing looks like this:

        if (model.getTimeSlicing()) {
            boolean lastTimeSlice = false;
            float unprocessedTime = params.getTimeQuantity();
            float processedTime = 0f;
            PricingCalculationParameters batchParams = (PricingCalculationParameters)params.clone();
            int reliefValve = 0;
            Map<String, PricingTierCalculationResult> resultMap = new LinkedHashMap<String, PricingTierCalculationResult>();
            while ( (unprocessedTime > 0) && (++reliefValve < 1000) ) {
               
                //first matching attempt is really just to help determine the time slice
                //once the slice duration is established, we rematch
               
                Collection<PricingTierCalculationResult> calcResults = matchTiers(tiers, null, batchParams);
                DateTime startTime = new DateTime(batchParams.getStartDate());
                DateTime endTime = new DateTime(batchParams.getEndDate());
                DateTime nextEndTime = null;
                DateTime candEndTime = null;
                for (PricingTierCalculationResult cand : calcResults) {
                    if (!cand.isMatched()) {
                        continue;
                    }
                    if (cand.getTier().getEndEffectiveTime() != null) {
                        LocalTime time = new LocalTime(cand.getTier().getEndEffectiveTime());
                        candEndTime = new DateTime(startTime.getYear(), startTime.getMonthOfYear(), startTime.getDayOfMonth(), time.getHourOfDay(), time.getMinuteOfHour());
                        if (candEndTime.isAfter(startTime)) {
                            cand.setAdjustedTimeQuantity(dateDiffInHours(startTime, candEndTime));
                            if ( (nextEndTime == null) || candEndTime.isBefore(nextEndTime)) {
                                nextEndTime = candEndTime;
                            }
                        }
                    }
                }
               
                if (nextEndTime == null) {
                    //set the next end time to midnight - this allows us to check for holidays, etc
                    nextEndTime = new DateTime(startTime.getYear(), startTime.getMonthOfYear(), startTime.getDayOfMonth(), 0, 0);
                    nextEndTime = nextEndTime.plusDays(1);
                }
               
                //this means we're on the last time slice and we're done
               
                if (nextEndTime.isAfter(endTime)) {
                    nextEndTime = endTime;
                    lastTimeSlice = true;
                }
               
                double sliceDuration = dateDiffInHours(startTime, nextEndTime);
               
                processedTime += sliceDuration;
                unprocessedTime -= sliceDuration;
                batchParams.setTimeQuantity(new Float(processedTime));
                calcResults = matchTiers(tiers, null, batchParams);
                           
                if (unprocessedTime <= 0) {
                    lastTimeSlice = true;
                }
               
                //adjust time quantities for slice duration
                PricingTierCalculationResult existingResult = null;
                for (PricingTierCalculationResult cand : calcResults) {
                    System.out.println(cand.getTier().getMatchingExpression());
                    cand.setAdjustedTimeQuantity(sliceDuration);
                    cand.setTimeQuantity(sliceDuration);
                    cand.setIgnoreTimeMinimums(true);
                    existingResult = resultMap.get(cand.getTier().getObjectIdentifier());
                    if (existingResult != null) {
                        cand.setTimeQuantity(cand.getTimeQuantity() + existingResult.getTimeQuantity());
                        cand.setAdjustedTimeQuantity(cand.getTimeQuantity());
                    }
                                       
                    double previousProcessedTime = processedTime - sliceDuration;
                    if ( (cand.getTier().getMaxTimeQty() != null) && (cand.getTier().getMaxTimeQty() > 0) && (cand.getTier().getMaxTimeQty() < previousProcessedTime) ) {
                        continue;
                    }
                    if ( (cand.getTier().getTimeQuantityOffset() < 0) && (Math.abs(cand.getTier().getTimeQuantityOffset()) <  previousProcessedTime)) {
                        cand.setIgnoreTimeOffset(true);
                    }
                    resultMap.put(cand.getTier().getObjectIdentifier(), cand);
                    //minimums are only enforced on the last matching time slice
                    if (lastTimeSlice && cand.getTier().getMinTimeQty() != null) {
                        if (  processedTime < cand.getTier().getMinTimeQty()) {
                            cand.setTimeQuantity(cand.getTier().getMinTimeQty().doubleValue() - processedTime + sliceDuration);
                            cand.setAdjustedTimeQuantity(cand.getTier().getMinTimeQty().doubleValue() - processedTime + sliceDuration);
                        }
                    }
                   
                }
               
               
                if (lastTimeSlice) {
                    break;
                }
               
               
               
                batchParams = (PricingCalculationParameters)batchParams.clone();
                batchParams.setStartDate(nextEndTime.toDate());
                batchParams.setTimeQuantity(unprocessedTime);
            }
           
           
            results = resultMap.values();
           
        }
        else {
            matchTiers(tiers, results, params);
        }
In a nutshell, this code starts by determining the total length of the call, initializes a variable called unprocessedTime with that value and then enters a while loop that keeps executing until the unprocessedTime value is 0.

In each loop iteration the matcher is executed twice, first to determine the length of the current time slice and a second time with the calculated time slice duration as an input value.

The length of the time slice is determined by selecting the earliest of any effective end times that may be configured for matched tiers.  A hard break is also placed at midnight so the time slicing logic can detect any transitions between holidays or weekends.  The next end time value is considered the end of the time slice and is compared with the start time to determine its total length, then this value is added to the currently processed time and fed into the matching algorithm.  This weeds out situations where overtime tiers may have matched for the total call duration, but not for a shorter time slice.

Once the results come back from the second matcher invocation, some logic is used to bypass standard minimum calculations (you don't want each time slice triggering minimums; the call has to be considered a whole).  You can see a branch where minimums are enforced if the loop is on the last pricing tier.  We also consolidate tiers that are matched in multiple time slices to prevent confusing clutter in the finished quote.

In Summary

So there you have it, a sneak preview of our new labor pricing system and a look at one of the more interesting bits of code required to make everything work.  We still have some polishing to do, but the new labor system is essentially finished and drops to QA this week.  The next major release of Flex 4 will include this new functionality.  After that, we'll veer off into high speed asynchronous scanning modes and then it's back to Phase 2 of labor where the emphasis will shift from financials to scheduling and conflict resolution.

Wednesday, June 13, 2012

Thank You

Good wishes have been coming in this morning from customers around the world in light of Flex receiving Infocomm's 2012 Rental and Staging Award for Best Rental Management Software.  I'd like to thank everyone for their kind words and everyone who voted for us.  I know a lot of people can be cynical about these awards, but it's hard for us to be cynical this morning.

This award may represent a lot of things to different people, but whatever it may represent, it comes as a much needed boost for us.  Flex as a product is not yet where we would like it to be, but getting it to this point took the hard work and sacrifice of a lot of people and years of commitment, over a decade for me if you count Shoptick and shoptick.com.

I think the picture from the ceremony of our CEO, Chris Stein, receiving the award in Las Vegas yesterday says a lot.


Chris looks happy and proud here, but he also looks exhausted, and it's not hard to understand why: This guy works harder than anyone I've ever met and I'm a software engineer, in a profession known for its workaholics and "software widows".  It's his dogged determination and focus on the product and user's point of view that took Shoptick's dozen or so customers to 160+ customers in a dozen countries.  Though Chris and I don't always agree, he's the linchpin that holds us together, he picks up the managerial slack when I'm stuck in the code, and I consider him an invaluable partner, the piece that was missing during the Shoptick days.

People who work with such dedication and passion aren't in it for the money.  I think that's what makes working for Flex such a great experience for me.  Not only do I get work on a product I created and believe in, but I get to work in a company culture that focuses on the right things, not just how to extract more money from the customer with the least amount of effort.

And though Chris's determination and limitless energy set the example for all of us, the whole team at Flex makes a big contribution every day.

Suman, from his home in Calcutta, never ceases to amaze me with his endurance and productivity. Whenever I set the bar a little higher, he always reaches it, forcing me to push the bar even higher next time.  He recently left E-Force Global to join Flex full time and he's continuing to do great work on our new labor and pricing tier system.

Roger's initiative and hard work allowed us to integrate with QuickBooks and QuickBooks Online in a flexible, meaningful way.  It would have been very easy for our QuickBooks integration to be just complete enough to justify the bullet point on our feature list, but Roger worked with our customers to make it as useful as it can be.  Roger's also put in countless hours designing and maintaining reports, which can be a dreary and thankless task.

And though Courtney's new to the team, she's quickly become the company's conscience, keeping us honest, and helping us avoid the temptation to just push to production without thorough testing.  Though there are always surprises and room for improvement, I personally think Courtney's dedication to her job has made a noticeable difference in the stability and quality of new software updates.

And I want to acknowledge Devon for his work supporting our customers.  He's had to do it under less than ideal circumstances, subject to ongoing legal threats and harassment from his former employer.  He's been put in a tough position and has handled it with poise and professionalism.  His trial by fire has made him a loyal and valuable collaborator, and a passionate advocate for the customer in our design and planning process.

The Future of Flex

An occasion like this always puts one in mind of the future.  I've often said that we still have a lot of work left to make Flex the product we want it to be and it always seems like new feature development never moves quite as fast as we or our customers would like.  Yet we do have some big things coming just over the horizon.

Labor

We're almost done with Phase 1 of a major enhancement intended to add more powerful labor planning features to Flex.  Chris has the latest development build with him at Infocomm this week, and in this unreleased version you'll be able to see our approach to labor and tiered pricing, with a powerful tier based model that takes into account complex after hours, overtime and other labor rules without forcing users to remember all the complexity.  Our design approach for labor pricing also enables tier based pricing for anything in Flex, including rentals and retail items.  This can be leveraged to provide bulk pricing or have different rental tiers without relying on multipliers.

The next phase will greatly expand how labor is scheduled, including new first-in-industry tools for booking freelancers and ensuring that labor requirements are covered without conflicts.

Faster Scanning

Another key enhancement that's been on our radar for some time is providing asynchronous scan processing for use cases where doing everything in real time is unnecessary - like returns.  We'll be working on several methods of making the scan process more efficient for high throughput customers.  We'll also be introducing methods for supporting in-the-field transfers between shows and bulk scan options intended to support rapid turnarounds.


Multi-Session

We've laid some groundwork and built much of the preliminary architecture needed to support multi-session event planning - for equipment and on-site labor.  More work on this is planned for the coming year.

Building a Better Cloud

One of the challenges we face is how to make Flex more fault tolerant as the customer count grows without adding cost and increasing our prices.  We'll be working on a number of major architectural enhancements designed to take better advantage of the cloud platform, a project we've taken to calling high-availability Flex or the TruCLOUD.  This project is intended to make Flex more fault tolerant without increasing our operating costs.  It will include multiple load balanced server instances, out of process report generation (for more consistent performance), and dynamic scaling for peak usage periods.

Mobile Platforms

The can that always seems to get kicked down the road around here is mobile development.  Part of the reason for this is that customers have never really stressed it as a priority, but that's changing.  We've done some proofs of concept and will likely roll out iPhone and possibly Android applications over the next 12 months, particularly in support of field transfers and our new labor module.

A Very Good Year

It's been a great 2012 so far at Flex and we're looking forward to what happens next.  As is often the case, our goals may outstrip our resources, but we'll do our best.  Thanks again to everyone for supporting Flex.  We'll keep working hard to earn your continued support.


Thursday, June 7, 2012

Fonts, Fonts, Fonts

One issue we've kind of side stepped around for awhile is the issue of custom fonts in reports, specifically reports built with the Jasper Reports framework. For a while we were just building all our reports in a standard font which was included in the build so this worked fine.

However, when we build custom reports for customers, they sometimes want text to be in specific fonts to match their branding. This is a completely reasonable requirement, but we just needed a sure fire way to deal with it.

In past times we would just take the font and install it on the customer's server however this was brittle since as soon as you would move the customer to another server, it would break their custom reports. Also this approach was problematic in rendering the PDFs correctly on any machine.

So the best way is to just include the fonts in the build. Fortunately Jasper Reports includes something called font extensions that make this very easy. Basically you just put a properties file in the default package called "jasperreports_extension.properties" and Jasper will automatically find the file. This file just sets a couple properties and tells Jasper where to find the font XML configuration file.

Example jasperreports_extension.properties:


net.sf.jasperreports.extension.registry.factory.fonts=net.sf.jasperreports.engine.fonts.SimpleFontExtensionsRegistryFactory
net.sf.jasperreports.extension.simple.font.families.frsfamily=fonts/frsfontsfamily.xml

Next you create a "fonts" directory, place your True Type Format fonts in there, and create an XML file that configures the fonts.

Example XML font config file:

<?xml version="1.0" encoding="UTF-8"?>
<fontFamilies>

   <fontFamily name="NewsGothicBT-Roman">
       <normal>fonts/News Gothic.ttf</normal>
       <bold>fonts/News Gothic Bold.ttf</bold>
       <pdfEmbedded>true</pdfEmbedded>
   </fontFamily>

   <fontFamily name="Planer">
       <normal>fonts/Planer_Reg.ttf</normal>
       <bold>fonts/planerdemibold.ttf</bold>
       <pdfEmbedded>true</pdfEmbedded>
   </fontFamily>

</fontFamilies>

Jasper reads the above XML and registers the fonts. Whenever the above font names are referenced in a report, the TTF fonts are used and embedded in the PDF itself (notice pdfEmbedded=true above) so that the PDF will be rendered correctly on any system that opens it.

So from now on whenever a custom font is needed, we just drop the TTF font files in the fonts directory, configure them in frsfontsfamily.xml and they are available everywhere.

Our next step might be to create a way to dynamically load fonts onto a live system allowing for more flexibility but for now we are happy with being able to include fonts in the build guaranteeing that they will be available and embedded in the PDFs.